Statistics 101: Linear Regression, The Very Basics 📈

Summary

In this instructional video, Brandon introduces the concept of simple linear regression, emphasizing its foundational role in statistics. He motivates viewers with encouragement and offers to connect on social media. The video explains that without an independent variable, the best prediction for a dependent variable is its mean. Brandon uses a restaurant tipping scenario to illustrate this, showing how to predict tip amounts based on their average. He also introduces the idea of residuals and the sum of squared residuals (SSE), which is key for assessing the fit of a regression model. The video sets the stage for upcoming discussions on more complex regression analyses.

Takeaways

• 😀 Stay positive and believe in your ability to overcome challenges in learning statistics.
• 📢 Follow the presenter on social media platforms to stay updated with new video releases.
• 👍 Engage with the content by liking, sharing, and providing constructive feedback to help improve future videos.
• 📊 Simple linear regression involves modeling the relationship between two variables: an independent variable and a dependent variable.
• 📈 The 'goodness' of a regression model is determined by how well it compares to a model that assumes no relationship between variables.
• 🔢 In the absence of an independent variable, the mean of the dependent variable is used as the best predictor.
• 📋 The term 'regression' specifically refers to simple linear regression unless mentioned otherwise.
• 📉 Residuals, or the differences between observed values and the mean prediction, are a key concept in understanding regression.
• 🔴 The sum of squared residuals (SSE) is a measure used to evaluate how well the regression line fits the data.
• 📉 The goal of simple linear regression is to find a line that minimizes the SSE, indicating a better fit to the data.
• 📖 The upcoming videos will delve deeper into the concepts and calculations involved in simple linear regression.

Q & A

What is the main theme of the video series?

The main theme of the video series is basic statistics, with a focus on simple linear regression in this particular video.

What is the first piece of advice given by Brandon to viewers who are struggling in their class?

The first piece of advice is to stay positive and keep their head up, acknowledging that they have already accomplished a lot and that hard work, practice, and patience will help them through their struggles.

How does Brandon encourage viewers to stay connected with his content?

Brandon encourages viewers to follow him on various social media platforms like YouTube, Twitter, Google Plus, and LinkedIn to be notified when new videos are uploaded.

What is the significance of liking and sharing the video as mentioned in the script?

Liking and sharing the video is a way to encourage Brandon to continue making educational content, and it helps to spread the knowledge to classmates, colleagues, or through playlists.

What is the purpose of the 'tips for service' example used in the video?

The 'tips for service' example is used to illustrate how regression can be used to predict the amount of tip one might expect based on the total bill amount at a restaurant.

What is the dependent variable in the 'tips for service' example?

In the 'tips for service' example, the dependent variable is the tip amount.

Why does Brandon emphasize the importance of understanding the underlying meaning behind good regression models?

Brandon emphasizes understanding the underlying meaning behind good regression models to ensure viewers not only know what is happening but also why and how to apply it.

What does Brandon mean when he says 'regression allows us to model mathematically the relationship between two or more variables'?

This means that regression analysis helps in establishing a mathematical model that describes how one variable (dependent) is related to one or more other variables (independent).

What is the best prediction for the next tip amount if only the tip data is available?

If only the tip data is available, the best prediction for the next tip amount is the mean of the existing tip amounts.

What is a residual in the context of this video?

A residual is the difference between the observed value (actual tip amount) and the predicted value (mean of tips or value on the best fit line).

Why are residuals squared in the calculation of the sum of squared residuals?

Residuals are squared to ensure all values are positive and to emphasize larger deviations from the mean, which helps in calculating the sum of squared residuals (SSE).

What is the goal of simple linear regression as described in the video?

The goal of simple linear regression is to create a linear model that minimizes the sum of squares of the residuals, which is another way of saying it minimizes the sum of squares of the error.

Outlines

00:00

📚 Introduction to Basic Statistics

Brandon begins the video by encouraging viewers struggling in a class to stay positive and reminding them of their accomplishments. He emphasizes the importance of hard work, practice, and patience. Brandon invites viewers to follow him on various social media platforms to stay updated with new content and stresses the value of connecting with his audience. He also encourages viewers to like, share, and give feedback on the video to help improve future content. The video aims to cover basic statistics concepts, specifically simple linear regression, in a slow and deliberate manner to ensure understanding.

05:00

📈 One-Variable Prediction Model

In this section, Brandon discusses how to predict the tip amount for future meals using only the data collected on tip amounts. He explains that with only one variable, the best prediction for any given tip amount is the mean of the sample, which in this case is $10. He demonstrates how to create a graph with meal numbers on the x-axis and tip amounts on the y-axis, and then plots the data points. Brandon emphasizes that the mean is the best predictor in the absence of additional variables and introduces the concept of residuals, which are the differences between observed values and the mean.

10:02

🔍 Understanding Residuals and Error

Brandon elaborates on the concept of residuals, which are the distances of the observed data points from the mean (best fit line). He explains that residuals are also known as errors and always sum up to zero. To quantify the deviation of each data point from the mean, he introduces the idea of squaring the residuals, which serves to make all values positive and emphasize larger deviations. The sum of these squared residuals is referred to as the Sum of Squared Residuals (SSR) or Sum of Squared Errors (SSE), which is a key metric in assessing the fit of a model to the data.

15:03

📉 Minimizing Sum of Squared Residuals

Here, Brandon reveals the core objective of simple linear regression, which is to create a linear model that minimizes the sum of squared residuals. He explains that by introducing an independent variable, the model can account for some of the error, thus reducing the SSE. The video uses a hypothetical scenario where the independent variable (bill amount) is ignored, and the model's effectiveness is judged by how much it can reduce the SSE compared to a model that only uses the mean of the dependent variable (tip amount).

20:05

🔄 Recap and Anticipation for Future Content

In the final paragraph, Brandon reviews the key points of the video, emphasizing that simple linear regression is a comparison between a model with an independent variable and one without. He reiterates that the best prediction for future values, when only the mean of the dependent variable is known, is that mean itself. Brandon also previews that future videos will delve deeper into the role of the independent variable in explaining the dependent variable and how it affects the residuals and the SSE.

Keywords

💡Statistics

Statistics is a branch of mathematics dealing with the collection, analysis, interpretation, presentation, and organization of data. In the video, statistics is the central theme, as the presenter discusses basic statistical concepts and their application in real-world scenarios such as predicting tip amounts in a restaurant.

💡Regression

Regression refers to a set of statistical methods used to understand the relationship between a dependent variable and one or more independent variables. The video focuses on simple linear regression, which is a basic form of regression analysis that uses a straight line to describe the relationship between two variables.

💡Dependent Variable

A dependent variable is the variable that is being predicted or explained by other variables in a regression analysis. In the context of the video, the tip amount is the dependent variable, which the presenter aims to predict based on other factors.

💡Independent Variable

An independent variable is a variable that is believed to influence the dependent variable. In the video, the presenter mentions that in a typical scenario, the total bill amount would be an independent variable that influences the tip amount.

💡Residual

Residuals are the differences between the actual observed values and the values predicted by a regression model. The video explains that residuals are calculated by subtracting the predicted tip amount (mean) from the actual tip amount and are crucial for evaluating the fit of the model.

💡Mean

The mean, often referred to as the average, is calculated by adding all values in a data set and then dividing by the number of values. In the video, the presenter uses the mean of the tip amounts as a baseline prediction when no other variables are considered.

💡Sum of Squared Residuals (SSR)

SSR is a measure used in regression analysis to quantify the discrepancy between the data and an estimation model. The video script explains that SSR is the sum of the squares of residuals and is used to evaluate how well the regression model fits the data.

💡Best Fit Line

The best fit line is the line that minimizes the sum of squared residuals between the observed values and the values predicted by the model. The video discusses how the goal of regression is to find this line that best fits the data.

💡Linear Model

A linear model is a statistical model in which the relationship between the dependent variable and independent variables is assumed to be linear. The video is part of a series on simple linear regression, which uses a linear model to predict outcomes.

💡Standard Deviation

Standard deviation is a measure of the amount of variation or dispersion in a set of values. Although not directly mentioned in the script, the concept is related to residuals and their squaring, which is a step in calculating standard deviation.

💡Constructive Comment

A constructive comment is a suggestion or critique that is intended to improve or provide useful feedback. The video script encourages viewers to leave constructive comments if they think there's something the presenter could do better.

Highlights

Encouragement for viewers struggling with statistics classes to stay positive and persevere.

Invitation to follow the presenter on various social media platforms for updates.

Request for viewers to give feedback through likes and constructive comments.

Introduction to the concept of simple linear regression as a basic statistical method.

Explanation of regression as a way to model relationships between variables using algebra.

Clarification that 'regression' in this context refers to simple linear regression.

Discussion on the importance of understanding the 'goodness' of a regression model.

Introduction of basic regression terminology and concepts.

Explanation of the absence of formulas or calculations in this introductory video.

Description of a real-world scenario involving predicting tip amounts in a restaurant.

Challenge to predict future tip amounts using only the data collected.

Introduction to the concept of visualizing data through graphs and charts.

Explanation of how to graph data points with meal numbers and tip amounts.

Discussion on using the mean as the best predictor when only one variable is available.

Introduction to the concept of residuals and their calculation.

Explanation of why residuals are squared in statistical analysis.

Introduction to the sum of squared residuals (SSE) and its significance.

Revealing the goal of simple linear regression: to minimize the sum of squares of the residuals.

Discussion on how introducing an independent variable can reduce the SSE.

Conclusion that simple linear regression is a comparison of two models: one with and one without an independent variable.

Final thoughts on the importance of understanding the comparison between models in regression analysis.

Transcripts

(gentle acoustic guitar music)

- [Brandon] Hello, thanks for watching,

and welcome to the next video in my series

on basic statistics.

Now as usual, a few things before we get started.

Number one, if you're watching this video

because you are struggling in a class right now,

I want you to stay positive and keep your head up.

If you're watching this, it means you've accomplished

quite a bit already.

You're very smart and talented,

but you may have just hit a temporary rough patch.

Now I know with the right amount of hard work, practice,

and patience, you can work through it.

I have faith in you,

many other people around you have faith in you,

so so should you.

Number two, please feel free to follow me here on YouTube,

on Twitter, on Google Plus, or on LinkedIn.

That way when I upload a new video, you know about it.

And it's always nice to connect with my viewers online.

I feel that life is much too short

and the world is much too large

for us to miss the chance to connect when we can.

Number three, if you like the video,

please give it a thumbs up.

Share it with classmates or colleagues,

or put it on a playlist.

That does encourage me to keep making them for you.

On the flip side,

if you think there's something I can do better,

please leave a constructive comment below the video,

and I will take those ideas into account

when I make new ones.

And finally, just keep in mind that these videos are meant

for individuals who are relatively new to stats.

So I'm just going over basic concepts.

And I will be doing so in a slow, deliberate manner.

Not only do I want you to know what is going on,

but also why, and how to apply it.

So all that being said, let's go ahead and get started.

Okay, so this is the first video in what will be,

or is, depending on when you're watching this,

a multi-part video series about simple linear regression.

In the next few minutes, we will cover the basics

of simple linear regression starting at square one.

And for the record, from now on if I say just regression,

I am referring to simple linear regression

as opposed to multiple regression

or models that are not linear,

which we will hopefully get to those at a later date.

Now regression allows us to model

mathematically the relationship between

two or more variables, using very simple algebra,

to be specific.

For now, we'll be working with just two variables;

An independent variable, and a dependent variable.

The truth is, when we talk about how quote "good"

a regression model is, we are actually comparing it

to another specific model.

Oftentimes, students don't realize this.

So in this video, we're gonna talk about that idea.

I will also begin introducing basic terminology and concepts

that will carry you through your work using regression.

There are no formulas or calculations in this video.

We're just introducing the underlying meaning

behind good regression models.

So if you are new to regression,

or are still trying to figure out exactly what it even is,

this video is for you.

So sit back, relax, and let's go ahead and get to work.

So as always, I like starting out my videos with a problem,

and a relatively real world problem at that.

So we'll call this one tips for service.

So let's assume that you are a small restaurant owner,

or a very business-minded server or waiter

in a nice restaurant.

Here in the US, tips are a very important part

of a waiter's pay.

Most of the time, the dollar amount of the tip

is related to the dollar amount of the total bill.

So if the bill is $5, that would have a smaller tip

than a bill that is $50.

Now as the waiter or the owner,

you would like to develop a model

that will allow you to make a prediction

about what amount of tip to expect

for any given bill amount.

So therefore, one evening, you collect data for six meals.

So a random sample of six meals.

But unfortunately, when you begin to look at your data,

you kind of forgot something.

You realize you collected data for the tip amount

and not the meal amount that goes with it.

So unfortunately right now,

this is the best data you have.

So you have a random sample of six meals

and the tip amount for each one of those meals.

So $5, $17, $11 and so on.

Now here's the question.

How might you predict the tip amount for future meals

using only this data?

There's only one variable here, the tip amount.

The meal number's just a descriptor.

So we have one variable, the tip amount.

But I still want to challenge you to come up with a model

that will allow you to predict within some reason

what the next tip is going to be.

How can you do that?

Think about it.

So the first thing we're gonna do

is we're going to visualize our data.

As you know if you watch my other videos,

I am a huge advocate of visualizing our problems,

making charts, graphs, diagrams, whatever we have to do

to make them visual.

So the first thing we'll do is we'll make a graph

of our tips.

Now on the x-axis on the bottom, we have our meal number.

Now that's not a variable,

that's just a descriptor of what meal we're graphing.

Now on the y-axis, or the vertical axis,

that's where we will graph our tip amount.

Let's go ahead and see what this looks like.

So for meal one, with a tip of $5,

so we'll go ahead and graph that at around $5.

For meal two, with a tip of $17, so that goes way up there.

For meal three, with a tip of $11, so that goes there.

Meal four, with a tip of $8, that goes there.

Meal five, that was a $14 tip.

And meal number six, that was a $5 tip.

So here are our data points.

Remember, we're only dealing with one variable,

that's the tip amount,

and the meals along the bottom just describe

where we're graphing each point.

And the order does not matter.

We could have graphed these in any order.

This just happens to be the one we ended up with.

Now, what's really the most you can figure out

about this data?

How would you predict what the tip for

meal number seven would be?

Is it going to be like meal number six, it's $5?

Is it gonna be like meal number two, to $17?

How would you come up with the best guess or estimate

for the next meal using only one variable?

Well, you would use its mean.

So the mean for all six tips is $10.

So guess what?

That's the best we can do.

With only one variable, the best estimate

for the best prediction,

for any given meal tip is $10.

So go ahead and put a line at $10.

So that for this model, that is our best fit line,

that's all we have.

One variable, tip amount, the mean is the best predictor

of any given tip amount.

Now obviously if you look at this chart,

our tips do not fall on the $10 line,

they're scattered around it.

But still, it's the mean.

So that's your best estimate for the next tip

for any given tip would be.

So here's our graph again with our tips, our mean,

and our tip amount.

I just want to stress that the tip amount is y bar,

so that's the mean of y, and that's for two reasons.

One, the dependent variable,

which it will be as we progress forward

is always the y of the x and y axes,

and of course we're graphing it on the y-axis,

so it should be y bar.

So here it is, the basic concept I really want

you to remember in your head as you go forward.

But obviously, simple linear regression

is about two variables.

But, we're starting off here,

'cause this is where it all begins.

With only one variable and no other information,

the best prediction for the next measurement

is the mean of the sample itself.

So the variability in the tip amount,

'cause they're not on the line, they're above and below,

the variability in the tip amounts can only be explained

by the tips themselves because that's all we have.

So the way they're above and below the line,

that's just the natural variation in the tips.

But the basic point is this;

With only one variable, the best way, the only way

we can make a prediction about what the next tip amount

in this case is the mean.

So our best prediction for the tip

of meal of number seven is $10.

So let's talk about the goodness of fit

for this line and our tips.

Now obviously we know that the data points,

the actual observed values do not fall on that line,

they do not fall on the $10 line,

some are above and some are below it.

So that tells us how good this line

fits these observed data points.

Now one way we can do that

is to measure the distance they are

from that best fit line.

Now we did this to some degree when we were talking about

standard deviation.

Remember, we're talking about the distance

each data point is from the mean.

But guess what we're doing here?

The distance that each data point is from the mean,

because the mean is our line of $10 here.

So, for meal number one, our tip was $5.

so that's $5 below our mean of $10, so that's negative five.

Meal number two, got a tip of $17,

that was $7 above our mean.

Meal three was $11, $1 above our mean.

Meal four was $8, that's two below our mean.

Meal five was $14, that's four above our mean.

And meal six is $5, that's five below our mean.

So these are the distances, in this case, dollar amounts

by which each observed value is different from

or is, deviates from

the mean of $10.

Now we have a name for these,

they're called residuals.

So the distance between the best fit line,

which in this case, 'cause it's one variable is $10,

the distance from the best fit line to the observed values

are called residuals.

Now they're also called the error.

So the distance is also called the error because

that's how far off the observed value is

from the best fit line.

Now you notice a few more things here.

If you add up the residuals on the top,

just above the line, seven plus one plus four, that's 12.

Add up the residuals below the line,

five, two and five, so that's minus 12.

So the residuals always add up to zero.

Now that's another important concept to keep in mind

as we go forward.

But if you remember in standard deviation,

one of the steps was that we took the deviations

from the mean and we squared them.

Well guess what?

We're gonna do the exact same thing here.

So the residual for meal one one was $5,

so it's $5 below, so we square that and it squares to 25.

Meal number two, it was $7 above,

we square seven, that's 49.

So on and so forth.

So the right-hand column of our table,

we have our squared residuals.

Now the question is why do we square them?

Well we square them for the same reasons

we square the deviations when calculating

the standard deviation.

Number one, it makes them all positive.

So if we square a negative number,

it obviously makes it positive.

And number two, it emphasizes the larger deviations.

So a deviation of two will square to four.

But a deviation of five will square to 25.

So the squaring really exaggerates

the points that are further away.

Now what we can do is we can take these residuals,

these squared residuals in the right-hand column

and we can add them up.

And they're called the sum of squared residuals,

or the sum of squared errors, or the SSE.

Now where have you heard that before?

Well you've heard it everywhere in statistics.

You've obviously heard it in standard deviations,

you've heard it in ANOVA.

Same idea, sum of the squared errors.

It's a fancy way of saying we add up the squared residuals.

And when we do so, it's 120.

Now when we say squaring the residuals,

we literally mean squaring them.

So, 25 over here in the left-hand side,

that's negative five squared.

49 is seven squared, and so forth.

Well we actually mean squares,

so when we square each residual, or error,

we're actually making squares.

So when we say sum of squares,

we literally mean the sum of squares.

So 49 plus 25 plus one plus four plus 16 plus 25

adds up to 120.

Now, here is sort of the blockbuster bombshell concept

of this video.

The goal of simple linear regression

is to create a linear model that minimizes

the sum of squares of the residuals,

same thing as the sum of squares of the error.

So what we're gonna do is we're gonna create

a different line through the data,

once we introduce an independent variable that will minimize

the size of these squares.

And actually mathematically,

we'll come up with the line through the data

that minimizes these squares as much as they can be.

And that will be our best fit line for the data.

But again, in this problem, we're only using one variable,

we're only using the dependent variable.

So when we introduce the independent variable,

it will sort of take away for its own self

some of this error we see here.

If our regression model is significant,

it will eat up some of the raw error we had

when we assumed, like in this problem,

that the independent variable did not even exist.

So what we're doing here in this problem,

we're taking a simple linear regression problem

that in theory has a independent variable,

called the bill amount,

and a dependent variable called the tip amount.

But what we're doing is pretending that the

bill amount doesn't even exist.

We're only using the tip amount.

So, that creates a sum of squared residuals of 120.

Now, when we introduce the independent variable of

bill amount, what will happen is that we'll create

a different best fit line through our data.

And what it will do is it will sort of

eat up some of this sum of squares.

So when we do regression,

we're gonna have sum of squares regression

and sum of squares error.

So by introducing that independent variable of bill amount,

it will create a new line that goes through the data.

That new line will explain some of the sum of squares,

and therefore it will reduce the SSE down,

the sum of these squares as much as it can be.

So the regression line will and should

literally fit the data better.

It will minimize the residuals.

So when conducting simple linear regression

with two variables, we will determine how good that line

fits the data by comparing it to this type,

where we pretend the second variable does not even exist.

So when we say a model is good,

a linear regression model is good,

what we're saying is that it reduces the sum of squares

of the error by a large amount.

Which is another way of saying

we're comparing the other best fit line

to this one you're looking at right here.

Simple linear regression is always in comparison

to what we would have if we only had the dependent variable.

So if a two variable regression model

looks like this example,

what does the other independent variable do

to help us explain the dependent variable?

Well, it does nothing.

If we introduced bill amount into a two variable

simple linear regression, but the best fit line

looks exactly like this,

then the bill amount didn't give us anything.

It didn't explain the variability in the tip amount

anymore than the tip amount itself did.

So we're always comparing our simple linear regression

best fit line to this one.

Basically, the mean of the dependent variable alone.

Okay, so quick review.

So simple linear regression

is really a comparison of two models.

The first one is where the independent variable

does not even exist and we just use the mean

of the dependent variable, like we did in this video.

And the other use uses the best fit regression line,

where we go ahead and introduce that second variable,

the independent variable in this case, the meal amount,

and that creates a different line,

and then we compare that to the first one.

But if there's only one variable like in this example,

for this video, the best prediction for other values

is the mean of that dependent variable.

In this case, it was $10.

Now the difference between the best fit line

and the observed value is called the residual, or the error.

The residuals are squared and then added together

to generate a sum of squares,

literally residuals or errors, or SSE.

So we square the residuals and add them together,

sum of squared residuals,

or most often it's called sum of squares error.

So simple linear regression is designed

to find the best fitting line through the data

that minimizes the SSE,

that minimizes the area of the sum of squares residuals,

that minimizes the area of the sum of squares error.

And actually through calculus, it is the best fitting line.

Now I'm not going to go into the calculus behind that,

but you're just gonna have to start trusting me on faith

that when we come up with a best fit line

in simple linear regression, that literally is the

best fit line that reduces the SSE.

Okay, so that wraps up our very first video of mini

on simple linear regression.

So I just want you to realize in this video

that later when we talk about the best fit line

in regression, we're actually comparing it to the situation

where we don't have the independent variable at all.

We're just comparing it to the case,

where we're looking at the mean of the dependent variable.

So in this case, in this video,

all we had was the mean of the tips, 10 bucks,

that's all we had to go on.

So therefore, our best guess or best prediction

for the next tip was $10.

Now later, when we introduced the meal amount,

what will happen is we'll get a different best fit line

that will explain or take up some of that error

in the regression, it'll reduce the error,

we'll have a different line,

then we'll have smaller, hopefully, residuals.

If the regression line is flat across,

like we saw in the first example of this one,

then the regression doesn't tell us anything,

the meal amount doesn't mean anything,

so the best guess is really just the mean of the tips.

So, we'll go more into this example in the second video,

just wanted to lay the foundations for that.

Look forward to seeing you next time.

(gentle acoustic guitar music)