Regression Analysis Basics

Regression analysis estimates how an outcome (Y) changes when one or more input variables (X) change, and uses that relationship to predict Y for new data. Linear regression, the most common form, fits the line Y = b0 + b1X that minimizes the sum of squared differences between predicted and actual values.

What is regression analysis used for?

Use regression to predict a continuous outcome from one or more input variables and to measure how much each input moves that outcome. For example, predict house price from size, bedrooms and location, and learn how many dollars each extra square foot adds.

That gives regression two jobs: prediction (what will Y be for this new case?) and explanation (which inputs matter, and by how much?).

How do you interpret regression coefficients and R-squared?

A coefficient is the expected change in Y when its X rises by one unit and the other inputs stay the same, and its sign gives the direction. R-squared is the proportion of the variance in Y that the model explains, from 0 (none) to 1 (all).

Coefficient (slope): Change in Y when X increases by 1 unit

• Size coefficient = 150 → Each extra sqft adds $150 to the predicted price
• Negative coefficient → Inverse relationship

R-squared: Percentage of variance explained

• R² = 0.85 → Model explains 85% of variation in Y
• Higher is better, but context matters
• R² never falls when you add a predictor, so compare models with different numbers of predictors using adjusted R², which penalizes inputs that don't improve the fit

P-value: Each coefficient's p-value tests the null hypothesis that its true value is zero. Below 0.05 is the usual threshold for calling a predictor statistically significant, and it only means something if the assumptions below hold.

What are the assumptions of linear regression?

Linear regression assumes a straight-line relationship between X and Y, independent observations, constant error variance (homoscedasticity) and, mainly for p-values and confidence intervals in small samples, normally distributed residuals. A residuals-vs-fitted plot checks most of them at once.

1. Linear relationship - Plot X vs Y to verify
2. Independent observations - No repeated measurements
3. Homoscedasticity - Error variance is constant
4. Normal residuals - For small samples especially

Our linear regression assumptions guide shows how to check each one and what to do when it fails.

How do you run a simple linear regression in Python?

For a single predictor, scipy.stats.linregress(x, y) returns the slope, intercept, correlation (r), p-value and standard error in one call; square r to get R². The example below fits price against size for seven houses.

from scipy import stats

size = [1100, 1400, 1600, 1900, 2200, 2500, 2800]           # square feet
price = [199000, 262000, 251000, 330000, 352000, 425000, 431000]

result = stats.linregress(size, price)
print(f"Slope: {result.slope:.1f}")
print(f"Intercept: {result.intercept:,.0f}")
print(f"R-squared: {result.rvalue**2:.3f}")
print(f"P-value: {result.pvalue:.5f}")

# Predict the price of an 1,800 sq ft house
print(f"Predicted: {result.intercept + result.slope * 1800:,.0f}")

# Slope: 142.6
# Intercept: 46,324
# R-squared: 0.963
# P-value: 0.00009
# Predicted: 303,088

Each extra square foot adds about $143 to the predicted price, size explains 96% of the variation in price across these houses, and the tiny p-value means a slope this steep would be very unlikely if size had no linear relationship with price. The fitted equation is price = 46,324 + 142.6 × size, so an 1,800 sq ft house is predicted at about $303,000. Don't read the intercept as the price of a 0 sq ft house: that is far outside the data. For several predictors, statsmodels' ols formula interface gives a summary table with every coefficient, p-value and R².

What is the difference between correlation and regression?

Correlation measures how strongly two variables move together on a unitless scale from −1 to 1 and treats them symmetrically; regression gives an equation that predicts Y from X in Y's units. In simple linear regression the two are linked: R² equals the squared correlation coefficient, r².

Neither one proves causation. A coefficient describes an association unless the data comes from a randomized experiment; see correlation and causation for the usual traps.

Common Mistakes

The three mistakes that most often produce misleading regressions are predicting outside the range of your data, ignoring correlated predictors and judging the model by R² alone.

Extrapolating beyond your data range - Model doesn't know what happens outside training range

Ignoring multicollinearity - Correlated predictors make coefficients unstable. Check the variance inflation factor (VIF); values above 5 to 10 are a common warning sign

Using R² alone to judge model - Plot residuals to check assumptions

Which type of regression should you use?

Choose by the type of outcome you are predicting: linear regression for a continuous number, logistic regression for a yes/no outcome, and polynomial terms or a non-linear model when the residual plot shows a curve.

• Predicting continuous outcome → Linear regression
• Predicting binary outcome (yes/no) → Logistic regression
• Non-linear patterns → Try polynomial terms or other models