Linear Regression

What it is

Linear regression predicts a continuous number, such as a house price, from one or more input features, by assuming the output is a weighted sum of the inputs plus a constant. With one feature it draws a straight line through the data. With many features it fits a flat surface in higher dimensions, but the idea is the same: each feature gets a weight that says how much it pushes the prediction up or down.

In the house-price example I learned on, the features were square footage and number of bedrooms, and the label was the sale price. The model's whole job was to find two weights and one bias so that weight1 * sqft + weight2 * bedrooms + bias lands as close as possible to the real prices.

What makes it 'linear' is not that the data is a line, but that the prediction is a linear combination of the features. You can feed it squared or logged features and it is still linear regression, because the weights still enter the formula in a straight-line way.

Why it's needed

It is the simplest model that actually learns from data, so it is where nearly every concept in machine learning first shows up: features, weights, bias, a loss function, gradient descent, and the train/test split. If you understand linear regression end to end, neural networks are mostly the same loop with more layers.

It is also still useful on its own. When the relationship is roughly linear and you need a model that is fast, explainable, and easy to debug, linear regression is often the right production choice. A weight of 150 on square footage means each extra square foot adds about 150 to the predicted price, and that kind of readable coefficient is something a stakeholder can check.

How it works

• Prediction: y_hat = (x1 * w1) + (x2 * w2) + ... + bias. With matrices this is X.dot(w) + b for the whole dataset in one step.
• Loss: Mean Squared Error, the average of (actual - predicted)^2 across all rows. Squaring makes large misses hurt far more than small ones.
• Learning: gradient descent nudges each weight in the direction that lowers the loss, repeated for hundreds or thousands of steps.
• Scale features first. If square footage is in the thousands and bedrooms in single digits, the sqft weight dominates the gradient and training crawls or diverges.
• A closed-form alternative, the Normal Equation, solves for the weights directly with linear algebra. scikit-learn's LinearRegression uses a least-squares solver of this kind rather than gradient descent.
• Check the fit on held-out data, not the training rows, and look at the residuals (actual minus predicted) for patterns a straight line is missing.

Code example

import numpy as np

# Toy house data: [square feet, bedrooms] -> price in thousands
X = np.array([[1400, 3], [1600, 3], [1700, 4], [1875, 4], [2100, 5], [2350, 5]], dtype=float)
y = np.array([245, 312, 279, 308, 405, 450], dtype=float)

# Scale features to 0-1 so one weight does not dominate the gradient
X_min, X_max = X.min(axis=0), X.max(axis=0)
X_scaled = (X - X_min) / (X_max - X_min)

weights = np.zeros(X.shape[1])
bias = 0.0
learning_rate = 0.1

for step in range(2000):
    prediction = X_scaled.dot(weights) + bias
    error = y - prediction
    w_grad = -2 * X_scaled.T.dot(error) / len(y)   # dMSE/dw
    b_grad = -2 * error.mean()                      # dMSE/db
    weights -= learning_rate * w_grad
    bias -= learning_rate * b_grad

mse = np.mean((y - (X_scaled.dot(weights) + bias)) ** 2)
print("weights:", weights.round(1), "bias:", round(bias, 1), "MSE:", round(mse, 1))

new_house = (np.array([2000, 4]) - X_min) / (X_max - X_min)
print("predicted price (k):", round(new_house.dot(weights) + bias, 1))

Use cases

• House and rent price estimation from size, location and age
• Sales or demand forecasting when the trend is roughly linear
• Estimating a continuous KPI from campaign spend or usage metrics
• A fast baseline to beat before reaching for a bigger model

Common mistakes

• Skipping feature scaling, then blaming the learning rate when training diverges.
• Reading a weight as 'importance' when the features are on different scales or correlated with each other.
• Fitting a straight line to a curved relationship and trusting a decent-looking training error.
• Forgetting the bias term, which forces the line through the origin and underfits once features are scaled.
• Evaluating on the same rows the model trained on.

Interview questions