Loss Functions (MAE, MSE, RMSE)

What it is

A loss function takes every prediction the model made, compares each one with its true label, and reduces all those individual errors to a single number. That number is what training tries to make smaller. Without it there is no definition of 'better', so there is nothing to optimise.

For regression, three loss functions cover most situations. Mean Absolute Error (MAE) is the average of the absolute differences: mean(|actual - predicted|). Mean Squared Error (MSE) squares each difference before averaging: mean((actual - predicted)^2). Root Mean Squared Error (RMSE) is the square root of MSE, which brings the number back into the same units as the label.

If my house-price model is off by 10 thousand on one house and 50 thousand on another, MAE reports 30 thousand. MSE reports (100 + 2500) / 2 = 1300 thousand-squared, which is hard to read, and RMSE reports about 36 thousand. The RMSE is larger than the MAE because the squaring step made the 50 thousand miss count for more.

Why it's needed

The choice of loss function changes what the model learns to care about. Squaring the error means a miss of 50 counts 25 times as much as a miss of 10, so MSE pushes the model hard to avoid large mistakes and will tolerate many small ones to do it. MAE treats every unit of error the same, so it is less distracted by a few outliers.

MSE is the default for training because its gradient is smooth and proportional to the error: a big miss gives a big push, a small miss gives a small push, and the push shrinks to zero as the prediction gets close. MAE's gradient is a constant plus or minus one, which does not slow down near the right answer and is undefined at exactly zero error. In practice I train with MSE and report RMSE or MAE, since those are in units a stakeholder can understand.

How it works

• Compute the error vector once: error = actual - prediction, with one entry per example
• MAE = np.mean(np.abs(error)); every unit of error counts the same, and the result is in the label's units
• MSE = np.mean(error ** 2); large errors are amplified, and the result is in squared units
• RMSE = np.sqrt(MSE); back in the label's units, but still more sensitive to outliers than MAE
• For training, pick the loss whose gradient behaves well (usually MSE); for reporting, pick the metric the business understands (usually MAE or RMSE)
• Track the loss on a validation set as well as the training set; the gap between them is your overfitting signal

Code example

import numpy as np

# True prices (thousands) and two models' predictions for 6 houses
actual = np.array([250, 340, 160, 520, 300, 410], dtype=float)
model_a = np.array([225, 365, 135, 545, 275, 435], dtype=float)  # six even misses of 25
model_b = np.array([250, 340, 160, 400, 300, 410], dtype=float)  # one big miss

def mae(y, y_hat):
    return np.mean(np.abs(y - y_hat))

def mse(y, y_hat):
    return np.mean((y - y_hat) ** 2)

def rmse(y, y_hat):
    return np.sqrt(mse(y, y_hat))

for name, preds in [("model A", model_a), ("model B", model_b)]:
    print(f"{name}: MAE={mae(actual, preds):6.1f}  "
          f"MSE={mse(actual, preds):8.1f}  RMSE={rmse(actual, preds):6.1f}")

# Model B has the lower MAE (one miss of 120 vs six misses of 25),
# but the higher RMSE, because squaring punishes the single big miss.

# Gradient of MSE with respect to the prediction, used by gradient descent:
# d/d(pred) of mean((y - pred)^2) = -2 * mean(y - pred)
error = actual - model_b
print("MSE gradient wrt prediction (model B):", (-2 * np.mean(error)).round(3))

Use cases

• Choosing the evaluation metric to report for a regression model
• Deciding which loss gradient descent should minimise during training
• Comparing model versions across experiments on the same validation set
• Detecting overfitting by watching training loss fall while validation loss rises

Common mistakes

• Reporting MSE to stakeholders; the squared units mean nothing to them, so convert to RMSE or MAE first
• Comparing loss values across datasets with different label scales; a loss of 100 on prices in thousands is not the same as 100 on prices in units
• Using MSE on data with a few extreme outliers and then wondering why the model fits the outliers and ignores everyone else
• Choosing a loss because it is the default rather than because it matches what a bad prediction costs the business
• Evaluating only on the training set; a low training loss with a high validation loss means the model memorised rather than learned

Interview questions