Normalization / Feature Scaling

What it is

Normalisation, or feature scaling, means transforming every feature so they all live on a similar range. The most common version is min-max scaling, which maps each column into 0 to 1 with the formula scaled = (value - min) / (max - min). The smallest house in the dataset becomes 0, the largest becomes 1, and everything else lands in between.

The other common version is standardisation, which subtracts the column mean and divides by the standard deviation. The result has a mean of 0 and a standard deviation of 1, and values are not confined to a fixed range. Both achieve the same goal: no feature dominates just because it happens to be measured in bigger units.

In my house-price data, square footage ranged from 800 to 2200 while bedrooms ranged from 2 to 5. Without scaling, a change of one unit in square footage is tiny while a change of one bedroom is huge, and the model has to compensate with wildly different weights.

Why it's needed

Gradient descent takes steps of the same size in every direction. If one feature spans thousands of units and another spans single digits, the loss surface is stretched into a long narrow valley. The optimiser bounces between the steep walls instead of moving along the floor, and training is slow or unstable. After scaling, the valley becomes round and the same learning rate works for every weight.

Scaling also matters for anything distance based. k-nearest neighbours, k-means clustering and cosine similarity on raw features all let the largest-scale feature decide the answer. Tree-based models are the main exception: a decision tree only cares about the ordering of values, so scaling does not change its splits.

How it works

• Compute the statistics per column, not over the whole matrix: X.min(axis=0) and X.max(axis=0) give one value per feature
• Min-max: X_scaled = (X - min) / (max - min), giving a 0 to 1 range per feature
• Standardisation: X_scaled = (X - mean) / std, giving mean 0 and standard deviation 1 per feature
• Fit the statistics on the training set only, then apply the same numbers to validation, test and live data
• Store the min/max or mean/std alongside the model; at prediction time a new house must be scaled with the training statistics, never with its own
• Scale features only, not the label, unless the label also has a huge range and you convert predictions back afterwards

Code example

import numpy as np

# Training data: [bedrooms, sqft, age] for 5 houses
X_train = np.array([
    [3, 1200, 20],
    [4, 1500, 5],
    [2, 800, 35],
    [5, 2200, 2],
    [3, 1000, 12],
], dtype=float)

# Statistics come from the TRAINING set only, one value per column
col_min = X_train.min(axis=0)
col_max = X_train.max(axis=0)

def min_max_scale(X):
    # Reuses the training min/max so every dataset lands on the same scale
    return (X - col_min) / (col_max - col_min)

X_train_scaled = min_max_scale(X_train)
print("scaled training data:\n", X_train_scaled.round(2))

# A new house at prediction time. Its sqft (2500) is beyond the training max,
# so its scaled value goes above 1. That is correct: we must NOT refit.
X_new = np.array([[4, 2500, 8]], dtype=float)
print("new house scaled:", min_max_scale(X_new).round(2))
# Standardisation alternative: mean 0, std 1 per column
col_mean, col_std = X_train.mean(axis=0), X_train.std(axis=0)
X_std = (X_train - col_mean) / col_std
print("standardised means:", X_std.mean(axis=0).round(6), "stds:", X_std.std(axis=0).round(6))

Use cases

• Any gradient-descent based model with more than one feature, including linear and logistic regression and neural networks
• Distance-based algorithms such as k-nearest neighbours and k-means, where an unscaled feature would dominate
• Preprocessing pipelines where training statistics are saved and reused at inference time
• Neural network inputs such as pixel values, which are routinely divided by 255 to land in 0 to 1

Common mistakes

• Recomputing min and max on the test set or on a single new example; the scale then differs from training and every prediction is wrong
• Scaling before the train/test split, which leaks information about the test set into the training statistics
• Dividing by zero when a column is constant (max equals min); check for it or add a small epsilon
• Forgetting to save the scaling parameters with the model, so the API serving it cannot reproduce the transformation
• Scaling one-hot encoded 0/1 columns with standardisation, which turns clean binary flags into odd fractional values for no benefit

Interview questions