Gradient Descent & Learning Rate

What it is

Gradient descent is the algorithm that turns 'the model has an error' into 'the model has slightly better weights'. The gradient of the loss with respect to a weight tells you two things: which direction increases the loss, and how steeply. Gradient descent moves the weight a small step in the opposite direction. Repeat that thousands of times and the weights settle where the loss is lowest.

The size of each step is the learning rate. The update rule is simply new_weight = old_weight - learning_rate * gradient. With a learning rate of 0.01 and a gradient of 5, the weight moves by 0.05. The gradient supplies the direction and relative size; the learning rate scales it.

The image I use is walking downhill in fog. You cannot see the valley floor, but you can feel the slope under your feet. Take a step downhill, feel the slope again, step again. The learning rate is your stride length. Small strides are safe but slow; huge strides can carry you straight past the bottom and up the other side.

Why it's needed

This is the mechanism by which almost every model actually learns. Linear regression, logistic regression and every neural network from a tiny classifier to a large language model are trained by some variant of gradient descent. If you understand this loop you understand what 'training' means, and you can reason about why a model is not converging rather than guessing.

The learning rate is the first hyperparameter to tune when training misbehaves. When I trained my house-price model with unscaled square footage and a learning rate that worked fine for scaled data, the loss exploded to infinity in a few steps. Dropping the learning rate fixed it, but scaling the features was the proper fix because it let one learning rate suit every weight.

How it works

• Start with initial weights and bias: zero for linear models, small random values for neural networks
• Forward pass: compute predictions for the whole batch with X @ weights + bias
• Compute the loss, for example MSE, and its gradient with respect to each weight and the bias
• For MSE the gradients are w_grad = -2 * mean(x * error) per feature and b_grad = -2 * mean(error)
• Update: weights -= learning_rate * w_grad and bias -= learning_rate * b_grad
• Repeat for a fixed number of steps or until the loss stops decreasing; record the loss each step so you can see whether it is falling smoothly, crawling or blowing up
• If the loss grows, cut the learning rate; if it falls very slowly, raise it or scale your features

Code example

import numpy as np

# One feature (sqft, already scaled to 0..1) and the price label (thousands)
x = np.array([0.0, 0.25, 0.5, 0.75, 1.0])
y = np.array([150, 220, 290, 360, 430], dtype=float)  # true line: 280x + 150

weight, bias = 0.0, 0.0
learning_rate = 0.05

for step in range(500):
    prediction = weight * x + bias
    error = y - prediction

    # Gradients of MSE with respect to weight and bias
    w_grad = -2 * np.mean(x * error)
    b_grad = -2 * np.mean(error)

    # Step against the gradient, scaled by the learning rate
    weight -= learning_rate * w_grad
    bias -= learning_rate * b_grad

    if step % 100 == 0:
        mse = np.mean(error ** 2)
        print(f"step {step:4d}  weight={weight:7.2f}  bias={bias:7.2f}  mse={mse:9.2f}")

print(f"learned: price = {weight:.1f} * sqft_scaled + {bias:.1f}")

# Try learning_rate = 2.0 to watch the loss diverge, or 0.005 to watch it crawl.

Use cases

• Training every neural network, from a two-layer classifier to a large language model
• Fitting linear and logistic regression from scratch to understand what library functions do internally
• Any optimisation problem with a differentiable loss, including tuning parameters in simulations and physics models
• Debugging training runs: the shape of the loss curve tells you whether the learning rate is right

Common mistakes

• Learning rate too high: the loss oscillates or shoots to infinity within a few steps, often showing up as NaN
• Learning rate too low: the loss decreases so slowly that you stop training long before the weights are good, and conclude the model does not work
• Running gradient descent on unscaled features, so one weight needs a tiny learning rate while another needs a large one, and no single value works
• Getting the sign wrong in the update and adding the gradient instead of subtracting it, which climbs the hill instead of descending
• Judging convergence from the training loss alone; it can keep falling while the validation loss rises, which means overfitting rather than progress

Interview questions