Neural Networks & Backpropagation

What it is

A single neuron computes sigmoid(weight * input + bias), which is exactly logistic regression. A neural network wires many of these neurons into layers: an input layer that holds the features, one or more hidden layers that transform them, and an output layer that produces the prediction. Each neuron in a layer connects to every neuron in the next, and every connection has its own weight.

The power comes from stacking. One neuron can only draw a straight boundary. A hidden layer lets the network bend that boundary, so it can solve problems like XOR where no single line separates the classes. Deeper networks compose simple curves into very complex functions, which is what image recognition and language models rely on.

Backpropagation is the algorithm that trains it. It takes the error at the output and pushes it backwards through the layers, using the chain rule, so each weight learns how much it contributed to the mistake.

Why it's needed

Most real problems are not linear. Pixel values do not add up to 'cat', and the meaning of a sentence is not a weighted sum of its words. Neural networks are the general tool for learning those relationships from data, and every modern deep learning system, including large language models, is built from the same forward-pass, loss, backward-pass loop.

Understanding backprop by hand also demystifies the frameworks. When PyTorch shows a vanishing gradient or a loss that explodes, you know which part of this loop went wrong.

How it works

• Forward pass: hidden = sigmoid(X.dot(W1) + b1), output = sigmoid(hidden.dot(W2) + b2). Data flows layer by layer to a prediction.
• Loss: compare the output with the target, for example mean squared error or cross-entropy.
• Backward pass: compute the output error, multiply by the sigmoid derivative to get the gradient at the output layer, then send it back through W2 to get the hidden layer's gradient.
• Update: every weight moves a small step against its gradient, scaled by the learning rate. Repeat for thousands of epochs.
• Initialise weights randomly, never all zero, so each hidden neuron starts different and can learn a different feature.
• Activation functions provide the non-linearity. Without them, stacked layers collapse into one big linear model.

Code example

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# XOR: no single straight line separates these two classes
X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([[0], [1], [1], [0]])

rng = np.random.default_rng(42)
W1, b1 = rng.normal(size=(2, 4)), np.zeros((1, 4))   # 2 inputs -> 4 hidden neurons
W2, b2 = rng.normal(size=(4, 1)), np.zeros((1, 1))   # 4 hidden -> 1 output
learning_rate = 0.5

for epoch in range(10000):
    # Forward pass
    hidden = sigmoid(X.dot(W1) + b1)
    output = sigmoid(hidden.dot(W2) + b2)
    # Backward pass: output error, then push it back through W2
    d_output = (output - y) * output * (1 - output)
    d_hidden = d_output.dot(W2.T) * hidden * (1 - hidden)
    # Update every weight against its gradient
    W2 -= learning_rate * hidden.T.dot(d_output)
    b2 -= learning_rate * d_output.sum(axis=0)
    W1 -= learning_rate * X.T.dot(d_hidden)
    b1 -= learning_rate * d_hidden.sum(axis=0)

print("XOR predictions:", output.round(2).ravel())   # close to [0, 1, 1, 0]

Use cases

• Image and speech recognition
• Any problem with complex, non-linear relationships, XOR being the textbook example
• Tabular problems where feature interactions matter and simpler models plateau
• The foundation architecture behind deep learning and large language models

Common mistakes

• Initialising all weights to zero, so every hidden neuron computes the same thing forever.
• Stacking layers with no activation function and wondering why the network cannot learn XOR.
• Using sigmoid in deep networks, where its tiny derivative makes gradients vanish; ReLU is the usual fix.
• A learning rate that is too high, which makes the loss bounce around or blow up to NaN.
• Not scaling inputs, so large raw values saturate the sigmoid and gradients drop to zero.

Interview questions