Vectors, Matrices & Matrix Multiplication

What it is

A vector is a list of numbers. In machine learning one vector usually describes one example: a house becomes [3, 1200, 20] for bedrooms, square feet and age. A matrix is a grid of numbers, which in practice means a stack of vectors: the whole dataset, with one row per example and one column per feature. A matrix with 1000 houses and 3 features has shape 1000 x 3.

Matrix multiplication is the operation that turns a dataset and a set of weights into predictions in one step. If X is the 1000 x 3 feature matrix and w is a vector of 3 weights, then X @ w is a vector of 1000 predictions. Each row of X is multiplied element by element with w and summed, which is exactly the weighted sum from the features and weights topic, applied to every house at once.

The rule that trips people up is the shape rule. To multiply A by B, the number of columns in A must equal the number of rows in B. A (m x n) times B (n x p) gives a result of shape (m x p). The inner dimensions must match and disappear; the outer dimensions survive.

Why it's needed

Computers do not understand houses or sentences, only numbers, so every input to a model is converted into a vector first. Once data is a matrix, the same code handles 10 examples or 10 million. More importantly, matrix operations are what GPUs are built to run. A forward pass through a neural network is a chain of matrix multiplications, and that is the reason training can scale.

When I first wrote a prediction loop in plain Python, it ran row by row and took seconds on a few thousand houses. Replacing the loop with a single X @ w call made it effectively instant. That difference grows with data size, and it is the difference between code that works on a laptop and code that works in production.

How it works

• Represent one example as a vector of its feature values, and the dataset as a matrix with one row per example
• Check shapes before multiplying: for A @ B, A.shape[1] must equal B.shape[0]
• The result of (m x n) @ (n x p) has shape (m x p): the inner dimension n is consumed
• X @ weights computes a weighted sum for every row at once, which is a full batch of predictions in one line
• Use the transpose (X.T) when the shapes do not line up; this is common when computing gradients
• Prefer NumPy array operations over Python for-loops; this is called vectorisation and is the single biggest speed-up in numerical code

Code example

import numpy as np

# Dataset: 4 houses x 3 features (bedrooms, sqft, age)
X = np.array([
    [3, 1200, 20],
    [4, 1500, 5],
    [2, 800, 35],
    [5, 2200, 2],
], dtype=float)
weights = np.array([20.0, 0.15, -1.0])

print("X shape:", X.shape)             # (4, 3)
print("weights shape:", weights.shape) # (3,)

# Slow way: a Python loop, one weighted sum per house
slow = [sum(value * w for value, w in zip(row, weights)) for row in X]

# Fast way: one matrix-vector product. Shapes (4,3) @ (3,) -> (4,)
fast = X @ weights

print("loop result:  ", np.round(slow, 1))
print("matrix result:", np.round(fast, 1))
print("identical:", np.allclose(slow, fast))

# Shape rule in action: this would fail because 3 != 4
try:
    weights @ X
except ValueError as e:
    print("shape error:", e)

Use cases

• Batch prediction across an entire dataset in one operation
• Forward passes in neural networks, where each layer is a matrix multiplication followed by an activation
• Computing similarity between an embedding and thousands of stored embeddings at once
• Any numerical computation you want to run without Python loops

Common mistakes

• Mixing up @ (matrix multiplication) with * (element-wise multiplication); NumPy will happily do the wrong one if the shapes broadcast
• Forgetting that a 1-D array of shape (3,) is not the same as a column vector of shape (3, 1); many shape bugs come from this
• Multiplying in the wrong order: A @ B and B @ A are different operations and usually only one of them is even valid
• Writing a Python loop over rows when a single vectorised expression would do; it is correct but can be 100x slower
• Assuming a shape error means a bug in the maths when it is usually a transpose that is missing

Interview questions