Logistic Regression & Sigmoid

What it is

Logistic regression is the go-to model for yes/no questions: will this student pass, is this email spam, will this lead convert. It starts with the same linear formula as linear regression, z = weight * x + bias, then passes z through the sigmoid function, sigmoid(z) = 1 / (1 + e^-z), which squashes any number into the range 0 to 1.

That squashed value is read as a probability. If the model outputs 0.83 for a student, it is saying there is an 83 percent chance of a pass. A decision threshold, usually 0.5, turns the probability into a class label.

The sigmoid has a useful shape. Near z = 0 it is steep, so small changes in the score move the probability a lot. Far from 0 it flattens, so once the model is confident, extra evidence barely changes the answer.

Why it's needed

Linear regression can output -40 or 1,000, which makes no sense as a probability. Classification needs an output that stays between 0 and 1 and gets more confident smoothly as the evidence grows. Sigmoid gives exactly that, and it is differentiable, so gradient descent still works.

It also matters because a single artificial neuron is literally logistic regression. Understanding this one model, including why log loss replaces mean squared error, is what makes neural networks feel familiar instead of magical.

How it works

• Compute the linear score: z = (x1 * w1) + (x2 * w2) + ... + bias.
• Squash it: p = sigmoid(z) = 1 / (1 + exp(-z)), a probability between 0 and 1.
• Decide: predict class 1 if p is above the threshold (0.5 by default), otherwise class 0.
• Measure error with log loss (binary cross-entropy): -[y * log(p) + (1 - y) * log(1 - p)], averaged over rows.
• Train with gradient descent. The gradient works out to the same clean form as linear regression: the mean of (p - y) * x.
• Tune the threshold on validation data when false positives and false negatives have different costs.

Code example

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Hours studied -> pass (1) / fail (0)
hours = np.array([0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0])
passed = np.array([0, 0, 0, 0, 1, 0, 1, 1, 1, 1])

weight, bias, learning_rate = 0.0, 0.0, 0.1

for step in range(5000):
    p = sigmoid(weight * hours + bias)
    # Gradient of log loss with respect to weight and bias
    w_grad = np.mean((p - passed) * hours)
    b_grad = np.mean(p - passed)
    weight -= learning_rate * w_grad
    bias -= learning_rate * b_grad

p = sigmoid(weight * hours + bias)
log_loss = -np.mean(passed * np.log(p) + (1 - passed) * np.log(1 - p))
print("weight:", round(weight, 2), "bias:", round(bias, 2), "log loss:", round(log_loss, 3))

for h in (1.0, 2.5, 4.0):
    prob = sigmoid(weight * h + bias)
    print(f"{h} hours -> P(pass) = {prob:.2f} ->", "Pass" if prob > 0.5 else "Fail")

Use cases

• Spam versus not-spam filtering
• Lead scoring: will this website visitor convert
• Pass/fail, approve/reject and churn/no-churn predictions
• The output layer of a binary classification neural network

Common mistakes

• Using mean squared error as the loss, which gives a bumpy surface with flat regions where gradient descent stalls.
• Treating 0.5 as the only sensible threshold when the two kinds of mistakes cost different amounts.
• Reporting accuracy on an imbalanced dataset, where predicting 'no' for everyone can score 95 percent.
• Forgetting that the output is only a calibrated probability if the training data reflects real class frequencies.
• Not scaling features, which lets one large-valued feature dominate and slows convergence.

Interview questions