Decision Trees

What it is

A decision tree predicts by asking a sequence of yes/no questions about the features: is credit score above 650, is income above 40,000, is the loan amount below 20,000. Each question splits the data into two groups, and the questions continue until a group is pure enough, meaning almost all of its rows share one answer. That final group is a leaf, and its majority answer becomes the prediction.

Unlike linear or logistic regression, there are no weights and no gradient descent. The tree is built greedily, top down: at each node it tries every feature and every cut point, measures how much purer the two child groups would be, and keeps the best split.

The result reads like a flowchart, which is why trees are popular when people need to understand or audit a decision.

Why it's needed

Trees handle things that linear models struggle with: non-linear relationships, interactions between features, mixed numeric and categorical data, and no need for feature scaling. They are also fully interpretable, so a loan officer can see exactly why an application was declined.

They also matter as building blocks. Random forests and gradient boosting, which win most tabular-data competitions and power a lot of production scoring systems, are ensembles of decision trees. Understanding one tree is the prerequisite.

How it works

• Start at the root with all rows. For each feature and each candidate threshold, split the rows into two groups.
• Score each split with Gini impurity, 1 - sum(p_class^2), or entropy. A pure group scores 0; a 50/50 group scores the maximum.
• Choose the split with the lowest weighted impurity across the two children, then repeat the process on each child.
• Stop when a group is pure, has too few rows, or the tree reaches max_depth. That node becomes a leaf.
• Predict by walking a new row from the root to a leaf and returning the leaf's majority class, or the mean value for regression.
• Control overfitting with max_depth, min_samples_leaf and min_samples_split, or by pruning after growth.

Code example

import numpy as np
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.model_selection import train_test_split

# Toy loan data: [credit_score, income_k, loan_k] -> approved (1) / declined (0)
rng = np.random.default_rng(1)
X = np.column_stack([rng.integers(500, 800, 300), rng.integers(20, 120, 300), rng.integers(5, 50, 300)])
y = ((X[:, 0] > 640) & (X[:, 2] < X[:, 1] * 0.5)).astype(int)
y[rng.random(300) < 0.08] ^= 1   # flip 8% of labels to simulate real-world noise

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=1)

for depth in (None, 3):
    tree = DecisionTreeClassifier(max_depth=depth, random_state=1).fit(X_train, y_train)
    print(f"max_depth={depth}: train {tree.score(X_train, y_train):.2f}  test {tree.score(X_test, y_test):.2f}")

# The depth-3 tree reads like a set of business rules
print(export_text(tree, feature_names=["credit_score", "income_k", "loan_k"]))

Use cases

• Loan approval and credit risk decisions where every rule must be explainable
• Customer churn or fraud flags on mixed numeric and categorical data
• Quick, scale-free baselines on tabular data
• The base learner inside random forests and gradient-boosted models

Common mistakes

• Growing the tree with no depth limit and reporting a training accuracy of 100 percent.
• Trusting a single tree's feature importances, which shift a lot when the training data changes slightly.
• Expecting a tree to extrapolate; it can only predict values it saw in training leaves.
• One-hot encoding a high-cardinality categorical feature, which spreads the signal across many weak splits.
• Using a tree on a problem that really is linear, where it needs many splits to approximate a simple line.

Interview questions