What are Decision Trees?

A Decision Tree is a machine learning algorithm that makes decisions by asking a series of simple questions. These questions are arranged in a tree-like structure, where each internal node represents a decision rule, each branch represents an outcome, and each leaf node represents the final prediction.

In simple words, a decision tree keeps splitting data into smaller groups until it can make a prediction.

It is one of the most practical and easy-to-understand methods in supervised learning, especially for classification and regression tasks.

‍

How a Decision Tree works

A decision tree learns from training data by finding the best rule to split the data step by step.

The basic process looks like this:

• Start with the full dataset
• Try different decision rules
• Measure how good each split is
• Choose the best split
• Split the data into child groups
• Repeat the process on each child group
• Stop when the groups are pure enough or no useful split remains

This repeated splitting is what builds the “tree.”

‍

Step 1: Explore the data

Before building a decision tree, we first inspect the dataset.

In a simple example, the dataset may have:

• Features such as size and color
• Target class such as circle or triangle

The model looks at these features and tries to learn which rules separate the classes best.

For example:

• Size = 1, Color = blue → Circle
• Size = 2, Color = red → Triangle
• Size = 1.5, Color = blue → Circle

This is how the model starts understanding patterns in the data.

‍

‍

Step 2: Measure impurity

A decision tree needs to know whether a group of data is “mixed” or “pure.”

A node is called pure when all records in that node belong to the same class.

For example:

• If all items in a node are circles, the node is pure
• If the node contains both circles and triangles, it is impure

One common way to measure this is Gini Impurity.

Simple idea of Gini Impurity

• Gini = 0 → the node is completely pure
• Higher Gini → the node is more mixed

So the goal of the decision tree is to create splits that reduce impurity as much as possible.

‍

‍

Step 3: Calculate information gain

Once impurity is measured, the model tries different possible rules.

For example:

• Is size ≥ 2?
• Is color blue?

Each rule splits the parent group into two child groups.

The model then checks how much impurity is reduced after the split. This improvement is called information gain.

In simple words:

• A good split makes the child groups more pure
• A bad split does not improve purity much

The best split is the one with the highest information gain

‍

‍

Step 4: Choose the best decision rule

A decision tree does not choose a split randomly.

It checks multiple possible decision rules and compares them.

For example, if the model finds:

• If size ≥ 2 → gain = 0.08
• If color is blue → gain = 0.04

Then the model will choose:

• If size ≥ 2

because it gives higher information gain.

This becomes the rule at the current node.

‍

‍

Step 5: Recursive binary splitting

After the best split is chosen, the data is divided into child nodes.

Then the same process is repeated again for each child node.

This is called recursive binary splitting.

The model keeps asking:

• Can this node be split again?
• Is there another rule that improves purity?
• Should I stop here?

The splitting usually stops when:

• The node becomes pure
• There are no useful decision rules left
• The tree reaches a stopping limit

This recursive process is what grows the tree deeper.

‍

‍

Step 6: Make predictions

The final nodes of a decision tree are called leaf nodes.

A leaf node does not split further. It gives the final output.

For classification, the leaf node predicts the class.

For example:

• This leaf predicts Circle
• That leaf predicts Triangle

Sometimes the leaf can also give prediction probabilities, such as:

• p(circle) = 1.0
• p(circle) = 0.5, p(triangle) = 0.5

So when new data comes in, the model simply moves through the decision rules until it reaches a leaf node.

That is the prediction.

‍

‍

‍

Limitations of Decision Trees

Even though they are useful, decision trees also have some weaknesses.

Some common limitations are:

• They can overfit the training data
• Small changes in data can produce a different tree
• Deep trees can become too complex
• They may not always give the best accuracy alone

Because of this, decision trees are often used as the foundation for stronger models like Random Forest and Gradient Boosted Trees.

‍

‍

Decision Trees in real life

Decision trees are used in many real-world applications, such as:

• Loan approval
• Medical diagnosis
• Customer churn prediction
• Fraud detection
• Spam filtering
• Product recommendation
• Risk analysis

Anywhere a system must make decisions based on a series of conditions, a decision tree can be useful.

‍

Simple Python example

Here is a very basic example using scikit-learn:

from sklearn import tree

X = [[0, 0], [1, 1]]
y = [0, 1]

clf = tree.DecisionTreeClassifier()
clf = clf.fit(X, y)

prediction = clf.predict([[2., 2.]])
probability = clf.predict_proba([[2., 2.]])

This example trains a simple decision tree and then uses it to make a prediction.

‍

Why AI learners should understand Decision Trees

Decision trees are one of the best beginner-friendly machine learning algorithms.

They help you understand key ideas like:

• Features
• Classes
• Decision rules
• Splitting
• Impurity
• Information gain
• Prediction

These concepts are useful even when you move on to more advanced algorithms.

If you are building your machine learning foundation, decision trees are a great place to start. You may also like What Should You Look for in an Online Machine Learning Course? and Learn Machine Learning Without Quitting Your Job.

Final thoughts

A decision tree is a model that learns by asking a sequence of questions.

It starts with the full dataset, finds the best rule, splits the data, and repeats the process until it can make a prediction.

In simple words:

• It organizes decision rules in a tree structure
• It uses impurity and information gain to choose splits
• It keeps splitting recursively
• It predicts using leaf nodes

That is what makes decision trees simple, visual, and powerful.

At AI Folks, we help learners understand practical AI and machine learning concepts in a simple way.

Join the AI Folks community to learn AI engineering step by step.

‍