What is a decision tree?

A decision tree is a unique kind of probability tree. A popular and powerful tool used for prediction and classification. The decision tree structure is the same as a flowchart with a tree structure. The internal node refers to the attribute test, the branch refers to the outcome of the test, and the leaf node(terminal) contains the class label.

A decision tree is an illustration of the decision-making process. In artificial intelligence (AI), the decision trees are used to decide the conclusion based on data available from past decisions. However, the conclusions are reserved values that are deployed to forecast the action(taken in near feature).

Supervised learning

A decision tree is an algorithmic and statistical model of machine learning, which learns and interprets the response from several problems and their consequences. However, the decision tree knows the decision-making rules in certain contexts depending upon the data availability. In a decision tree, the learning process is continuous and feedbacks are used to improve the learning outcome. Thus, this type of learning is known as supervised learning and the models in the decision tree support supervised learning.

Terminologies of decision tree

The important terminologies used in the decision tree are,

• Root node represents the entire sample or population and this node is further divided into two or multiple homogeneous sets.
• Decision node is represented in a square shape. Decision node splits the sub-node into additional sub-nodes.
• Chance node is represented in circle shape and shows the probablities of vertain results
• End node is represented in a triangle shape and shows the final output of the decision path.
• Splitting is the process that divides the node into more than one sub-node.
• Pruning is the process of removing a sub-node from the decision tree.
• Branch tree: The entire tree’s subsection is called a sub-tree or branch tree.

Symbols in decision tree

Types of decision tree

The decision tree is broadly classified into two kinds,

Classification tree

The classification tree analysis is used, when the outcome of the predicate is a class (where it has the data). In other words, a dataset is classified into available datasets or classes. Example: Examining a Facebook comment and classifying the text as either positive or negative.

Regression tree

The regression tree analysis is used, when the outcome of the predicate is a real number. In other words, the prediction depends on either single or multiple predictors. Example: Length of patient’s stay at the hospital.

Classification and regression tree(CART) analysis is a kind of umbrella term, which refers to the above procedures. These procedures are introduced in the year 1984. Both the trees have certain similarities, but the major difference is the procedure to discover the split.

Metrics

The algorithm to establish a decision tree will work in a top-down manner. It determines the variable at each step to splits the item set. To calculate the best, different metrics are used by different algorithms and some of the metrics are,

Gini impurity

Measures the count of randomly selected items from a set, which could be labeled incorrectly (randomly named depending on the distribution label at the subset). CART algorithm uses Gini impurity for a classification tree. The Gini impurity of the data set is defined as,

$Gini = 1 - \sum _{i=1}^n {\left(p_i\right)}^2$

Entropy

The measure of randomness in the data being processed. Higher entropy, harder to attain the conclusion. Mathematically entropy for a single attribute is,

$$\begin{gathered} E\left(S\right) =\underset{i=1}{\overset{c}{\sum -p_i {\log }_2 p_i}}. \\ Where S – current state, Pi – probability of the event\left(i\right) of state\left(S\right) or percentage of the class\left(i\right) in state node\left(S\right). \end{gathered}$$ 

Mathematically entropy for a single attribute is,

$$\begin{gathered} E(T,X) = \underset{c\in X}{\overset{c}{\sum P\left(c\right) E\left(c\right)}} \\ Where T- current state and X- selected attribute \end{gathered}$$

Information gain

The tree algorithm like ID3, C5.0, and C4.5 use information gain(IG). It is based on the concept of information content and entropy from the information theory. This is used to find the feature to splits at every step at tree construction. The mathematical representation of IG is,

$$\begin{gathered} Information Gain = Entropy\left(before\right) – \sum _{j=1}^k Entropy(j, after) \\ Where before – dataset before the split, k – subsets generated by u\sin g spilt, and (j,after) – subset j after the split. \end{gathered}$$

Variance reduction

The reduction variance is used, when the decision tree works under regression; however, the output will be continuous. The algorithm uses a variance formula to split the population,

$$\begin{gathered} Variance = {\frac{\sum _{}^{} (X- \overline{X})}{n}}^2 \\ Where \overline{X}- mean of the value, \end{gathered}$$

X- actual value,

n- number of value.

Decision tree algorithm

The decision tree algorithm comes under the supervised learning algorithm. This algorithm solves regression problems and classification problems. The decision tree aims to develop a training model, which predicts the target variable value and class with decision rules from the training data (prior data). For the record, a class label is predicted from the tree root and compares the root attribute value with its record attribute. This could be a basic comparison as it follows the branch values and jumps to its next node. The following are the algorithms used in creating a decision tree,

• Iterative Dichotomiser 3(ID3).
• C4.5( ID3 Successor).
• CART.
• Multivariate Adaptive Regression Splines(MARS).
• Chi-square Automatic Interaction Detector(CHAID).

Example

The following example explains the options for mobile phone production. Each of the units has high and low-profit margins. In the end, it contains terminator nodes with their results. On the basis of that, Technology A has been chosen while Technology B has been rejected.

Context and Applications

This topic is important for postgraduate and undergraduate courses, particularly for,

• Bachelors in computer science engineering.
• Associate of science in computer science.

Practice Problems

Question 1: ____ is used to predict and classify data.

a) Flowchart

b) Decision tree

c) B+ tree

d) Regression tree

Answer: Option is b correct.

Explanation: A decision tree is a unique kind of probability tree, which is both a popular and powerful tool used for prediction and classification. The internal node refers to the attribute test, the branch refers to the test outcome, and the leaf node contains the class label.

Question 2: How many types of decision trees are there?

a) 5

b) 3

c) 2

d) 4

Answer: Option c is correct.

Explanation: A decision tree is a unique kind of probability tree, which is categorized into two kinds namely, regression and classification tree. Both trees have certain similarities, but the major difference is the procedure to discover the split.

Question 3: The subsection of a whole tree is called ___.

a) Branch tree

b) Internal node

c) Training data

d) Regression tree

Answer: Option a is correct.

Explanation: Subtree is the subsection of a whole tree; branch tree is the other name of the subtree. This is one of the important terminology used in the decision trees, and the others are root node, decision node, splitting, end node, change node, and pruning.

Question 4: Select the metric used in the construction of the decision tree.

a) AdaBoost

b) Gini impurity

c) Linear regression

d) None of the above

Answer: Option b is correct.

Explanation: Gini impurity is a metric of the decision tree. The decision tree supports several metrics like Information gain, chi-square, variance reduction, gain ratio, entropy, and Gini impurity. To measure the best, various metrics are used by various algorithms.

Question 5: The end node is represented in _____ shape.

a) Triangle

b) Circle

c) Rectangle

d) None of the above

Answer: Option a is correct.

Explanation: The decision tree uses different shapes to represent different nodes. The end node is represented in a triangle shape and shows the final output of the decision path.