Decision Tree
Decision Tree
•Decision Tree is a Supervised learning technique that can be used for both classification and Regression
problems, but mostly it is preferred for solving Classification problems. It is a tree-structured classifier, where internal
nodes represent the features of a dataset, branches represent the decision rules and each leaf node
represents the outcome.
•In a Decision tree, there are two nodes, which are the Decision Node and Leaf Node. Decision nodes are used to
make any decision and have multiple branches, whereas Leaf nodes are the output of those decisions and do not
contain any further branches.
•The decisions or the test are performed on the basis of features of the given dataset.
•It is a graphical representation for getting all the possible solutions to a problem/decision based on given
conditions.
•It is called a decision tree because, similar to a tree, it starts with the root node, which expands on further branches
and constructs a tree-like structure.
•In order to build a tree, we use the CART algorithm, which stands for Classification and Regression Tree
algorithm.
•A decision tree simply asks a question, and based on the answer (Yes/No), it further split the tree into subtrees.
•ID3 (Iterative Dichotomiser 3) is a specific algorithm developed to create decision trees. It uses a particular
approach (Information Gain) to determine how to split the data at each node.
•Below diagram explains the general structure of a decision tree:
Structure of a Decision Tree
•Root Node: Root node is from where the decision tree starts. It represents the entire dataset, which further gets
divided into two or more homogeneous sets.
•Leaf Node: Leaf nodes are the final output node, and the tree cannot be segregated further after getting a leaf node.
•Splitting: Splitting is the process of dividing the decision node/root node into sub-nodes according to the given
conditions.
•Branch/Sub Tree: A tree formed by splitting the tree.
•Pruning: Pruning is the process of removing the unwanted branches from the tree.
•Parent/Child node: The root node of the tree is called the parent node, and other nodes are called the child nodes.
While implementing a Decision tree, the main issue arises that how to select the best attribute for the root node and for sub-nodes. So,
to solve such problems there is a technique which is called as Attribute selection measure or ASM. By this measurement, we can
easily select the best attribute for the nodes of the tree. There are two popular techniques for ASM, which are:
•Information Gain
•Gini Index or Gini impurity
1. Information Gain:
•Information gain is the measurement of changes in entropy after the segmentation of a dataset based on an attribute.
•It calculates how much information a feature provides us about a class.
•According to the value of information gain, we split the node and build the decision tree.
•A decision tree algorithm always tries to maximize the value of information gain, and a node/attribute having the highest information
gain is split first. It can be calculated using the below formula:
Entropy: Entropy is a metric to measure the impurity in a given attribute. It specifies randomness in data. Entropy can be calculated
as:
Where,
•S= Total number of samples
•P(yes)= probability of yes
•P(no)= probability of no
Entropy: is the measure of uncertainty of a random variable,
it characterizes the impurity of an arbitrary collection of examples.
The higher the entropy more the information content.
2. Gini Index:
•Gini index is a measure of impurity or purity used while creating a decision tree in the
CART(Classification and Regression Tree) algorithm.
•An attribute with the low Gini index should be preferred as compared to the high Gini index.
•It only creates binary splits, and the CART algorithm uses the Gini index to create binary splits.
•Gini index can be calculated using the below formula:
Decision Tree Algorithm
1. What is the entropy of this collection of training examples with respect to the target function classification?
2. What is the information gain of a1 and a2 relative to these training examples?
3. Draw decision tree for the given dataset.
Advantages of Decision Trees:
•Interpretability: Decision trees are easy to understand and visualize.
•No Need for Feature Scaling: They work well with both numerical and categorical data without the need for
normalization.
•Handle Non-linear Relationships: They can capture complex relationships between features and the target
variable.
Disadvantages of Decision Trees:
•Overfitting: Decision trees can create overly complex models that do not generalize well to new data.
•Instability: Small changes in the data can result in a completely different tree structure.
•Bias towards Dominant Classes: If one class dominates the dataset, the tree may be biased towards that class.
