Decision Tree Learning and Impurity

machine-learning data-mining random-forest decision-tree
Jony picture Jony · Feb 8, 2011 · Viewed 7k times · Source

There are three ways to measure impurity:

Entropy

Gini Index

Classification Error

What are the differences and appropriate use cases for each method?

Answer

David Weiser picture David Weiser · Feb 8, 2011

If the p_i's are very small, then doing multiplication on very small numbers (Gini index) can lead to rounding error. Because of that, it is better to add the logs (Entropy). Classification error, following your definition, provides a gross estimate since it uses the single largest p_i to compute its value.

Related questions

What is the difference between linear regression and logistic regression?

When we have to predict the value of a categorical (or discrete) outcome we use logistic regression. I believe we use linear regression to also predict the value of an outcome given the input values. Then, what is the difference …

Read More
Difference between classification and clustering in data mining?

Can someone explain what the difference is between classification and clustering in data mining? If you can, please give examples of both to understand the main idea.

Read More
Calculate AUC in R?

Given a vector of scores and a vector of actual class labels, how do you calculate a single-number AUC metric for a binary classifier in the R language or in simple English? Page 9 of "AUC: a Better Measure..." seems to …

Read More