What is a weak learner?

machine-learning ensemble-learning
Stu picture Stu · Dec 7, 2013 · Viewed 15.1k times · Source

I want to compare different error rates of different classifiers with the error rate from a weak learner (better than random guessing). So, my question is, what are a few choices for a simple, easy to process weak learner? Or, do I understand the concept incorrectly, and is a weak learner simply any benchmark that I choose (for example, a linear regression)?

Answer

Raff.Edward picture Raff.Edward · Dec 7, 2013

better than random guessing

That is basically the only requirement for a weak learner. So long as you can consistently beat random guessing, any true boosting algorithm will be able to increase the accuracy of the final ensemble. What weak learner you should choose is then a trade off between 3 factors:

  1. The bias of the model. A lower bias is almost always better, but you don't want to pick something that will overfit (yes, boosting can and does overfit)
  2. The training time for the weak learner. Generally we want to be able to learn a weak learner quickly, as we are going to be building a few hundred (or thousand) of them.
  3. The prediction time for our weak learner. If we use a model that has a slow prediction rate, our ensemble of them is going to be a few hundred times slower!

The classic weak learner is a decision tree. By changing the maximum depth of the tree, you can control all 3 factors. This makes them incredibly popular for boosting. What you should be using depends on your individual problem, but decision trees is a good starting point.

NOTE: So long as the algorithm supports weighted data instances, any algorithm can be used for boosting. A guest speaker at my University was boosting 5 layer deep neural networks for his work in computational biology.

Related questions

How to handle categorical variables in sklearn GradientBoostingClassifier?

I am attempting to train models with GradientBoostingClassifier using categorical variables. The following is a primitive code sample, just for trying to input categorical variables into GradientBoostingClassifier. from sklearn import datasets from sklearn.ensemble import GradientBoostingClassifier import pandas iris = datasets.…

Read More
sklearn.ensemble.AdaBoostClassifier cannot accecpt SVM as base_estimator?

I am doing a text classification task. Now I want to use ensemble.AdaBoostClassifier with LinearSVC as base_estimator. However, when I try to run the code clf = AdaBoostClassifier(svm.LinearSVC(),n_estimators=50, learning_rate=1.0, algorithm='SAMME.R') clf.fit(…

Read More
How to use adaboost with different base estimator in scikit-learn?

I want to use adaboost with several base estimators for regression in scikit-learning, but I don't find any class that can do it. Is there any way to do this things except changing source code?

Read More