how to work with the catboost overfitting detector

Question

I am trying to understand the catboost overfitting detector. It is described here:

https://tech.yandex.com/catboost/doc/dg/concepts/overfitting-detector-docpage/#overfitting-detector

Other gradient boosting packages like lightgbm and xgboost use a parameter called early_stopping_rounds, which is easy to understand (it stops the training once the validation error hasn't decreased in early_stopping_round steps).

However I have a hard time understanding the p_value approach used by catboost. Can anyone explain how this overfitting detector works and when it stops the training?

Answer 1

It's not documented on the Yandex website or at the github repository, but if you look carefully through the python code posted to github (specifically here), you will see that the overfitting detector is activated by setting "od_type" in the parameters. Reviewing the recent commits on github, the catboost developers also recently implemented a tool similar to the "early_stopping_rounds" parameter used by lightGBM and xgboost, called "Iter." To set the number of rounds after the most recent best iteration to wait before stopping, provide a numeric value in the "od_wait" parameter.

For example:

fit_param <- list(
  iterations = 500,
  thread_count = 10,
  loss_function = "Logloss",
  depth = 6,
  learning_rate = 0.03,
  od_type = "Iter",
  od_wait = 100
)

I am using the catboost library with R 3.4.1. I have found that setting the "od_type" and "od_wait" parameters in the fit_param list works well for my purposes.

I realize this is not answering your question about the way to use the p_value approach also implemented by the catboost developers; unfortunately I cannot help you there. Hopefully someone else can explain that setting to the both of us.