I'm trying use f-score from scikit-learn as evaluation metric in xgb classifier. Here is my code:
clf = xgb.XGBClassifier(max_depth=8, learning_rate=0.004,
n_estimators=100,
silent=False, objective='binary:logistic',
nthread=-1, gamma=0,
min_child_weight=1, max_delta_step=0, subsample=0.8,
colsample_bytree=0.6,
base_score=0.5,
seed=0, missing=None)
scores = []
predictions = []
for train, test, ans_train, y_test in zip(trains, tests, ans_trains, ans_tests):
clf.fit(train, ans_train, eval_metric=xgb_f1,
eval_set=[(train, ans_train), (test, y_test)],
early_stopping_rounds=900)
y_pred = clf.predict(test)
predictions.append(y_pred)
scores.append(f1_score(y_test, y_pred))
def xgb_f1(y, t):
t = t.get_label()
return "f1", f1_score(t, y)
But there is an error: Can't handle mix of binary and continuous
The problem is that f1_score
is trying to compare non-binary vs. binary targets and by default this method does binary averaging. From documentation "average : string, [None, ‘binary’ (default), ‘micro’, ‘macro’, ‘samples’, ‘weighted’]".
Anyways, the error is saying that your prediction is continuous like this [0.001, 0.7889,0.33...]
but your target is binary [0,1,0...]
. So if you know your threshold I recommend you to preprocess your result before sending it to the f1_score
function. Usual value of the threshold would be 0.5
.
Tested example of your evaluation function. Does not output error anymore:
def xgb_f1(y, t, threshold=0.5):
t = t.get_label()
y_bin = [1. if y_cont > threshold else 0. for y_cont in y] # binarizing your output
return 'f1',f1_score(t,y_bin)
As suggested by @smci a less_verbose/more_efficient solution could be:
def xgb_f1(y, t, threshold=0.5):
t = t.get_label()
y_bin = (y > threshold).astype(int) # works for both type(y) == <class 'numpy.ndarray'> and type(y) == <class 'pandas.core.series.Series'>
return 'f1',f1_score(t,y_bin)