feature_names must be unique - Xgboost

python pandas xgboost sklearn-pandas
user2728024 picture user2728024 · Apr 24, 2017 · Viewed 10.7k times · Source

I am running the xgboost model for a very sparse matrix.

I am getting this error. ValueError: feature_names must be unique

How can I deal with this?

This is my code. 

  yprob = bst.predict(xgb.DMatrix(test_df))[:,1]

Answer

andrew_reece picture andrew_reece · Apr 24, 2017

According the the xgboost source code documentation, this error only occurs in one place - in a DMatrix internal function. Here's the source code excerpt:

if len(feature_names) != len(set(feature_names)):
    raise ValueError('feature_names must be unique')

So, the error text is pretty literal here; your test_df has at least one duplicate feature/column name.

You've tagged pandas on this post; that suggests test_df is a Pandas DataFrame. In this case, DMatrix literally runs df.columns to extract feature_names. Check your test_df for repeat column names, remove or rename them, and then try DMatrix() again.

Related questions

ValueError: feature_names mismatch: in xgboost in the predict() function

I have trained an XGBoostRegressor model. When I have to use this trained model for predicting for a new input, the predict() function throws a feature_names mismatch error, although the input feature vector has the same structure as the …

Read More
XGBoost plot_importance doesn't show feature names

I'm using XGBoost with Python and have successfully trained a model using the XGBoost train() function called on DMatrix data. The matrix was created from a Pandas dataframe, which has feature names for the columns. Xtrain, Xval, ytrain, yval = train_…

Read More
How to select rows from a DataFrame based on column values

How can I select rows from a DataFrame based on values in some column in Pandas? In SQL, I would use: SELECT * FROM table WHERE colume_name = some_value I tried to look at Pandas' documentation, but I did not …

Read More