How to generate a custom cross-validation generator in scikit-learn?

Question

I have an unbalanced dataset, so I have an strategy for oversampling that I only apply during training of my data. I'd like to use classes of scikit-learn like GridSearchCV or cross_val_score to explore or cross validate some parameters on my estimator(e.g. SVC). However I see that you either pass the number of cv folds or an standard cross validation generator.

I'd like to create a custom cv generator so I get and Stratified 5 fold and oversample only my training data(4 folds) and let scikit-learn look through the grid of parameters of my estimator and score using the remaining fold for validation.

Thanks in advance.

Answer 1

The cross-validation generator returns an iterable of length n_folds, each element of which is a 2-tuple of numpy 1-d arrays (train_index, test_index) containing the indices of the test and training sets for that cross-validation run.

So for 10-fold cross-validation, your custom cross-validation generator needs to contain 10 elements, each of which contains a tuple with two elements:

• An array of the indices for the training subset for that run, covering 90% of your data
• An array of the indices for the testing subset for that run, covering 10% of the data

I was working on a similar problem in which I created integer labels for the different folds of my data. My dataset is stored in a Pandas dataframe myDf which has the column cvLabel for the cross-validation labels. I construct the custom cross-validation generator myCViterator as follows:

myCViterator = []
for i in range(nFolds):
    trainIndices = myDf[ myDf['cvLabel']!=i ].index.values.astype(int)
    testIndices =  myDf[ myDf['cvLabel']==i ].index.values.astype(int)
    myCViterator.append( (trainIndices, testIndices) )