ROCR error: Format of predictions is invalid

Question

After getting my predictions from glmnet, I am trying to use "prediction" function, in "ROCR" package, to get tpr, fpr, etc but get this error:

pred <- prediction(pred_glmnet_s5_3class, y)
Error in prediction(pred_glmnet_s5_3class, y) : 
Format of predictions is invalid.

I have output both glmnet predictions and labels and they look like they are in similar format and hence I don't understand what is invalid here.

The code is as follows and input can be found here input. It is a small dataset and should not take much time to run.

library("ROCR")
library("caret")
sensor6data_s5_3class <- read.csv("/home/sensei/clustering /sensor6data_f21_s5_with3Labels.csv")
sensor6data_s5_3class <- within(sensor6data_s5_3class, Class <- as.factor(Class))
sensor6data_s5_3class$Class2 <- relevel(sensor6data_s5_3class$Class,ref="1")

set.seed("4321")
inTrain_s5_3class <- createDataPartition(y = sensor6data_s5_3class$Class, p = .75, list = FALSE)
training_s5_3class <- sensor6data_s5_3class[inTrain_s5_3class,]
testing_s5_3class <- sensor6data_s5_3class[-inTrain_s5_3class,] 
y <- testing_s5_3class[,22]

ctrl_s5_3class <- trainControl(method = "repeatedcv", number = 10, repeats = 10 , savePredictions = TRUE)
model_train_glmnet_s5_3class <- train(Class2 ~ ZCR + Energy + SpectralC + SpectralS + SpectralE + SpectralF + SpectralR + MFCC1 + MFCC2 + MFCC3 + MFCC4 + MFCC5 + MFCC6 + MFCC7 + MFCC8 + MFCC9 + MFCC10 + MFCC11 + MFCC12 + MFCC13, data = training_s5_3class, method="glmnet", trControl = ctrl_s5_3class)
pred_glmnet_s5_3class = predict(model_train_glmnet_s5_3class, newdata=testing_s5_3class, s = "model_train_glmnet_s5_3class$finalModel$lambdaOpt")

pred <- prediction(pred_glmnet_s5_3class, y)

Appreciate your help!

Answer 1

The main problem is that prediction takes "a vector, matrix, list, or data frame" for both predictions and labels arguments. Even though pred_glmnet_s5_3class and y look like vectors, they are not, e.g.

sapply(c(is.vector, is.matrix, is.list, is.data.frame), do.call, list(y))
# [1] FALSE FALSE FALSE FALSE

In fact, they are factors (which can be seen from e.g. class(y)), and ?is.vector informs us to

Note that factors are not vectors; ‘is.vector’ returns ‘FALSE’ and ‘as.vector’ converts a factor to a character vector for ‘mode = "any"’.

We can convert both objects to numeric:

pred <- prediction(as.numeric(pred_glmnet_s5_3class), as.numeric(y))
#   Number of classes is not equal to 2.
# ROCR currently supports only evaluation of binary classification tasks.

Unfortunately, it produces a different problem which is beyond the scope of this question.