ROC curve in R using rpart package?

Question

I split Train data set and Test data set.

I used a package rpart for CART (classification tree) in R (only train set). And I want to carry out a ROC analysis using the ROCR package.

Variable is `n. use' (response varible... 1=yes, 0=no):

> Pred2 = prediction(Pred.cart, Test$n.use)
Error in prediction(Pred.cart, Test$n.use) : 
  **Format of predictions is invalid.**

This is my code. What is problem? And what is right type ("class" or "prob"?

library(rpart)
train.cart = rpart(n.use~., data=Train, method="class")

Pred.cart = predict(train.cart, newdata = Test, type = "class")

Pred2 = prediction(Pred.cart, Test$n.use)
roc.cart = performance(Pred2, "tpr", "fpr")

Answer 1

The prediction() function from the ROCR package expects the predicted "success" probabilities and the observed factor of failures vs. successes. In order to obtain the former you need to apply predict(..., type = "prob") to the rpart object (i.e., not "class"). However, as this returns a matrix of probabilities with one column per response class you need to select the "success" class column.

As your example, unfortunately, is not reproducible I'm using the kyphosis data from the rpart package for illustration:

library("rpart")
data("kyphosis", package = "rpart")
rp <- rpart(Kyphosis ~ ., data = kyphosis)

Then you can apply the prediction() function from ROCR. Here, I'm using the in-sample (training) data but the same can be applied out of sample (test data):

library("ROCR")
pred <- prediction(predict(rp, type = "prob")[, 2], kyphosis$Kyphosis)

And you can visualize the ROC curve:

plot(performance(pred, "tpr", "fpr"))
abline(0, 1, lty = 2)

Or the accuracy across cutoffs:

plot(performance(pred, "acc"))

Or any of the other plots and summaries supported by ROCR.