Predict function error for probabilities in glmnet?

Question

I am trying to predict probabilities in a dataset using glmnet. My code reads:

bank <- read.table("http://www.stat.columbia.edu/~madigan/W2025/data/BankSortedMissing.TXT",header=TRUE)
bank$rich<-sample(c(0:1), 233, replace=TRUE)
    train=bank[1:200,];
    test=bank[201:233,]
    x=model.matrix(rich~., bank)[,-1]
    cv.out=cv.glmnet(x, train$rich, alpha=0, family="binomial")
ridge.mod=glmnet(x, train$rich, alpha=0, family="binomial")
    bank$rich <- NULL
newx = data.matrix(test$rich)
ridge.pred=predict(ridge.mod,newx=newx)

train = data[1:2500,];
test = data[2501:5088,];
x=model.matrix(Y~x1+x2+x3+x4+x5+x6, data)[,-1]
cv.out=cv.glmnet(x, data$Y, alpha=0, family="binomial")
    bestlam=cv.out$lambda.min
ridge.mod=glmnet(x, data$Y, alpha=0, family="binomial")
    test$Y <- NULL
newx = data.matrix(test)
ridge.pred = predict(ridge.mod,newx=newx, type="response")

I keep getting this error message when using predict:

Error in as.matrix(cbind2(1, newx) %*% nbeta) : error in evaluating the argument 'x' in selecting a method for function 'as.matrix': Error in t(.Call(Csparse_dense_crossprod, y, t(x))) : error in evaluating the argument 'x' in selecting a method for function 't': Error: Cholmod error 'X and/or Y have wrong dimensions' at file ../MatrixOps/cholmod_sdmult.c, line 90

I've tried this on the "Hitters" dataset and it works perfectly fine.

library(ISLR);
library(glmnet)
Hitters=na.omit(Hitters)

Hitters$Rich<-ifelse(Hitters$Salary>500,1,0)
Hitters.train = Hitters[1:200,]
Hitters.test = Hitters[201:dim(Hitters)[1],]
x=model.matrix(Rich~.,Hitters)[,-1]
cv.out=cv.glmnet(x, Hitters$Rich, alpha=0, family="binomial")
    bestlam=cv.out$lambda.min
ridge.mod=glmnet(x, Hitters$Rich, alpha=0,lambda=bestlam, family="binomial")
    Hitters.test$Rich <- NULL
newx = data.matrix(Hitters.test)
ridge.pred=predict(ridge.mod,newx=newx, type="response")
head(ridge.pred)
ridge.pred[1:10,]

Does anyone know how I can fix this?

Answer 1

I had the same issue and I think it is caused by training and testing set having different factors thus different dimension for the sparse matrices.

My solution is to create the sparse matrix X for the combined dataset

traintest=rbind(training,testing)

X = sparse.model.matrix(as.formula(paste("y ~", paste(colnames(training[,-1]), sep = "", collapse=" +"))), data = traintest)
model = cv.glmnet(X[1:nrow(training),], training[,1], family = "binomial",type.measure = "auc",nfolds = 10)
plot(model)
model$lambda.min
#predict on test set
pred = predict(model, s='lambda.min', newx=X[-(1:nrow(training)),], type="response")

This is just to make sure test set has the same dimension.