Is there a faster lm function

r lm
Bangyou picture Bangyou · Aug 21, 2014 · Viewed 9.3k times · Source

I would like to get the slope of a linear regression fit for 1M separate data sets (1M * 50 rows for data.frame, or 1M * 50 for array). Now I am using the lm() function, which takes a very long time (about 10 min).

Is there any faster function for linear regression?

Answer

Dirk Eddelbuettel picture Dirk Eddelbuettel · Aug 21, 2014

Yes there are:

  • R itself has lm.fit() which is more bare-bones: no formula notation, much simpler result set

  • several of our Rcpp-related packages have fastLm() implementations: RcppArmadillo, RcppEigen, RcppGSL.

We have described fastLm() in a number of blog posts and presentations. If you want it in the fastest way, do not use the formula interface: parsing the formula and preparing the model matrix takes more time than the actual regression.

That said, if you are regressing a single vector on a single vector you can simplify this as no matrix package is needed.

Related questions

Predict() - Maybe I'm not understanding it

I posted earlier today about an error I was getting with using the predict function. I was able to get that corrected, and thought I was on the right path. I have a number of observations (actuals) and I have …

Read More
Extract regression coefficient values

I have a regression model for some time series data investigating drug utilisation. The purpose is to fit a spline to a time series and work out 95% CI etc. The model goes as follows: id <- ts(1:length(drug$…

Read More
How to succinctly write a formula with many variables from a data frame?

Suppose I have a response variable and a data containing three covariates (as a toy example): y = c(1,4,6) d = data.frame(x1 = c(4,-1,3), x2 = c(3,9,8), x3 = c(4,-4,-2)) I want to fit a linear regression to the data: …

Read More