How can I prevent rbind() from geting really slow as dataframe grows larger?

r performance dataframe append rbind
Mark picture Mark · Feb 4, 2013 · Viewed 11.7k times · Source

I have a dataframe with only 1 row. To this I start to add rows by using rbind

df #mydataframe with only one row
for (i in 1:20000)
{
    df<- rbind(df, newrow)

}

this gets very slow as i grows. Why is that? and how can I make this type of code faster?

Answer

joran picture joran · Feb 4, 2013

You are in the 2nd circle of hell, namely failing to pre-allocate data structures.

Growing objects in this fashion is a Very Very Bad Thing in R. Either pre-allocate and insert:

df <- data.frame(x = rep(NA,20000),y = rep(NA,20000))

or restructure your code to avoid this sort of incremental addition of rows. As discussed at the link I cite, the reason for the slowness is that each time you add a row, R needs to find a new contiguous block of memory to fit the data frame in. Lots 'o copying.

Related questions

Split a large dataframe into a list of data frames based on common value in column

I have a data frame with 10 columns, collecting actions of "users", where one of the columns contains an ID (not unique, identifying user)(column 10). the length of the data frame is about 750000 rows. I am trying to extract individual data …

Read More
Fastest way to replace NAs in a large data.table

I have a large data.table, with many missing values scattered throughout its ~200k rows and 200 columns. I would like to re code those NA values to zeros as efficiently as possible. I see two options: 1: Convert to a data.…

Read More
In R, how do you loop over the rows of a data frame really fast?

Suppose that you have a data frame with many rows and many columns. The columns have names. You want to access rows by number, and columns by name. For example, one (possibly slow) way to loop over the rows is …

Read More