Memory efficient alternative to rbind - in-place rbind?

r dataframe rbind
Sebastian picture Sebastian · Aug 17, 2011 · Viewed 13k times · Source

I need to rbind two large data frames. Right now I use

df <- rbind(df, df.extension)

but I (almost) instantly run out of memory. I guess its because df is held in the memory twice. I might see even bigger data frames in the future, so I need some kind of in-place rbind.

So my question is: Is there a way to avoid data duplication in memory when using rbind?

I found this question, which uses SqlLite, but I really want to avoid using the hard drive as a cache.

Answer

Ari B. Friedman picture Ari B. Friedman · Aug 18, 2012

data.table is your friend!

C.f. http://www.mail-archive.com/[email protected]/msg175877.html

Following up on nikola's comment, here is ?rbindlist's description (new in v1.8.2) :

Same as do.call("rbind",l), but much faster.

Related questions

Converting nested list to dataframe

The goal is to convert a nested list which sometimes contain missing records into a data frame. An example of the structure when there are missing records is: str(mylist) List of 3 $ :List of 7 ..$ Hit : chr "True" ..$ Project: chr "Blue" ..$ …

Read More
rbind dataframes with a different column name

I've 12 data frames, each one contains 6 columns: 5 have the same name, 1 is different. Then when I call rbind() I get: Error in match.names(clabs, names(xi)) : names do not match previous names The column that differs is: "goal1Completions". …

Read More
R: losing column names when adding rows to an empty data frame

I am just starting with R and encountered a strange behaviour: when inserting the first row in an empty data frame, the original column names get lost. example: a<-data.frame(one = numeric(0), two = numeric(0)) a #[1] one two #<0 …

Read More