Load a small random sample from a large csv file into R data frame

r csv random dataframe bigdata
P.Escondido picture P.Escondido · Mar 7, 2014 · Viewed 14.9k times · Source

The csv file to be processed does not fit into the memory. How can one read ~20K random lines of it to do basic statistics on the selected data frame?

Answer

Jed picture Jed · Mar 7, 2014

You can also just do it in the terminal with perl.

perl -ne 'print if (rand() < .01)' biglist.txt > subset.txt

This won't necessarily get you exactly 20,000 lines. (Here it'll grab about .01 or 1% of the total lines.) It will, however, be really really fast, and you'll have a nice copy of both files in your directory. You can then load the smaller file into R however you want.

Related questions

How to import multiple .csv files at once?

Suppose we have a folder containing multiple data.csv files, each containing the same number of variables but each from different times. Is there a way in R to import them all simultaneously rather than having to import them all …

Read More
How do I import a CSV file in R?

I have a .csv file in my workstation. How can I open that file in R and do statistical calculation?

Read More
Subset of rows containing NA (missing) values in a chosen column of a data frame

We have a data frame from a CSV file. The data frame DF has columns that contain observed values and a column (VaR2) that contains the date at which a measurement has been taken. If the date was not recorded, …

Read More