How to read only lines that fulfil a condition from a csv into R?

r large-data read.csv
Hernan picture Hernan · Apr 21, 2014 · Viewed 31.5k times · Source

I am trying to read a large csv file into R. I only want to read and work with some of the rows that fulfil a particular condition (e.g. Variable2 >= 3). This is a much smaller dataset.

I want to read these lines directly into a dataframe, rather than load the whole dataset into a dataframe and then select according to the condition, since the whole dataset does not easily fit into memory.

Answer

Karsten W. picture Karsten W. · Apr 21, 2014

You could use the read.csv.sql function in the sqldf package and filter using SQL select. From the help page of read.csv.sql:

library(sqldf)
write.csv(iris, "iris.csv", quote = FALSE, row.names = FALSE)
iris2 <- read.csv.sql("iris.csv", 
    sql = "select * from file where `Sepal.Length` > 5", eol = "\n")

Related questions

Add lines to a file

I'm new using R. I'm trying to add new lines to a file with my existing data in R. The problem is that my data is of about 30000 rows and 13000 cols. I already try to add a line with the …

Read More
How much data can R handle?

By "handle" I mean manipulate multi-columnar rows of data. How does R stack up against tools like Excel, SPSS, SAS, and others? Is R a viable tool for looking at "BIG DATA" (hundreds of millions to billions of rows)? If …

Read More
How to plot with a png as background?

I made a plot with a 3 million points and saved it as PNG. It took a few hours and I would like to avoid re-drawing all the points. How can I generate a new plot that has this PNG as …

Read More