How do I read a large CSV file with Scala Stream class?

scala csv streaming large-files
Jan Willem Tulp picture Jan Willem Tulp · Nov 23, 2010 · Viewed 26.5k times · Source

How do I read a large CSV file (> 1 Gb) with a Scala Stream? Do you have a code example? Or would you use a different way to read a large CSV file without loading it into memory first?

Answer

Kevin Wright picture Kevin Wright · Nov 23, 2010

Just use Source.fromFile(...).getLines as you already stated.

That returns an Iterator, which is already lazy (You'd use stream as a lazy collection where you wanted previously retrieved values to be memoized, so you can read them again)

If you're getting memory problems, then the problem will lie in what you're doing after getLines. Any operation like toList, which forces a strict collection, will cause the problem.

Related questions

Write single CSV file using spark-csv

I am using https://github.com/databricks/spark-csv , I am trying to write a single CSV, but not able to, it is making a folder. Need a Scala function which will take parameter like path and file name and write …

Read More
How do I skip a header from CSV files in Spark?

Suppose I give three files paths to a Spark context to read and each file has a schema in the first row. How can we skip schema lines from headers? val rdd=sc.textFile("file1,file2,file3") Now, how can …

Read More
Use Scala parser combinator to parse CSV files

I'm trying to write a CSV parser using Scala parser combinators. The grammar is based on RFC4180. I came up with the following code. It almost works, but I cannot get it to correctly separate different records. What did I …

Read More