How to write to CSV in Spark

file csv hadoop apache-spark distributed-computing
Karusmeister picture Karusmeister · May 7, 2014 · Viewed 44k times · Source

I'm trying to find an effective way of saving the result of my Spark Job as a csv file. I'm using Spark with Hadoop and so far all my files are saved as part-00000.

Any ideas how to make my spark saving to file with a specified file name?

Answer

Tathagata Das picture Tathagata Das · May 8, 2014

Since Spark uses Hadoop File System API to write data to files, this is sort of inevitable. If you do 

rdd.saveAsTextFile("foo")

It will be saved as "foo/part-XXXXX" with one part-* file every partition in the RDD you are trying to save. The reason each partition in the RDD is written a separate file is for fault-tolerance. If the task writing 3rd partition (i.e. to part-00002) fails, Spark simply re-run the task and overwrite the partially written/corrupted part-00002, with no effect on other parts. If they all wrote to the same file, then it is much harder recover a single task for failures.

The part-XXXXX files are usually not a problem if you are going to consume it again in Spark / Hadoop-based frameworks because since they all use HDFS API, if you ask them to read "foo", they will all read all the part-XXXXX files inside foo as well.

Related questions

Writing data into CSV file in C#

I am trying to write into a csv file row by row using C# language. Here is my function string first = reader[0].ToString(); string second=image.ToString(); string csv = string.Format("{0},{1}\n", first, second); File.WriteAllText(filePath, csv); The whole …

Read More
How to export data as CSV format from SQL Server using sqlcmd?

I can quite easily dump data into a text file such as: sqlcmd -S myServer -d myDB -E -Q "select col1, col2, col3 from SomeTable" -o "MyData.txt" However, I have looked at the help files for SQLCMD but have …

Read More
Python write to CSV line by line

I have data which is being accessed via http request and is sent back by the server in a comma separated format, I have the following code : site= 'www.example.com' hdr = {'User-Agent': 'Mozilla/5.0'} req = urllib2.Request(site,headers=…

Read More