Which setting to use in Spark to specify compression of `Output`?

hadoop apache-spark hadoop-plugins
nikk picture nikk · Aug 14, 2016 · Viewed 11.9k times · Source

So, Spark has the file spark-defaults.xml for specifying what settings, including which compression codec is to used and at what stage (RDD, Shuffle). Most of the settings can be set at the application level.

EDITED:

conf = SparkConf() conf.set("spark.hadoop.mapred.output.compress", "true") conf.set("spark.hadoop.mapred.output.compression.codec", "org.apache.hadoop.io.compress.snappy")

How can I use spark-defaults.xml to tell Spark to use a particular codec to compress Spark outputs only?

Option 1 spark.hadoop.mapred.output.compress true spark.hadoop.mapred.output.compression.codec snappy

Option 2: spark.mapreduce.output.fileoutputformat.compress true spark.mapreduce.output.fileoutputformat.compress.codec snappy

Option 3: mapreduce.output.fileoutputformat.compress true mapreduce.output.fileoutputformat.compress.codec snappy

Anyone has the proper way to sethe this (from any of these options or something similar)? I am running Spark 1.6.1.

Answer

ronhash picture ronhash · May 24, 2017

You should add this to your spark-defaults.xml:

<property>
    <name>spark.hadoop.mapred.output.compress</name>
    <value>true</value>
</property>
<property>
    <name>spark.hadoop.mapred.output.compression.codec</name>
    <value>snappy</value>
</property>

This is the same as adding these in the spark-submit command:

--conf spark.hadoop.mapred.output.compress=true
--conf spark.hadoop.mapred.output.compression.codec=snappy

Related questions

Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to export data from Spark SQL to CSV

This command works with HiveQL: insert overwrite directory '/data/home.csv' select * from testtable; But with Spark SQL I'm getting an error with an org.apache.spark.sql.hive.HiveQl stack trace: java.lang.RuntimeException: Unsupported language features in …

Read More
How to turn off INFO logging in Spark?

I installed Spark using the AWS EC2 guide and I can launch the program fine using the bin/pyspark script to get to the spark prompt and can also do the Quick Start quide successfully. However, I cannot for the …

Read More