Spark SQL - difference between gzip vs snappy vs lzo compression formats

apache-spark gzip parquet snappy lzo
Shankar picture Shankar · Mar 4, 2016 · Viewed 26.9k times · Source

I am trying to use Spark SQL to write parquet file.

By default Spark SQL supports gzip, but it also supports other compression formats like snappy and lzo.

What is the difference between these compression formats?

Answer

Garren S picture Garren S · May 30, 2017

Use Snappy if you can handle higher disk usage for the performance benefits (lower CPU + Splittable).

When Spark switched from GZIP to Snappy by default, this was the reasoning:

Based on our tests, gzip decompression is very slow (< 100MB/s), making queries decompression bound. Snappy can decompress at ~ 500MB/s on a single core.

Snappy:

  • Storage Space: High
  • CPU Usage: Low
  • Splittable: Yes (1)

GZIP:

  • Storage Space: Medium
  • CPU Usage: Medium
  • Splittable: No

1) http://boristyukin.com/is-snappy-compressed-parquet-file-splittable/

Related questions

Is gzip format supported in Spark?

For a Big Data project, I'm planning to use spark, which has some nice features like in-memory-computations for repeated workloads. It can run on local files or on top of HDFS. However, in the official documentation, I can't find any …

Read More
Ho to read ".gz" compressed file using spark DF or DS?

I have a compressed file with .gz format, Is it possible to read the file directly using spark DF/DS? Details : File is csv with tab delimited.

Read More
Dealing with a large gzipped file in Spark

I have a large (about 85 GB compressed) gzipped file from s3 that I am trying to process with Spark on AWS EMR (right now with an m4.xlarge master instance and two m4.10xlarge core instances each with a 100 GB …

Read More