Comparison between lz4 vs lz4_hc vs blosc vs snappy vs fastlz

snappy lossless-compression lz4
Sayantan Ghosh picture Sayantan Ghosh · Jun 3, 2016 · Viewed 19.7k times · Source

I have a large file of size 500 mb to compress in a minute with the best possible compression ratio. I have found out these algorithms to be suitable for my use.

  1. lz4
  2. lz4_hc
  3. snappy
  4. quicklz
  5. blosc

Can someone give a comparison of speed and compression ratios between these algorithms?

Answer

aalaap picture aalaap · Oct 20, 2016

Yann Collet's lz4, hands down.

enter image description here

Related questions

Parquet vs ORC vs ORC with Snappy

I am running a few tests on the storage formats available with Hive and using Parquet and ORC as major options. I included ORC once with default compression and once with Snappy. I have read many a documents that state …

Read More
How do I read Snappy compressed files on HDFS without using Hadoop?

I'm storing files on HDFS in Snappy compression format. I'd like to be able to examine these files on my local Linux file system to make sure that the Hadoop process that created them has performed correctly. When I copy …

Read More
Spark SQL - difference between gzip vs snappy vs lzo compression formats

I am trying to use Spark SQL to write parquet file. By default Spark SQL supports gzip, but it also supports other compression formats like snappy and lzo. What is the difference between these compression formats?

Read More