How to decompress the hadoop reduce output file end with snappy?

hadoop snappy
DeepNightTwo picture DeepNightTwo · Nov 6, 2013 · Viewed 18.3k times · Source

Our hadoop cluster using snappy as default codec. Hadoop job reduce output file name is like part-r-00000.snappy. JSnappy fails to decompress the file bcz JSnappy requires the file start with SNZ. The reduce output file start with some bytes 0 somehow.

How could I decompress the file?

Answer

arviarya picture arviarya · Mar 11, 2014

Use "Hadoop fs -text" to read this file and pipe it to txt file. ex:

hadoop fs -text part-r-00001.snappy > /tmp/mydatafile.txt

Related questions

Parquet vs ORC vs ORC with Snappy

I am running a few tests on the storage formats available with Hive and using Parquet and ORC as major options. I included ORC once with default compression and once with Snappy. I have read many a documents that state …

Read More
How do I read Snappy compressed files on HDFS without using Hadoop?

I'm storing files on HDFS in Snappy compression format. I'd like to be able to examine these files on my local Linux file system to make sure that the Hadoop process that created them has performed correctly. When I copy …

Read More
hadoop mapreduce: java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

I am trying to write a snappy block compressed sequence file from a map-reduce job. I am using hadoop 2.0.0-cdh4.5.0, and snappy-java 1.0.4.1 Here is my code: package jinvestor.jhouse.mr; import java.io.ByteArrayOutputStream; import java.io.IOException; import java.…

Read More