Amazon Athena and compressed S3 files

amazon-web-services amazon-athena
MattY picture MattY · Dec 19, 2016 · Viewed 8k times · Source

I have an S3 bucket with several zipped CSV files (utilization logs.) I'd like to query this data with Athena, but the output is completely garbled.

It appears Athena is trying to parse the zip files without decompressing them first. Is it possible to force Hive to recognize my files as compressed data?

Answer

jens walter picture jens walter · Dec 19, 2016

For Athena compression is supported, but the supported formats are

  • Snappy (.snappy)
  • Zlib (.bz2)
  • GZIP (.gz)

Those formats are detected by their filename suffix. If the suffix doesn't match, the reader does not decode the content. I tested it with a test.csv.gz file and it worked right away. So try changing the compression from zip to gzip and it should work.

Related questions

Amazon Athena: no viable alternative at input

While creating a table in Athena; it gives me following exception: no viable alternative at input

Read More
Athena greater than condition in date column

I have the following query that I am trying to run on Athena. SELECT observation_date, COUNT(*) AS count FROM db.table_name WHERE observation_date > '2017-12-31' GROUP BY observation_date However it is producing this …

Read More
AWS Athena and date_format

I have some issue while formatting a timestamp with Amazon Athena service. select date_format(current_timestamp, 'y') Returns just 'y' (the string). The only way I found to format dates in Amazon Athena is trough CONCAT + YEAR + MONTH + DAY …

Read More