Convert file of JSON objects to Parquet file

json apache parquet apache-drill
danieltahara picture danieltahara · Feb 11, 2014 · Viewed 11.3k times · Source

Motivation: I want to load the data into Apache Drill. I understand that Drill can handle JSON input, but I want to see how it performs on Parquet data.

Is there any way to do this without first loading the data into Hive, etc and then using one of the Parquet connectors to generate an output file?

Answer

blue picture blue · May 28, 2015

Kite has support for importing JSON to both Avro and Parquet formats via its command-line utility, kite-dataset.

First, you would infer the schema of your JSON:

kite-dataset json-schema sample-file.json -o schema.avsc

Then you can use that file to create a Parquet Hive table:

kite-dataset create mytable --schema schema.avsc --format parquet

And finally, you can load your JSON into the dataset.

kite-dataset json-import sample-file.json mytable

You can also import an entire directly stored in HDFS. In that case, Kite will use a MR job to do the import.

Related questions

PHP Fatal error: Call to undefined function json_decode()

Apache is logging PHP Fatal error: Call to undefined function json_decode(). After some googling, it seems this problem is a result of not having the latest version of php. Oddly, running php --version ouputs PHP 5.5.1-2+debphp.org~precise+2 (…

Read More
Returning application/json content-type for .json files on RHEL/Apache

OS is specifically RHEL6 running Apache2. So in my Apache configuration, I have: LoadModule mime_module modules/mod_mime.so And in /etc/mime.types, I have the following: application/json json Yet when requesting a file with the .json …

Read More
D3.js can't load json file

I am trying to create a D3.js packed circle diagram. When I embed the data in the HTML file, it works fine. When I put the data in an external file, I get nothing (blank DOM, no console msgs). …

Read More