Python: save pandas data frame to parquet file

python-3.x hdfs parquet
Edamame picture Edamame · Dec 9, 2016 · Viewed 29.3k times · Source

Is it possible to save a pandas data frame directly to a parquet file? If not, what would be the suggested process?

The aim is to be able to send the parquet file to another team, which they can use scala code to read/open it. Thanks!

Answer

ben26941 picture ben26941 · Mar 10, 2018

Pandas has a core function to_parquet(). Just write the dataframe to parquet format like this: 

df.to_parquet('myfile.parquet')

You still need to install a parquet library such as fastparquet. If you have more than one parquet library installed, you also need to specify which engine you want pandas to use, otherwise it will take the first one to be installed (as in the documentation). For example:

df.to_parquet('myfile.parquet', engine='fastparquet')

Related questions

Convert bytes to a string

I'm using this code to get standard output from an external program: >>> from subprocess import * >>> command_stdout = Popen(['ls', '-l'], stdout=PIPE).communicate()[0] The communicate() method returns an array of bytes: >>> …

Read More
Best way to convert string to bytes in Python 3?

There appear to be two different ways to convert a string to bytes, as seen in the answers to TypeError: 'str' does not support the buffer interface Which of these methods would be better or more Pythonic? Or is it …

Read More
How to install pip with Python 3?

I want to install pip. It should support Python 3, but it requires setuptools, which is available only for Python 2. How can I install pip with Python 3?

Read More