How to write parquet file from pandas dataframe in S3 in python

python-3.x amazon-s3 parquet
Alexsander picture Alexsander · Nov 21, 2018 · Viewed 23.4k times · Source

I have a pandas dataframe. i want to write this dataframe to parquet file in S3. I need a sample code for the same.I tried to google it. but i could not get a working sample code.

Answer

Wai Kiat picture Wai Kiat · Nov 27, 2018

For your reference, I have the following code works.

s3_url = 's3://bucket/folder/bucket.parquet.gzip'
df.to_parquet(s3_url, compression='gzip')

In order to use to_parquet, you need pyarrow or fastparquet to be installed. Also, make sure you have correct information in your config and credentials files, located at .aws folder.

Edit: Additionally, s3fs is needed. see https://stackoverflow.com/a/54006942/1862909

Related questions

Download file from AWS S3 using Python

I am trying to download a file from Amazon S3 bucket to my local using the below code but I get an error saying "Unable to locate credentials" Given below is the code I have written: from boto3.session import …

Read More
Boto3 S3, sort bucket by last modified

I need to fetch a list of items from S3 using Boto3, but instead of returning default sort order (descending) I want it to return it via reverse order. I know you can do it via awscli: aws s3api …

Read More
Unable to import module 'lambda_function': No module named 'pandas'

START RequestId: 3d5691d9-ad79-4eed-a26c-5bc3f1a23a99 Version: $LATEST Unable to import module 'lambda_function': No module named 'pandas' END RequestId: 3d5691d9-ad79-4eed-a26c-5bc3f1a23a99 I'm using Windows 7 64-bit …

Read More