Assign schema to pa.Table.from_pandas()

python pandas parquet pyarrow
Carlos P Ceballos picture Carlos P Ceballos · Mar 30, 2018 · Viewed 8k times · Source

Im getting this error when transforming a pandas.DF to parquet using pyArrow:

ArrowInvalid('Error converting from Python objects to Int64: Got Python object of type str but can only handle these types: integer

To find out which column is the problem I made a new df in a for loop, first with the first column and for each loop adding another column. I realized that the error is in a column of dtype: object that starts with 0s, I guess that's why pyArrow wants to convert the column to int but fails because other values are UUID

Im trying to pass a schema: (not sure if this is the way to go)

table = pa.Table.from_pandas(df, schema=schema, preserve_index=False)

where schema is: df.dtypes

Answer

Alexander picture Alexander · Mar 30, 2018

Carlos have you tried converting the column to one of the pandas types listed here https://arrow.apache.org/docs/python/pandas.html?

Can you post the output of df.dtypes?

If changing the pandas column type doesn't help you can define a pyarrow schema to pass in.

fields = [
    pa.field('id', pa.int64()),
    pa.field('secondaryid', pa.int64()),
    pa.field('date', pa.timestamp('ms')),
]

my_schema = pa.schema(fields)

table = pa.Table.from_pandas(sample_df, schema=my_schema, preserve_index=False)

More information here:

https://arrow.apache.org/docs/python/data.html https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.from_pandas https://arrow.apache.org/docs/python/generated/pyarrow.schema.html

Related questions

What are the differences between feather and parquet?

Both are columnar (disk-)storage formats for use in data analysis systems. Both are integrated within Apache Arrow (pyarrow package for python) and are designed to correspond with Arrow as a columnar in-memory analytics layer. How do both formats differ? …

Read More
Using pyarrow how do you append to parquet file?

How do you append/update to a parquet file with pyarrow? import pandas as pd import pyarrow as pa import pyarrow.parquet as pq table2 = pd.DataFrame({'one': [-1, np.nan, 2.5], 'two': ['foo', 'bar', 'baz'], 'three': [True, False, True]}) table3 = …

Read More
How to write a partitioned Parquet file using Pandas

I'm trying to write a Pandas dataframe to a partitioned file: df.to_parquet('output.parquet', engine='pyarrow', partition_cols = ['partone', 'partwo']) TypeError: __cinit__() got an unexpected keyword argument 'partition_cols' From the documentation I expected that the partition_cols …

Read More