pyarrow error: toPandas attempted Arrow optimization

pyspark pyarrow
user5768866 picture user5768866 · Aug 28, 2018 · Viewed 7.2k times · Source

when I set pyarrow to true we using spark session, but when I run toPandas(), it throws the error:

"toPandas attempted Arrow optimization because 'spark.sql.execution.arrow.enabled' is set to true. Please set it to false to disable this"

May I know why it happens?

Answer

Maneesh Bishnoi picture Maneesh Bishnoi · Aug 22, 2019

By default PyArrow is disabled but it seems in your case it is enabled, you have to manually disable this configuration either from the current spark application session or permanently from the Spark configuration file.

If you want to disable this for all of you spark sessions, add below line to your Spark configuration at SPARK_HOME/conf/spark-defaults .conf. spark.sql.execution.arrow.enabled=false

But I would suggest using PyArrow if you are using pandas in your spark application, it will speed the data conversion between spark and pandas.

For more on PyArrow please visit my blog.

Related questions

ModuleNotFoundError: No module named 'pyarrow'

I am trying to run a simple pandas UDF example on my server. From here I have created a fresh environment just for the purpose of running this code. (PySparkEnv) $ conda list # packages in environment at /home/shekhar/.conda/envs/…

Read More
How to change dataframe column names in pyspark?

I come from pandas background and am used to reading data from CSV files into a dataframe and then simply changing the column names to something useful using the simple command: df.columns = new_column_name_list However, the same …

Read More
Load CSV file with Spark

I'm new to Spark and I'm trying to read CSV data from a file with Spark. Here's what I am doing : sc.textFile('file.csv') .map(lambda line: (line.split(',')[0], line.split(',')[1])) .collect() I would expect …

Read More