How to transform Dask.DataFrame to pd.DataFrame?

python pandas dask
Philipp_Kats picture Philipp_Kats · Aug 18, 2016 · Viewed 15.3k times · Source

How can I transform my resulting dask.DataFrame into pandas.DataFrame (let's say I am done with heavy lifting, and just want to apply sklearn to my aggregate result)?

Answer

MRocklin picture MRocklin · Aug 18, 2016

You can call the .compute() method to transform a dask.dataframe to a pandas dataframe:

df = df.compute()

Related questions

Convert Pandas dataframe to Dask dataframe

Suppose I have pandas dataframe as: df=pd.DataFrame({'a':[1,2,3],'b':[4,5,6]}) When I convert it into dask dataframe what should name and divisions parameter consist of: from dask import dataframe as dd sd=dd.DataFrame(df.to_dict(),divisions=1,meta=…

Read More
At what situation I can use Dask instead of Apache Spark?

I am currently using Pandas and Spark for data analysis. I found Dask provides parallelized NumPy array and Pandas DataFrame. Pandas is easy and intuitive for doing data analysis in Python. But I find difficulty in handling multiple bigger dataframes …

Read More
python dask DataFrame, support for (trivially parallelizable) row apply?

I recently found dask module that aims to be an easy-to-use python parallel processing module. Big selling point for me is that it works with pandas. After reading a bit on its manual page, I can't find a way to …

Read More