How to profile large datasets with Pandas profiling?

python pandas profiling pandas-profiling
Viv picture Viv · May 8, 2019 · Viewed 11.2k times · Source

Data is not perfectly clean, but is used without issue with pandas. The pandas library provides many extremely useful functions for EDA.

But when I use profiling for large data i.e 100 million records with 10 columns, reading it from a database table, it does not complete and my laptop runs out of memory, the size of data in csv is around 6 gb and my RAM is 14 GB my idle usage is around 3 - 4 GB approximately.

df = pd.read_sql_query("select * from table", conn_params)
profile = pandas.profiling.ProfileReport(df)
profile.to_file(outputfile="myoutput.html")

I have also tried with check_recoded = False option as well. But it does not help in profiling entirely. Is there any way to chunk and read the data and finally generate the summary report as a whole? OR any other method to use this function with large dataset.

Answer

Giorgos Myrianthous picture Giorgos Myrianthous · Mar 18, 2020

v2.4 introduced the minimal mode that disables expensive computations (such as correlations and dynamic binning):

from pandas_profiling import ProfileReport


profile = ProfileReport(df, minimal=True)
profile.to_file(output_file="output.html")

Related questions

Calculate summary statistics of columns in dataframe

I have a dataframe of the following form (for example) shopper_num,is_martian,number_of_items,count_pineapples,birth_country,tranpsortation_method 1,FALSE,0,0,MX, 2,FALSE,1,0,MX, 3,FALSE,0,0,MX, 4,FALSE,22,0,MX, 5,FALSE,0,0,MX, 6,FALSE,0,0,MX, 7,FALSE,5,0,MX, 8,FALSE,0,0,MX, 9,FALSE,4,0,…

Read More
How to select rows from a DataFrame based on column values

How can I select rows from a DataFrame based on values in some column in Pandas? In SQL, I would use: SELECT * FROM table WHERE colume_name = some_value I tried to look at Pandas' documentation, but I did not …

Read More
Renaming columns in pandas

I have a DataFrame using pandas and column labels that I need to edit to replace the original column labels. I'd like to change the column names in a DataFrame A where the original column names are: ['$a', '$…

Read More