Use TQDM Progress Bar with Pandas

python pandas tqdm
sslack88 picture sslack88 · Nov 3, 2017 · Viewed 37.3k times · Source

Is it possible to use TQDM progress bar when importing and indexing large datasets using Pandas?

Here is an example of of some 5-minute data I am importing, indexing, and using to_datetime. It takes a while and it would be nice to see a progress bar.

#Import csv files into a Pandas dataframes and convert to Pandas datetime and set to index

eurusd_ask = pd.read_csv('EURUSD_Candlestick_5_m_ASK_01.01.2012-05.08.2017.csv')
eurusd_ask.index = pd.to_datetime(eurusd_ask.pop('Gmt time'))

Answer

Arjun Kava picture Arjun Kava · Sep 13, 2018

Find length by getting shape

for index, row in tqdm(df.iterrows(), total=df.shape[0]):
   print("index",index)
   print("row",row)

Related questions

How to use tqdm with pandas in a jupyter notebook?

I'm doing some analysis with pandas in a jupyter notebook and since my apply function takes a long time I would like to see a progress bar. Through this post here I found the tqdm library that provides a simple …

Read More
How do I make a progress bar for loading pandas DataFrame from a large xlsx file?

from https://pypi.org/project/tqdm/: import pandas as pd import numpy as np from tqdm import tqdm df = pd.DataFrame(np.random.randint(0, 100, (100000, 6))) tqdm.pandas(desc="my bar!")p` df.progress_apply(lambda x: x**2) I took this code …

Read More
Progress bar for pandas.DataFrame.to_sql

I want to migrate data from a large csv file to sqlite3 database. My code on Python 3.5 using pandas: con = sqlite3.connect(DB_FILENAME) df = pd.read_csv(MLS_FULLPATH) df.to_sql(con=con, name="MLS", if_exists="replace", …

Read More