How can we use tqdm in a parallel execution with joblib?
I want to run a function in parallel, and wait until all parallel nodes are done, using joblib. Like in the example:
from math import sqrt
from joblib import Parallel, delayed
Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))
But, I want that the execution will be seen in a single progressbar like with tqdm, showing how many jobs has been completed.
How would you do that?
Answer
Just put range(10) inside tqdm(...)! It probably seemed too good to be true for you, but it really works (on my machine):
from math import sqrt
from joblib import Parallel, delayed
from tqdm import tqdm
result = Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in tqdm(range(100000)))