Python: Wait on all of `concurrent.futures.ThreadPoolExecutor`'s futures

python concurrency future
Ram Rachum picture Ram Rachum · Jan 15, 2014 · Viewed 44.3k times · Source

I've given concurrent.futures.ThreadPoolExecutor a bunch of tasks, and I want to wait until they're all completed before proceeding with the flow. How can I do that, without having to save all the futures and call wait on them? (I want an action on the executor.)

Answer

Bakuriu picture Bakuriu · Jan 15, 2014

Just call Executor.shutdown:

shutdown(wait=True)

Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to Executor.submit() and Executor.map() made after shutdown will raise RuntimeError.

If wait is True then this method will not return until all the pending futures are done executing and the resources associated with the executor have been freed.

However if you keep track of your futures in a list then you can avoid shutting the executor down for future use using the futures.wait() function:

concurrent.futures.wait(fs, timeout=None, return_when=ALL_COMPLETED)

Wait for the Future instances (possibly created by different Executor instances) given by fs to complete. Returns a named 2-tuple of sets. The first set, named done, contains the futures that completed (finished or were cancelled) before the wait completed. The second set, named not_done, contains uncompleted futures.

note that if you don't provide a timeout it waits until all futures have completed.

You can also use futures.as_completed() instead, however you'd have to iterate over it.

Related questions

Pass multiple parameters to concurrent.futures.Executor.map?

The concurrent.futures.Executor.map takes a variable number of iterables from which the function given is called. How should I call it if I have a generator that produces tuples that are normally unpacked in place? The following doesn't …

Read More
ProcessPoolExecutor from concurrent.futures way slower than multiprocessing.Pool

I was experimenting with the new shiny concurrent.futures module introduced in Python 3.2, and I've noticed that, almost with identical code, using the Pool from concurrent.futures is way slower than using multiprocessing.Pool. This is the version using multiprocessing: …

Read More
How can I use threading in Python?

I am trying to understand threading in Python. I've looked at the documentation and examples, but quite frankly, many examples are overly sophisticated and I'm having trouble understanding them. How do you clearly show tasks being divided for multi-threading?

Read More