Example of using concurrent.futures (backport for 2.7):
import concurrent.futures # line 01
def f(x): # line 02
return x * x # line 03
data = [1, 2, 3, None, 5] # line 04
with concurrent.futures.ThreadPoolExecutor(len(data)) as executor: # line 05
futures = [executor.submit(f, n) for n in data] # line 06
for future in futures: # line 07
print(future.result()) # line 08
Output:
1
4
9
Traceback (most recent call last):
File "C:\test.py", line 8, in <module>
print future.result() # line 08
File "C:\dev\Python27\lib\site-packages\futures-2.1.4-py2.7.egg\concurrent\futures\_base.py", line 397, in result
return self.__get_result()
File "C:\dev\Python27\lib\site-packages\futures-2.1.4-py2.7.egg\concurrent\futures\_base.py", line 356, in __get_result
raise self._exception
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'
String "...\_base.py", line 356, in __get_result"
is not endpoint I expected to see. Is it possible to get real line where exception was thrown? Something like:
File "C:\test.py", line 3, in f
return x * x # line 03
Python3 seems to show correct line number in this case. Why can't python2.7? And is there any workaround?
I was in your same situation and I really needed to have the traceback of the raised exceptions.
I was able to develop this workaround which consists in using the following subclass of the
ThreadPoolExecutor
.
import sys
import traceback
from concurrent.futures import ThreadPoolExecutor
class ThreadPoolExecutorStackTraced(ThreadPoolExecutor):
def submit(self, fn, *args, **kwargs):
"""Submits the wrapped function instead of `fn`"""
return super(ThreadPoolExecutorStackTraced, self).submit(
self._function_wrapper, fn, *args, **kwargs)
def _function_wrapper(self, fn, *args, **kwargs):
"""Wraps `fn` in order to preserve the traceback of any kind of
raised exception
"""
try:
return fn(*args, **kwargs)
except Exception:
raise sys.exc_info()[0](traceback.format_exc()) # Creates an
# exception of the
# same type with the
# traceback as
# message
If you use this subclass and run the following snippet:
def f(x):
return x * x
data = [1, 2, 3, None, 5]
with ThreadPoolExecutorStackTraced(max_workers=len(data)) as executor:
futures = [executor.submit(f, n) for n in data]
for future in futures:
try:
print future.result()
except TypeError as e:
print e
the output will be something like:
1
4
9
Traceback (most recent call last):
File "future_traceback.py", line 17, in _function_wrapper
return fn(*args, **kwargs)
File "future_traceback.py", line 24, in f
return x * x
TypeError: unsupported operand type(s) for *: 'NoneType' and 'NoneType'
25
The problem is in the usage of sys.exc_info()
by the futures
library. From the
documentation:
This function returns a tuple of three values that give information about the exception that is currently being handled. [...] If no exception is being handled anywhere on the stack, a tuple containing three None values is returned. Otherwise, the values returned are (type, value, traceback). Their meaning is: type gets the exception type of the exception being handled (a class object); value gets the exception parameter (its associated value or the second argument to raise, which is always a class instance if the exception type is a class object); traceback gets a traceback object which encapsulates the call stack at the point where the exception originally occurred.
Now, if you look at the source code of futures
you can see by yourself why the traceback is
lost: when an exception raises and it is to be set to the Future
object only
sys.exc_info()[1]
is passed. See:
https://code.google.com/p/pythonfutures/source/browse/concurrent/futures/thread.py (L:63) https://code.google.com/p/pythonfutures/source/browse/concurrent/futures/_base.py (L:356)
So, to avoid losing the traceback, you have to save it somewhere. My workaround is to wrap
the function to submit into a wrapper whose only task is to catch every kind of exception and
to raise an exception of the same type whose message is the traceback. By doing this, when an
exception is raised it is captured and reraised by the wrapper, then when sys.exc_info()[1]
is assigned to the exception of the Future
object, the traceback is not lost.