Why can't I handle a KeyboardInterrupt in python?

Question

I'm writing python 2.6.6 code on windows that looks like this:

try:
    dostuff()
except KeyboardInterrupt:
    print "Interrupted!"
except:
    print "Some other exception?"
finally:
    print "cleaning up...."
    print "done."

dostuff() is a function that loops forever, reading a line at a time from an input stream and acting on it. I want to be able to stop it and clean up when I hit ctrl-c.

What's happening instead is that the code under except KeyboardInterrupt: isn't running at all. The only thing that gets printed is "cleaning up...", and then a traceback is printed that looks like this:

Traceback (most recent call last):
  File "filename.py", line 119, in <module>
    print 'cleaning up...'
KeyboardInterrupt

So, exception handling code is NOT running, and the traceback claims that a KeyboardInterrupt occurred during the finally clause, which doesn't make sense because hitting ctrl-c is what caused that part to run in the first place! Even the generic except: clause isn't running.

EDIT: Based on the comments, I replaced the contents of the try: block with sys.stdin.read(). The problem still occurs exactly as described, with the first line of the finally: block running and then printing the same traceback.

EDIT #2: If I add pretty much anything after the read, the handler works. So, this fails:

try:
    sys.stdin.read()
except KeyboardInterrupt:
    ...

But this works:

try:
    sys.stdin.read()
    print "Done reading."
except KeyboardInterrupt:
    ...

Here's what's printed:

Done reading. Interrupted!
cleaning up...
done.

So, for some reason, the "Done reading." line is printed, even though the exception occurred on the previous line. That's not really a problem - obviously I have to be able to handle an exception anywhere inside the "try" block. However, the print doesn't work normally - it doesn't print a newline afterwards like it's supposed to! The "Interruped" is printed on the same line... with a space before it, for some reason...? Anyway, after that the code does what it's supposed to.

It seems to me that this is a bug in handling an interrupt during a blocked system call.

Answer 1

Asynchronous exception handling is unfortunately not reliable (exceptions raised by signal handlers, outside contexts via C API, etc). You can increase your chances of handling the async exception properly if there is some coordination in the code about what piece of code is responsible for catching them (highest possible in the call stack seems appropriate except for very critical functions).

The called function (dostuff) or functions further down the stack may itself have a catch for KeyboardInterrupt or BaseException that you didn't/couldn't account for.

This trivial case worked just fine with python 2.6.6 (x64) interactive + Windows 7 (64bit):

>>> import time
>>> def foo():
...     try:
...             time.sleep(100)
...     except KeyboardInterrupt:
...             print "INTERRUPTED!"
...
>>> foo()
INTERRUPTED!  #after pressing ctrl+c

EDIT:

Upon further investigation, I tried what I believe is the example that others have used to reproduce the issue. I was lazy so I left out the "finally"

>>> def foo():
...     try:
...             sys.stdin.read()
...     except KeyboardInterrupt:
...             print "BLAH"
...
>>> foo()

This returns immediately after hitting CTRL+C. The interesting thing happened when I immediately tried to call foo again:

>>> foo()

Traceback (most recent call last):
  File "c:\Python26\lib\encodings\cp437.py", line 14, in decode
    def decode(self,input,errors='strict'):
KeyboardInterrupt

The exception was raised immediately without me hitting CTRL+C.

This would seem to make sense - it appears that we are dealing with nuances in how asynchronous exceptions are handled in Python. It can take several bytecode instructions before the async exception is actually popped and then raised within the current execution context. (That's the behavior that I've seen when playing with it in the past)

See the C API: http://docs.python.org/c-api/init.html#PyThreadState_SetAsyncExc

So this somewhat explains why KeyboardInterrupt gets raised in the context of the execution of the finally statement in this example:

>>> def foo():
...     try:
...             sys.stdin.read()
...     except KeyboardInterrupt:
...             print "interrupt"
...     finally:
...             print "FINALLY"
...
>>> foo()
FINALLY
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 7, in foo
KeyboardInterrupt

There could be some crazy mixing of custom signal handlers mixed with the interpreter's standard KeyboardInterrupt/CTRL+C handler that's resulting in this sort of behavior. For example, the read() call sees the signal and bails, but it re-raises the signal after deregistering it's handler. I wouldn't know for sure without inspecting the interpreter codebase.

This is why I generally shy away from making use of async exceptions....

EDIT 2

I think there's a good case for a bug report.

Again more theories...(just based on reading code) See the file object source: http://svn.python.org/view/python/branches/release26-maint/Objects/fileobject.c?revision=81277&view=markup

file_read calls Py_UniversalNewlineFread(). fread can return with an error with errno = EINTR (it performs its own signal handling). In this case Py_UniversalNewlineFread() bails but does not perform any signal checking with PyErr_CheckSignals() so that the handlers can get called synchronously. file_read clears the file error but also does not call PyErr_CheckSignals().

See getline() and getline_via_fgets() for examples of how it's used. The pattern is documented in this bug report for a similar issue: ( http://bugs.python.org/issue1195 ). So it seems that the signal is handled at an indeterminate time by the interpreter.

I guess there's little value in diving any deeper since it's still not clear whether the sys.stdin.read() example is a proper analog of your "dostuff()" function. (there could be multiple bugs at play)