python sigkill catching strategies

python linux memory sigkill
Hoopes picture Hoopes · Jun 9, 2015 · Viewed 14.6k times · Source

I was wondering if there was any way to catch the sigkill from the OOM killer. I have a task queue, and every so often a mammoth task is created that is killed by OOM. This:

catch Exception as ex:
    # clean up!

does not work, as SIGKILL can't be caught. So........is there ANY strategy to clean up after a SIGKILL? Can I fork, and watch the child process? If so, any resources opened by the child process would have to be known in advance by the parent? Or could I just do some version of

ps -ef | grep <child pid> | xargs kill -9  (you get the idea...)

Currently, if I don't clean up after an OOM kill, I leave behind plenty of child processes and other things that just make it worse when the task is retried, and soon enough, the server is unreachable.

Finally, is it enough to just do:

kill -9 <process id>

to test this exact situation?

Thanks very much!

Answer

Mike Shoup picture Mike Shoup · Jun 9, 2015

SIGKILL by its very nature cannot be trapped.

See http://en.wikipedia.org/wiki/Unix_signal#SIGKILL:

SIGKILL

The SIGKILL signal is sent to a process to cause it to terminate immediately (kill). In contrast to SIGTERM and SIGINT, this signal cannot be caught or ignored, and the receiving process cannot perform any clean-up upon receiving this signal.

The best thing to do is the next time your process launches, look for anything that needs to be cleaned up.

And yes, kill -9 <pid> will send a SIGKILL to the process. (To be precise, it sends the 9th signal - it just happens that SIGKILL has the number 9 on pretty much every system. You could alternatively write kill -KILL <pid>, which lets you specify the signal by name instead of by number in a portable way.)

Related questions

Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory"

Note: This question was originally asked here but the bounty time expired even though an acceptable answer was not actually found. I am re-asking this question including all details provided in the original question. A python script is running a …

Read More
How to limit memory usage within a python process

I run Python 2.7 on a Linux machine with 16GB Ram and 64 bit OS. A python script I wrote can load too much data into memory, which slows the machine down to the point where I cannot even kill the process …

Read More
open() in Python does not create a file if it doesn't exist

What is the best way to open a file as read/write if it exists, or if it does not, then create it and open it as read/write? From what I read, file = open('myfile.dat', 'rw') should do …

Read More