Need to avoid subprocess deadlock without communicate

Question

I need a execute a command that produces a lot of output and takes a lot of time to execute (> 30 minutes). I was thinking of using subprocess.Popen to do it. I need to capture the output of the command, so I pass PIPE to stdout and stderr.

A deadlock problem when using Popen.wait() is well documented on a lot of forums, so Popen.communicate() is the proposed way of avoiding the deadlock. The problem with that solution is that communicate() blocks until the command is completed. I need to print everything that arrives on stdout while the command is executed. If there is no output after 20 minutes, the script execution will be killed.

Here are some constraints that I need to respect:

• My Python version is 2.4.2 and I can't upgrade.
• If the solution is still to use subprocess, I need to pass subprocess.PIPE to all std handles to avoid this bug: http://bugs.python.org/issue1124861

Is there a way to do it?

Answer 1

• to workaround python bug #1124861 on Python2.4 you could attach stdin to a NUL device

import os
from subprocess import PIPE, STDOUT, Popen

lines = []
p = Popen(cmd, bufsize=1, stdin=open(os.devnull), stdout=PIPE, stderr=STDOUT)
for line in iter(p.stdout.readline, ''):
      print line,          # print to stdout immediately
      lines.append(line)   # capture for later
p.stdout.close()
p.wait()