Piping output of subprocess.Popen to files

user280867 picture user280867 · Feb 25, 2010 · Viewed 61.6k times · Source

I need to launch a number of long-running processes with subprocess.Popen, and would like to have the stdout and stderr from each automatically piped to separate log files. Each process will run simultaneously for several minutes, and I want two log files (stdout and stderr) per process to be written to as the processes run.

Do I need to continually call p.communicate() on each process in a loop in order to update each log file, or is there some way to invoke the original Popen command so that stdout and stderr are automatically streamed to open file handles?

Answer

John La Rooy picture John La Rooy · Feb 25, 2010

You can pass stdout and stderr as parameters to Popen()

subprocess.Popen(self, args, bufsize=0, executable=None, stdin=None, stdout=None,
                 stderr=None, preexec_fn=None, close_fds=False, shell=False,
                 cwd=None, env=None, universal_newlines=False, startupinfo=None, 
                 creationflags=0)

For example

>>> import subprocess
>>> with open("stdout.txt","wb") as out, open("stderr.txt","wb") as err:
...    subprocess.Popen("ls",stdout=out,stderr=err)
... 
<subprocess.Popen object at 0xa3519ec>
>>>