real time subprocess.Popen via stdout and PIPE

python logging subprocess pipe popen
alfredodeza picture alfredodeza · Jan 17, 2010 · Viewed 38.9k times · Source

I am trying to grab stdout from a subprocess.Popen call and although I am achieving this easily by doing:

cmd = subprocess.Popen('ls -l', shell=True, stdout=PIPE)
for line in cmd.stdout.readlines():
    print line

I would like to grab stdout in "real time". With the above method, PIPE is waiting to grab all the stdout and then it returns.

So for logging purposes, this doesn't meet my requirements (e.g. "see" what is going on while it happens).

Is there a way to get line by line, stdout while is running? Or is this a limitation of subprocess(having to wait until the PIPE closes).

EDIT If I switch readlines() for readline() I only get the last line of the stdout (not ideal):

In [75]: cmd = Popen('ls -l', shell=True, stdout=PIPE)
In [76]: for i in cmd.stdout.readline(): print i
....: 
t
o
t
a
l

1
0
4

Answer

Jeff picture Jeff · Feb 26, 2010

Your interpreter is buffering. Add a call to sys.stdout.flush() after your print statement.

Related questions

live output from subprocess command

I'm using a python script as a driver for a hydrodynamics code. When it comes time to run the simulation, I use subprocess.Popen to run the code, collect the output from stdout and stderr into a subprocess.PIPE --- …

Read More
Run subprocess and print output to logging

I am looking for the way to call shell scripts from python and write their stdout and stderr to file using logging. Here is my code: import logging import tempfile import shlex import os def run_shell_command(command_line): …

Read More
Run ffmpeg without outputting configuration information?

I'm invoking ffmpeg with subprocess.Popen, and trying to capture the stderr output and write it to logging. args = ['ffmpeg', '-i', path] if start: args += ['-ss', start] if end: args += ['-t', end] args += [ '-vcodec', 'copy', '-acodec', 'copy', '-scodec', 'copy', '-f', …

Read More