Turn off buffering

Question

Where is the buffer in this following ... and how do I turn it off?

I am writing out to stdout in a python program like so:

for line in sys.stdin:
    print line

There is some buffering going on here:

tail -f data.txt | grep -e APL | python -u Interpret.py

I tried the following to shake off possible buffering ... with no luck:

• as above using the -u flag with python invocation
• calling sys.stdout.flush() after each sys.stdout.write() call ... all of these create a buffered stream with python waiting something like a minute to print out the first few lines.

• used the following modified command:

  stdbuf -o0 tail -f data.txt | stdbuf -o0 -i0 grep -e APL | stdbuf -i0 -o0 python -u Interpret.py

To benchmark my expectations, I tried:

tail -f data.txt | grep -e APL 

This produces a steady flow of lines ... it surely is not as buffered as the python command.

So, how do I turn off buffering? ANSWER: It turns out there is buffering on both ends of the pipe.

Answer 1

file.readlines() and for line in file have internal buffering which is not affected by -u option (see -u option note). Use

while True:
   l=sys.stdin.readline()
   sys.stdout.write(l)

instead.

By the way, sys.stdout is line-buffered by default if it points to terminal and sys.stderr is unbuffered (see stdio buffering).