Run programs in background and redirect their outputs to file in real time

Question

I want to run several python scripts simultaneously in one bash session and to inspect their outputs respectively in real time. To fulfil this task, I wrote a simple bash script which is shown below:

#!/bin/bash
python 1.py > 1.output &
python 2.py > 2.output &
python 3.py > 3.output &

When I use cat 1.output command to check what have been printed in the middle of executing, however, nothing can be seen.

After thinking a while, I realise that the 1.output must be filled when 1.py finishes executing. In other words, the method which I used here is not a real time fashion.

You may propse that to wait for these python scripts' finish. The fact, unfortunately, is that all there python scripts are actually long run programs, they maybe finish after days or months, and that is why I want to inspect their outputs in real time.

Also, you may suggest me to modify the python scripts to print message to file directly instead of stdout. Sorry, the scripts here are too complex to modify, there are chucks of print function in them.

what can I do now?

Answer 1

The -u switch and the equivalent PYTHONUNBUFFERED environment variable forces stdout to be unbuffered. Try this:

#!/bin/bash
python -u 1.py > 1.output &
python -u 2.py > 2.output &
python -u 3.py > 3.output &

or

#!/bin/bash
export PYTHONUNBUFFERED=yes
python 1.py > 1.output &
python 2.py > 2.output &
python 3.py > 3.output &

Note that -u has side effects: read the doc to learn more.

Reference:

• https://docs.python.org/2/using/cmdline.html#cmdoption-u
• https://docs.python.org/2/using/cmdline.html#envvar-PYTHONUNBUFFERED