How to avoid a Broken Pipe error when printing a large amount of formatted data?

Thanasis Petsas picture Thanasis Petsas · Apr 3, 2013 · Viewed 22.9k times · Source

I am trying to print a list of tuples formatted in my stdout. For this, I use the str.format method. Everything works fine, but when I pipe the output to see the first lines using the head command a IOError occurs.

Here is my code:

# creating the data
data = []$
for i in range(0,  1000):                                            
  pid = 'pid%d' % i
  uid = 'uid%d' % i
  pname = 'pname%d' % i
  data.append( (pid, uid, pname) )

# find max leghed string for each field
pids, uids, pnames = zip(*data)
max_pid = len("%s" % max( pids) )
max_uid = len("%s" % max( uids) )
max_pname = len("%s" % max( pnames) )

# my template for the formatted strings
template = "{0:%d}\t{1:%d}\t{2:%d}" % (max_pid, max_uid, max_pname)

# print the formatted output to stdout
for pid, uid, pname in data:
  print template.format(pid, uid, pname)

And here is the error I get after running the command: python myscript.py | head

Traceback (most recent call last):
  File "lala.py", line 16, in <module>
    print template.format(pid, uid, pname)
IOError: [Errno 32] Broken pipe

Can anyone help me on this?

I tried to put print in a try-except block to handle the error, but after that there was another message in the console:

close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

I also tried to flush immediately the data through a two consecutive sys.stdout.write and sys.stdout.flush calls, but nothing happend..

Answer

Martijn Pieters picture Martijn Pieters · Apr 3, 2013

head reads from stdout then closes it. This causes print to fail, internally it writes to sys.stdout, now closed.

You can simply catch the IOError and exit silently:

try:
    for pid, uid, pname in data:
        print template.format(pid, uid, pname)
except IOError:
    # stdout is closed, no point in continuing
    # Attempt to close them explicitly to prevent cleanup problems:
    try:
        sys.stdout.close()
    except IOError:
        pass
    try:
        sys.stderr.close()
    except IOError:
        pass