python pexpect clearing or flushing the line

Question

I am trying to clear any characters on the line after recovering from an unknown state, as in some cases they contain prompts and other keywords that I use in future expect method calls. I have tried multiple approaches with mixed results, as I keep running into non-expected behaviors.

Seen unexpected behaviors (using pexpect V3.3 with Python 2.7.9):

1. After performing the following code, when I subsequently try to read from the buffer, occasionally I see erratic behavior where only part of the accumulated characters were cleared. This reeks havoc on my downstream logic. I assume that this is because the whole thread is put to sleep for 5 seconds, so when it wakes up it does not have time to fetch the full incoming buffer before executing the read_nonblocking() command.

   time.sleep(5)
   flushedStuff += child.read_nonblocking(100000, timeout=0)
   

2. When I try to use an .expect call to flush the line in a non-blocking way, I found out that after a TIMEOUT exception the incoming buffer is not cleared. Its content can be found as expected in the child.before property, but it will also be parsed during the next .expect method call. So this does not flush the line at all! I also noticed that, read_nonblocking() does not read from the local buffer, but reads directly from the line through the OS, so it does not see this.

   try:
       child.expect("ZzqQJjSh_Impossible_String", timeout = 5)
   except pexpect.TIMEOUT:
       pass
   flushedStuff = child.before
   

So after all this, my current solution to provide a reliable way to flush the line is to extend the spawn class and add a method that does the following ... which accesses an undocumented property:

class ExtendedSpawn(pexpect.spawn):
    def flushBuffer(delay)
        try:
            # Greedily read in all the incoming characters
            self.expect("ZzqQJjSh_Impossible_String", timeout = delay)
        except pexpect.TIMEOUT:
            pass
        # Clear local input buffer inside the spawn Class
        self.buffer = self.string_type()
        return self.before

The above method can also be used for a non-blocking sleep command.

This seems to be way too complex of an approach for something that should be simple, not to mention I wasted a couple of days on it. Is there a better way of doing this?

Thanks!

Answer 1

To see the problem, you can run the following script. (A shell script will echo the numbers 1 to 10, and sleep 1 second after each number. pexpect will wait 2 seconds for a timeout and print the numbers seen.):

#!/usr/bin/env python
import pexpect
child = pexpect.spawn('/bin/sh -c "i=0; while [ $i -lt 10 ]:; do echo $((i=i+1)); sleep 1; done"')
while True:
    i=child.expect ([pexpect.TIMEOUT, pexpect.EOF], timeout=2)
    if i==0:
        print "before=%s" % child.before
    else:
        break

The output will look like this (We get also the numbers already read before):

before='1\r\n2\r\n'
before='1\r\n2\r\n3\r\n4\r\n'
before='1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n'
before='1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n8\r\n'
before='1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n8\r\n9\r\n10\r\n'

To solve the problem two lines can be added to the script:

#!/usr/bin/env python
import pexpect
child = pexpect.spawn('/bin/sh -c "i=0; while [ $i -lt 10 ]:; do echo $((i=i+1)); sleep 1; done"')
while True:
    i=child.expect ([pexpect.TIMEOUT, pexpect.EOF], timeout=2)
    if i==0:
        print "before=%s" % repr(child.before)
        if child.before:
            child.expect (r'.+')
    else:
        break

And the output will be as expected:

before='1\r\n2\r\n'
before='3\r\n4\r\n'
before='5\r\n6\r\n'
before='7\r\n8\r\n'
before='9\r\n10\r\n'