How can I improve PySerial read speed

Question

I'm currently building a machine that uses an Arduino Mega2560 as its main controller. The Arduino is connected to over serial, gets a command, executes it and spits out a bunch of measurement data every 1ms. I have a Raspberry Pi running Python to give the user a nice GUI to send the command, and to present the data in a readable form.

The problem I face: the Arduino is able to spit out 15 byte of data each millisecond (so that's only 15kbyte/s), but the code I'm running can only cope with about 15 byte each 10 milliseconds, so 1.5kB/s.

When I run cat /dev/ttyACM0 > somefile, I nicely see all datapoints.

I have the following slimmed down Python code

# Reset Arduino by starting serial
microprocBusy = True
serialPort = serial.Serial("/dev/ttyACM0", baudrate=460800, timeout=0)
time.sleep(0.22);
serialPort.setDTR(False);
time.sleep(0.22);
serialPort.setDTR(True);
time.sleep(0.10);

logfile = open(logfilenamePrefix + "_" + datetime.datetime.now().isoformat() + '.txt', 'a')

# Bootloader has some timeout, we need to wait for that
serialPort.flushInput()
while(serialPort.inWaiting() == 0):
    time.sleep(0.05)

# Wait for welcome message
time.sleep(0.1)
logfile.write(serialPort.readline().decode('ascii'))
logfile.flush()

# Send command
serialPort.write((command + '\n').encode('ascii'))

# Now, receive data
while(True):
    incomingData = serialPort.readline().decode('ascii')
    logfile.write(incomingData)
    logfile.flush() 

    if(incomingData[:5] == "FATAL" or incomingData[:6] == "HALTED" or incomingData[:5] == "RESET"):
        break;
    elif(incomingData[:6] == "RESULT"):
            resultData = incomingData;

logfile.flush() 

When I run this, the first ~350 datapoints come in, then I see some mangled data and miss about 2000 datapoints, after which I see another 350 or so datapoints. The CPU usage is at 100% during the process

What is going wrong? Is PySerial poorly optimized, or is there some mistake in my code I missed? I could just run cat /dev/ttyACM0 > somefile from Python and then read that file, but that's not really a nice solution, is it?

Thanks a lot :)

Answer 1

I've switched from PySerial to PyTTY, which solves my problem. Just plugging it into this code (with some small changes, like replacing serialPort.inWaiting() == 0 by serialPort.peek() == b'' for example) makes my code able to handle the datastream and not get above 50% CPU usage, which means it is at least 10x as fast. I'm still using PySerial to set the DTR lines though.

So, I guess the answer to the question is that indeed PySerial is indeed poorly optimised.