Reading serial data in realtime in Python

Question

I am using a script in Python to collect data from a PIC microcontroller via serial port at 2Mbps.

The PIC works with perfect timing at 2Mbps, also the FTDI usb-serial port works great at 2Mbps (both verified with oscilloscope)

Im sending messages (size of about 15 chars) about 100-150x times a second and the number there increments (to check if i have messages being lost and so on)

On my laptop I have Xubuntu running as virtual machine, I can read the serial port via Putty and via my script (python 2.7 and pySerial)

The problem:

• When opening the serial port via Putty I see all messages (the counter in the message increments 1 by 1). Perfect!
• When opening the serial port via pySerial I see all messages but instead of receiving 100-150x per second i receive them at about 5 per second (still the message increments 1 by 1) but they are probably stored in some buffer as when I power off the PIC, i can go to the kitchen and come back and im still receiving messages.

Here is the code (I omitted most part of the code, but the loop is the same):

ser = serial.Serial('/dev/ttyUSB0', 2000000, timeout=2, xonxoff=False, rtscts=False, dsrdtr=False) #Tried with and without the last 3 parameters, and also at 1Mbps, same happens.
ser.flushInput()
ser.flushOutput()
While True:
  data_raw = ser.readline()
  print(data_raw)

Anyone knows why pySerial takes so much time to read from the serial port till the end of the line? Any help?

I want to have this in real time.

Thank you

Answer 1

You can use inWaiting() to get the amount of bytes available at the input queue.

Then you can use read() to read the bytes, something like that:

While True:
    bytesToRead = ser.inWaiting()
    ser.read(bytesToRead)

Why not to use readline() at this case from Docs:

Read a line which is terminated with end-of-line (eol) character (\n by default) or until timeout.

You are waiting for the timeout at each reading since it waits for eol. the serial input Q remains the same it just a lot of time to get to the "end" of the buffer, To understand it better: you are writing to the input Q like a race car, and reading like an old car :)