PySerial loses data
My problem is, that PySerial seems to lose some data packages and I don't know why.
I have two python scripts, the first one reads data from a text file and writes it to a microcontroller, where the data is manipulated. Then, the microcontroller sends the modified data over a different serial port back to the PC. (for clarification: I need two serial ports, because in the final application, the scripts will be running on different PCs. However, for testing purpose, it is easier to run both scripts on one PC)
So basically, my hardware setup looks like:
PC ----(serial port 1)----> microcontroller
<---(serial port 2)-----
After writing the data to the microcontroller I am expecting to get a certain amount of data bytes back. If I use a terminal program (like HyperTerminal) to monitor the data which is received, everything looks fine. However, if I try to read the data with a Python script, I only get a fraction of the expected data bytes.
For example:
+--------------------+--------------------+
| with HyperTerminal | with Python script |
+--------------------+--------------------+
| 1:W:00522 | 1:W:00522 |
| 1:W:00532 | 1:W:00532 |
| 1:W:00518 | 1:W:00522 |
| 1:W:00522 | 1:W:00526 |
| 1:W:00522 | 1:W:00514 |
| 1:W:00526 | 1:W:00520 |
| 1:W:00514 | 1:W:00514 |
| 1:W:00520 | 1:W:00522 |
| 1:W:00520 | 1:W:00526 |
| 1:W:00514 | 1:W:00520 |
| 1:W:00516 | 1:W:00526 |
| 1:W:00522 | 1:W:00520 |
| 1:W:00526 | 1:W:00524 |
| 1:W:00520 | 1:W:00526 |
| 1:W:00520 | 1:W:00532 |
| 1:W:00526 | 1:W:00506 |
| 1:W:00522 | 1:W:00520 |
| 1:W:00520 | 1:W:00526 |
| 1:W:00524 | 1:W:00524 |
| 1:W:00522 | 1:W:00526 |
| 1:W:00526 | 1:W:00514 |
| 1:W:00514 | 1:W:00522 |
| 1:W:00532 | 1:W:00520 |
| 1:W:00506 | 1:W:00510 |
| 1:W:00522 | 1:W:00506 |
| 1:W:00520 | |
| 1:W:00526 | |
| 1:W:00530 | |
| 1:W:00524 | |
| 1:W:00526 | |
| 1:W:00514 | |
| 1:W:00514 | |
| 1:W:00522 | |
| 1:W:00524 | |
| 1:W:00520 | |
| 1:W:00510 | |
| 1:W:00506 | |
+--------------------+--------------------+
As you can see, if I try to read from the serial port with my Python script, I am missing some data. Due to the fact, that I get the expected data if I use a terminal program, I assume, that my Python script has an error.
My Python script for sending data to the microcontroller looks like:
import serial
import re
import time
class digiRealTest():
def __init__(self):
#configure serial port
self.ser = serial.Serial(7, 9600, parity=serial.PARITY_NONE)
def main(self):
filepath = 'C:\\Users\\Bernhard\\Desktop\\TomatoView\\Qt\\test_output.txt'
with open(filepath, 'r') as content_file:
content = content_file.read().decode("hex")
for match in re.finditer('[\02](.*?)[\03]', content, re.S):
res = match.group(1)
complete = '\x02' + res + '\x03'
# time.sleep(0.3) <-- if i uncomment this line, it work's!!!
self.ser.write(complete)
if __name__ == "__main__": #wenn Modul direkt ausgefuehrt wird
d = digiRealTest()
d.main()
My Python script for receiving the data sent from the microcontroller:
import Queue
import threading
import serial
class mySerial(threading.Thread):
def __init__(self, queue):
super(mySerial, self).__init__()
self.queue = queue #the received data is put in a queue
self.buffer = ''
#configure serial connection
self.ser = serial.Serial(timeout = 0, port = 3, baudrate=9600)
def run(self):
while True:
self.buffer += self.ser.read(self.ser.inWaiting()) #read all char in buffer
if '\n' in self.buffer: #split data line by line and store it in var
var, self.buffer = self.buffer.split('\n')[-2:]
self.queue.put(var) #put received line in the queue
time.sleep(0.01) #do not monopolize CPU
class Base():
def __init__(self):
self.queue = Queue.Queue(0) #create a new queue
self.ser = mySerial(self.queue)
self.ser.start() #run thread
def main(self ):
while(True):
try:
var = self.queue.get(False) #try to fetch a value from queue
except Queue.Empty:
pass #if it is empty, do nothing
else:
print(var)
if __name__ == '__main__':
b = Base()
b.main()
I don't know why, but if I uncomment the line #time.sleep(0.3) in the sending script, everything works fine and I get the expected data back. So to me, it seems, that somehow my script for reading the data from the serial port is too slow....but why?
Answer
Your receiver's split is throwing lines away. This should work better:
def run(self):
while True:
self.buffer += self.ser.read(self.ser.inWaiting()) #read all char in buffer
while '\n' in self.buffer: #split data line by line and store it in var
var, self.buffer = self.buffer.split('\n', 1)
self.queue.put(var) #put received line in the queue
time.sleep(0.01) #do not monopolize CPU
You could also get rid of the polling/sleeping by getting rid of the timeout when you create the serial port and ask for 1 byte when there is nothing in the queue.
class mySerial(threading.Thread):
def __init__(self, queue):
super(mySerial, self).__init__()
self.queue = queue #the received data is put in a queue
self.buffer = ''
#configure serial connection
self.ser = serial.Serial(port = 3, baudrate=9600)
def run(self):
while True:
self.buffer += self.ser.read(self.ser.inWaiting() or 1) #read all char in buffer
while '\n' in self.buffer: #split data line by line and store it in var
var, self.buffer = self.buffer.split('\n', 1)
self.queue.put(var) #put received line in the queue