Performance issues when using the raw TCP API of lwIP
I use lwIP to add networking functionality to my system. On my platform i built a buffer that i want to send every time it is full. This can happen quite quickly. The system is directly connected to a switch in a private LAN. Initially the sending of data had a very large time gap in between of 2 seconds. Additionally the packets had a size of 720 bytes if my memory serves me correctly. The used buffer currently has capacity for about 20000 bytes and I might decide to increase this in the future. The network has 100 mbit speeds and I would like to come close to these speeds on my platform.
When searching for the cause of the slow speeds I ended up at lwIP's configuration. Prior to that I altered my sending mechanism. I use the raw lwIP API and at present I write the data as follows:
tcp_write(<pcb>, (const void*) data, <bytes>, TCP_WRITE_FLAG_COPY);
//<bytes> is at most tcp_sndbuf(<pcb>)
I know the copy flag creates a performance hit, but this is added because I don't want to overwrite data before it's actually sent. (and the flag isn't the main problem but something to polish once it's working properly) In a prior solution i omitted the flag and simply waited for all bytes to be ACK'd (after forcing data to be send after writing by calling tcp_output()) by using the callback function. (This might be worse performance wise and I dont think it's related)
I played a little with the settings in the of lwIP and that seemed to make some difference. I think the window size especially made a difference although I'm not quite sure. At the moment I increased the window size significantly and even though I get a burst of packets with about 2ms between them (instead of 2s!) this is followed with a long period of "nothing" and then a burst again. I want it to continuously send at the speed it should be capable of which should be 100 mbit at most but atleast 10 mbit is not weird to expect, right?
I loaded up wireshark to see what was happening.
192.168.1.26 is my desktop computer running windows. 192.168.1.192 is the embedded system that's using lwIP.
Initially I send a start request from the desktop to the lwIP system, letting the system know it should start sending the buffer each time it is full. In case it is relevant, this is the corresponding part of the trace:
5 2.007754 192.168.1.26 192.168.1.192 TCP 61 [TCP segment of a reassembled PDU]
6 2.008196 192.168.1.192 192.168.1.26 TCP 60 patrolview > afs3-fileserver [SYN] Seq=0 Win=65535 Len=0 MSS=1400
7 2.226238 192.168.1.192 192.168.1.26 TCP 60 afs3-fileserver > 50015 [ACK] Seq=1 Ack=8 Win=65528 Len=0
13 4.976858 192.168.1.192 192.168.1.26 TCP 60 patrolview > afs3-fileserver [ACK] Seq=1 Ack=1 Win=65535 Len=0
22 6.976572 192.168.1.192 192.168.1.26 TCP 60 [TCP segment of a reassembled PDU]
23 7.177903 192.168.1.26 192.168.1.192 TCP 54 50015 > afs3-fileserver [ACK] Seq=8 Ack=2 Win=64399 Len=0
I believe this is alright although i'm not certain. Anyway, after this the actual sending happens. The relevant trace looks as follows The start time is 207.992115 which should be considered the starting time. The difference between that and the 7.177903 is expected:
2578 207.992115 192.168.1.192 192.168.1.26 Gryphon 1422 - Invalid -
2581 208.194336 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=1369 Win=64400 Len=0
2582 208.195880 192.168.1.192 192.168.1.26 TCP 1422 [TCP segment of a reassembled PDU]
2583 208.197035 192.168.1.192 192.168.1.26 TCP 1422 [TCP segment of a reassembled PDU]
2584 208.197134 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=4105 Win=64400 Len=0
2585 208.198712 192.168.1.192 192.168.1.26 TCP 1422 [TCP segment of a reassembled PDU]
2586 208.199867 192.168.1.192 192.168.1.26 TCP 1422 [TCP segment of a reassembled PDU]
2587 208.199965 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=6841 Win=64400 Len=0
2588 208.200927 192.168.1.192 192.168.1.26 TCP 1314 [TCP segment of a reassembled PDU]
2590 208.397469 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
It seems that I currently send things quicker than the desktop is ACKing. The traffic after the trace above shows as black bars and looks like:
2591 208.399051 192.168.1.192 192.168.1.26 TCP 1422 [TCP Previous segment lost] [TCP segment of a reassembled PDU]
2592 208.399136 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2590#1] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2593 208.400208 192.168.1.192 192.168.1.26 Gryphon 1422
2594 208.400285 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2590#2] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2595 208.401361 192.168.1.192 192.168.1.26 Gryphon 1422 - Invalid -
2596 208.401445 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2590#3] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2597 208.402425 192.168.1.192 192.168.1.26 Gryphon 1314
2598 208.402516 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2590#4] afs3-fileserver > patrolview [ACK] Seq=1 Ack=8101 Win=63140 Len=0
2599 208.403588 192.168.1.192 192.168.1.26 Gryphon 1422 [TCP Fast Retransmission] Command response
2600 208.403685 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=14833 Win=64400 Len=0
2605 209.992237 192.168.1.192 192.168.1.26 Gryphon 1422 - Invalid -
2607 210.200219 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2608 210.201819 192.168.1.192 192.168.1.26 Gryphon 1422 [TCP Previous segment lost] - Invalid -
2609 210.201903 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2607#1] afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2609 210.201903 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2607#1] afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2611 210.203070 192.168.1.26 192.168.1.192 TCP 54 [TCP Dup ACK 2607#2] afs3-fileserver > patrolview [ACK] Seq=1 Ack=16201 Win=63032 Len=0
2955 345.001223 192.168.1.192 192.168.1.26 Gryphon 1422 [TCP Retransmission]
Now after this point there is a huge delay that I cannot explain. The next packets arrive at 345 seconds, this is a 135 second difference. (although in most cases it was a bit less, but still wayy too high) It starts as follows:
2955 345.001223 192.168.1.192 192.168.1.26 Gryphon 1422 [TCP Retransmission]
2958 345.001707 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=20305 Win=64400 Len=0
2959 345.003336 192.168.1.192 192.168.1.26 TCP 1422 [TCP segment of a reassembled PDU]
2960 345.004395 192.168.1.192 192.168.1.26 TCP 1314 [TCP segment of a reassembled PDU]
2961 345.004494 192.168.1.26 192.168.1.192 TCP 54 afs3-fileserver > patrolview [ACK] Seq=1 Ack=22933 Win=64400 Len=0
etc.
Later on a similar problems occur although the mentioned delay is shorter. My question is: How can I fix the problem of the slow sending from my platform and how should i configure my lwIP settings to expect decent/good results? I want to send the data at fast speeds. (my network is capable of 100Mbps, the closer the better) I think i currently messed up my settings entirely but I am not sure how to finetune them for my needs. Here are some (hopefully) relevant settings from my lwipopts.h
file:
#define MEM_SIZE 65000
#define PBUF_POOL_SIZE 1024
#define IP_FRAG_USES_STATIC_BUF 0
#define TCP_WND 65535
#define TCP_MSS 1400
#define TCP_SND_BUF 65535
#define PBUF_POOL_BUFSIZE LWIP_MEM_ALIGN_SIZE(1528)
#define LWIP_TCP_KEEPALIVE 1
#define LWIP_SO_RCVTIMEO 1
Answer
I ran into similar problems using beaglebone and following setting
#define MEM_SIZE (1024 * 1024) /* 1MiB */
#define MEMP_NUM_PBUF 1024
#define MEMP_NUM_TCP_PCB 32
#define PBUF_POOL_SIZE 1024
#define TCP_MSS 1460
#define TCP_WND (4*TCP_MSS)
#define TCP_SND_BUF 65535
#define TCP_OVERSIZE TCP_MSS
#define TCP_SND_QUEUELEN 512
#define MEMP_NUM_TCP_SEG 512
Using a tcp_sent polled function I basically checked how many bytes I have in the buffer and filled them in the polled function itself immediately with another samples. This was to check the throughout
I was quite surprised that wire shark shown bursts of packets during roughly few milliseconds and then 700ms of nothing.
Going deep into the stack I've found, that this happens exactly at the moment there are 65535 bytes sent (or roughly around this).
All that was solved by DISABLING MEMORY SANITY CHECK
Look in your lwipopts.h, whether by chance you do not define somewhere:
#define MEMP_SANITY_CHECK 1
If so, remove that line or set it to zero. As such, my packet sending performance did not increase (I'm still around 11Mbits when firing on maximum speed the data), but the total throughput increased considerably as the time between two sent packet is now constant and takes roughly 100us.
It needs to be said, that this still does not resolve the issue of having only 11MBits on 100MBit line completely dedicated only to this equipment