Linux socket: How to make send() wait for recv()

Question

I am making a simple client-server application using TCP protocal.

I Know that by default. recv() will block until the other side call a send() to this socket. But is it possible that send() block itself until the other side has recv()ed the msg instead of keeping send()ing to outgoing queue and later to find the other side recv() got a whole bunch of msg sent by multiple send()s

In other words. Is it possible to let every send() wait for the other side's recv() before it can call another send()?

To ilustate my question. I'll post a simple code here:

client.c

#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <netdb.h>
#include <stdio.h>
#include <poll.h>
#include <string.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <arpa/inet.h> 

int main(int argc, char *argv[])
{
    int sockfd = 0;
    char sendBuff[1024];
    struct sockaddr_in serv_addr;
    int i;

    if(argc != 2)
    {
        printf("\n Usage: %s <ip of server> \n",argv[0]);
        return 1;
    } 

    if((sockfd = socket(AF_INET, SOCK_STREAM, 0)) < 0)
    {
        printf("\n Error : Could not create socket \n");
        return 1;
    } 

    memset(&serv_addr, '0', sizeof(serv_addr)); 

    serv_addr.sin_family = AF_INET;
    serv_addr.sin_port = htons(5000); 

    if(inet_pton(AF_INET, argv[1], &serv_addr.sin_addr)<=0)
    {
        printf("\n inet_pton error occured\n");
        return 1;
    } 

    if( connect(sockfd, (struct sockaddr *)&serv_addr, sizeof(serv_addr)) < 0)
    {
        printf("\n Error : Connect Failed \n");
        return 1;
    }

    do{
        memset(sendBuff, '\0', sizeof(sendBuff));
        sprintf(sendBuff, "This is line %d", i);
        send(sockfd, sendBuff, strlen(sendBuff), 0);
        //sleep(1);
    }while(++i<100);

    return 0;
}

server.c

#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <time.h> 

int main(int argc, char *argv[])
{
    int listenfd = 0, connfd = 0;
    struct sockaddr_in serv_addr; 

    char sendBuff[1025];
    char recvBuff[100];

    int i = 0;

    listenfd = socket(AF_INET, SOCK_STREAM, 0);
    memset(&serv_addr, '0', sizeof(serv_addr));
    memset(sendBuff, '0', sizeof(sendBuff)); 

    serv_addr.sin_family = AF_INET;
    serv_addr.sin_addr.s_addr = htonl(INADDR_ANY);
    serv_addr.sin_port = htons(5000); 

    bind(listenfd, (struct sockaddr*)&serv_addr, sizeof(serv_addr)); 

    listen(listenfd, 10); 

    connfd = accept(listenfd, (struct sockaddr*)NULL, NULL); 

    do{  
        memset(recvBuff, '\0', sizeof(recvBuff));
        recv(connfd, recvBuff, sizeof(recvBuff),0);
        printf( "%s\n", recvBuff);
    }while(++i<100); 

    return 0;
} 

What I expect on the server side outcome is to print:

This is line 0
This is line 1
This is line 2
This is line 3
...

However, the actual outcome is like this:

This is line 0
This is line 1This is line 2This is line3This is line 4
This is line 5This is line 6This is line 7This is line 8This is line 9This is line 10
This is line 11This is line 12...

However this is easy to explain: when the client side issued a send(), it didn't wait for the server side's recv() to finish, and for some reason, the server side recv() loop is slower than client side's send(). Thus several send()s on client side may get piled together and received by server as a whole. (is my explanation right?)

Actually there seems to be a very silly and loose solution. just add a sleep(1)(as I commented out) after each send() in the loop. I know this will make the code very inefficient and if the recv() loop will take longer time to implement some other complicated actions( This is obviously unpredictable when the program gets large) which will take longer than 1 sec. This solution fails!

So is there a better solid way to let the two sides communicate with each other to ensure the msg sent by one single send() received by a single recv()?

Answer 1

The only way for the sender to know that the receiver has received the message is for the receiver to reply.

In other words, if you want to keep the sender and receiver in sync, you need to implement your own explicit flow control.

It is true that the TCP protocol requires that receivers acknowledge received packets, so you might think that it is possible for the receiver to simply ask the TCP implementation if the data has been acknowledged. This is generally possible, although I don't believe there is a cross-platform way of doing so. On Linux, for example, you can use the ioctl code SIOCOUTQ to check the size of the outbound buffer. But as far as I know there is no mechanism for blocking until the outbound buffer is empty, so the best you could do would be to periodically poll. That is not satisfactory.

In any event, the result would be misleading because there is also a buffer on the receiving end, and if processing is the bottleneck, the receiving buffer will be the first to fill up. Low-level data acknowledgement only indicates that the data has reached the receiving TCP stack (that is, the low-level IP interface, generally inside of the kernel). It does not indicate that the data has been processed by the receiving process, and there is no mechanism for the sender to query the usage of the receiving buffer.

Furthermore, waiting for the acknowledge message to arrive at the sending end will generally slow communications down unnecessarily, since it adds a transmission delay (receiver → sender) to every transaction. This is not so dramatic as your sleep solution, but it is somewhat similar.

If your messages are very short, you might be running into the Nagle algorithm, which causes small data packets to be retained for a small amount of time by the sender, in the hopes that several packets can be consolidated. You can turn the Nagle algorithm off for a TCP connection using setsockopt, with the TCP_NODELAY option. But don't do this unless you are sure that it will be useful. (The canonical use of TCP_NODELAY is a telnet-type connection, where the data consists of individual keystrokes. )