Apache Kafka - linger.ms and batch.size settings

apache-kafka kafka-producer-api
Vivek Agarwal picture Vivek Agarwal · Jul 25, 2018 · Viewed 13.6k times · Source

In kafka producer settings, what is the expected behavior if you have linger.ms set to 0 and a non zero batch.size? How long will the producer wait for batch.size to be reached before sending the messages? Will it wait forever till the size of messages is less than specified batch size OR since linger.ms is zero, it will not do any batching and just send every request?

Answer

dossani picture dossani · Jul 25, 2018

No, this does not mean that the producer will wait for the batch to become full. The producer will send half-full batches and even batches with just a single message in them.

Therefore, setting the batch size too large will not cause delays in sending messages; it will just use more memory for the batches. Setting the batch size too small will add some overhead because the producer will need to send messages more frequently.

By default (linger.ms=0), the producer will send messages as soon as there is a sender thread available to send them, even if there’s just one message in the batch.

Hope it helps.

Thanks.

Related questions

How to check whether Kafka Server is running?

I want to ensure whether kafka server is running or not before starting production and consumption jobs. It is in windows environment and here's my kafka server's code in eclipse... Properties kafka = new Properties(); kafka.setProperty("broker.id", "1"); kafka.setProperty("…

Read More
Understanding Kafka Topics and Partitions

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: When a producer is producing a message - it will specify the topic it wants to send the message to, is …

Read More
Is key required as part of sending messages to Kafka?

KeyedMessage<String, byte[]> keyedMessage = new KeyedMessage<String, byte[]>(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed messages, will it still work with delete.retention.…

Read More