is kafka producer capable of sending multiple messages in a list?

producer apache-kafka
Jeff Gong picture Jeff Gong · Jul 7, 2015 · Viewed 8k times · Source

In Kafka v 0.8.2, is it still possible/beneficial to send a list of ProducerRecords as a producer as opposed to only a single one at a time? Examining the API it seems like org.apache.kafka.clients.producer.KafkaProducer does not have the ability to send multiple producerRecords using one send call.

However, in the producer under javaapi, it's possible to send a list of keyedMessages. Can someone explain the difference or point me in the right direction? Much appreciated.

Answer

Lundahl picture Lundahl · Jul 8, 2015

The producer under javaapi is the old implementation which has been superseded by the new one in clients. The new producer implementation can still batch messages together but will do this in the background and when doing a call to send you'll get a future back for every message.

Batch size and latency can be controlled with parameters "batch.size" and "linger.ms". See documentation for details.

Related questions

Leader Not Available Kafka in Console Producer

I am trying to use Kafka. All configurations are done properly but when I try to produce message from console I keep getting the following error WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE} (…

Read More
Topic can't be found when producing messages: UNKNOWN_TOPIC_OR_PARTITION

I have a two-nodes Kafka cluster (EC2 instances) where each node is used as a separate broker. When I run a producer on the leader instance with the following command: kafka-console-producer.sh --broker-list localhost:9092 --topic test I get the following …

Read More
Interact with kafka docker container from outside of docker host

I have built a kafka docker container and orchestrate it using docker-compose. Calling docker ps I get the following putput: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 5bde6f76246e hieutrtr/docker-kafka:0.0.1 "/start.sh" About an hour ago Up …

Read More