How to send message to a particular partition in Kafka?

apache-kafka apache-zookeeper kafka-producer-api
Amr916 picture Amr916 · May 14, 2018 · Viewed 18.1k times · Source

I have created a topic that has many partitions. Using the console producer I want to send messages to particular partitions and view the through the console consumer. On the console producer I have tried this,

kafka-console-producer.bat --broker-list localhost:9092 --topic sample  --property parse.key=true --property key.separator=,

Send messages as,

key1,another-message

But I am just confused on whether key1 represents partition number.

Using the console consumer I viewed the messages,

kafka-console-consumer.bat --zookeeper localhost:2181 --topic sample

I want to view the messages according to the partitions. Is this the right way to view the messages on the console consumer? Can anyone please provide a clear understanding on this?

Answer

Katya Gorshkova picture Katya Gorshkova · May 14, 2018

The key is not the partition number but Kafka uses the key to specify the target partition. The default strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.

If you need a custom algorithm to map the messages to partitions you need to implement org.apache.kafka.clients.producer.Partitioner interface. The name of you class must be set as a partitioner.class property of the producer.

Update: You can also specify partition number directly in the ProducerRecord

Related questions

Error in Publishing message to Kafka topic

I am new to Kafka and try to setup enviroment for it . I am trying to run a single node Kafka but I am getting error in that. Following following steps on mac 1. brew install zookeeper 2. brew install kafka 3. zkServer …

Read More
Kafka : How to connect kafka-console-consumer to fetch remote broker topic content?

I have setup a kafka zookeeper and 3 brokers on one machine on ec2 with ports 9092..9094 and am trying to consume the topic content from another machine. The ports 2181 (zk), 9092, 9093 and 9094 (servers) are open to the consumer machine. I can even …

Read More
INFO Closed socket connection for client /127.0.0.1:48452 which had sessionid 0x15698f5ac360001 (org.apache.zookeeper.server.NIOServerCnxn)

I installed fresh zookeeper & kafka both. I started them both. Then when I want to see the list of topics with this command: bin/kafka-topics.sh --list --zookeeper localhost 2181 It gives me the socket connection closed. Here is the …

Read More