How to check if Kafka Consumer is ready

apache-kafka rebalancing
Nagireddy Hanisha picture Nagireddy Hanisha · Jan 3, 2018 · Viewed 8.5k times · Source

I have Kafka commit policy set to latest and missing first few messages. If I give a sleep of 20 seconds before starting to send the messages to the input topic, everything is working as desired. I am not sure if the problem is with consumer taking long time for partition rebalancing. Is there a way to know if the consumer is ready before starting to poll ?

Answer

donm picture donm · Jan 10, 2018

  • You can use consumer.assignment(), it will return set of partitions and verify whether all of the partitions are assigned which are available for that topic.

  • If you are using spring-kafka project, you can include spring-kafka-test dependancy and use below method to wait for topic assignment , but you need to have container. ContainerTestUtils.waitForAssignment(Object container, int partitions);

Related questions

List all kafka topics

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?

Read More
Purge Kafka Topic

Is there a way to purge the topic in kafka? I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error: kafka.common.InvalidMessageSizeException: invalid message size Increasing the …

Read More
Is there a way to delete all the data from a topic or delete the topic before every run?

Is there a way to delete all the data from a topic or delete the topic before every run? Can I modify the KafkaConfig.scala file to change the logRetentionHours property? Is there a way the messages gets deleted as …

Read More