How to determine a Kafka consumer's offset

vegemite4me picture vegemite4me · Mar 2, 2017 · Viewed 7.9k times · Source

We have an issue where it appears that a Kafka consumer is not receiving messages published to a topic. (I say appears as I have not yet got to the bottom of this, and I could be wrong.)

I am using Spring for Apache Kafka and my consumer is actually a method annotated with @KafkaListener.

This issue is intermittent and I am having trouble recreating it.

Is there a way for me to look at the Kafka broker's logs, or any other tool to help me find out the offset for my consumer? I want concrete evidence that my consumer is receiving the message or not.

Answer

Sönke Liebau picture Sönke Liebau · Mar 2, 2017

Take a look at the kafka-consumer-groups tool, which can be used to check offsets and lag of consumers (consumer has to be active at the time you run this command).

./kafka-consumer-groups --bootstrap-server 127.0.0.1:9092 --new-consumer --describe --group console-consumer-55936

GROUP                          TOPIC                          PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             OWNER
console-consumer-55936         test                           0          6               6               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           1          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           2          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           3          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           4          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           5          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           6          1               1               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           7          2               2               0               consumer-1_/192.168.0.83
console-consumer-55936         test                           8          1               1               0               consumer-1_/192.168.0.83

This should allow you to track whether anything is actually being consumed or not.