I'm using 0.9.0.0 version of Kafka and I want to count the number of messages in a topic without using the admin script kafka-console-consumer.sh.
I have tried all the commands in the answer Java, How to get number of messages in a topic in apache kafka but none are yielding the result. Can anyone help me out here?
You could try to execute the command below:
bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092,localhost:9093,localhost:9094 --topic test-topic --time -1
Then, sum up all the counts for each partition.
Updated: Java implementation
Properties props = new Properties();
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
......
try (final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(Arrays.asList("your_topic"));
Set<TopicPartition> assignment;
while ((assignment = consumer.assignment()).isEmpty()) {
consumer.poll(Duration.ofMillis(100));
}
final Map<TopicPartition, Long> endOffsets = consumer.endOffsets(assignment);
final Map<TopicPartition, Long> beginningOffsets = consumer.beginningOffsets(assignment);
assert (endOffsets.size() == beginningOffsets.size());
assert (endOffsets.keySet().equals(beginningOffsets.keySet()));
Long totalCount = beginningOffsets.entrySet().stream().mapToLong(entry -> {
TopicPartition tp = entry.getKey();
Long beginningOffset = entry.getValue();
Long endOffset = endOffsets.get(tp);
return endOffset - beginningOffset;
}).sum();
System.out.println(totalCount);
}