How can I get the last/end offset of a kafka topic partition?

java apache-kafka kafka-consumer-api
Neptune picture Neptune · Jul 18, 2016 · Viewed 58.8k times · Source

I'm writing a kafka consumer using Java. I want to keep the real time of the message, so if there are too many messages waiting for consuming, such as 1000 or more, I should abandon the unconsumed messages and start consuming from the last offset.

For this problem, I try to compare the last committed offset and the end offset of a topic(only 1 partition), if the difference between these two offsets is larger than a certain amount, I will set the last committed offset of the topic as next offset so that I can abandon those redundant messages.

Now my problem is how to get the end offset of a topic, some people say I can use old consumer, but it's too complicated, do new consumer has this function?

Answer

lynn picture lynn · Jul 19, 2016

The new consumer is also complicated.

//assign the topic consumer.assign();

//seek to end of the topic consumer.seekToEnd();

//the position is the latest offset consumer.position();

Related questions

What determines Kafka consumer offset?

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it …

Read More
understanding consumer group id

I did fresh installation of Apache Kafka 0.10.1.0. I was able to send / receive messages on command prompt. While using Producer / Consumer Java Example, I am not able to know group.id parameter on Consumer Example. Let me know on how …

Read More
Kafka consumer offsets out of range with no configured reset policy for partitions

I'm receiving exception when start Kafka consumer. org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions{test-0=29898318} I'm using Kafka version 9.0.0 with Java 7.

Read More