What determines Kafka consumer offset?

java apache-kafka kafka-consumer-api distributed-computing
Asif Iqbal picture Asif Iqbal · Sep 4, 2015 · Viewed 92.5k times · Source

I am relatively new to Kafka. I have done a bit of experimenting with it, but a few things are unclear to me regarding consumer offset. From what I have understood so far, when a consumer starts, the offset it will start reading from is determined by the configuration setting auto.offset.reset (correct me if I am wrong).

Now say for example that there are 10 messages (offsets 0 to 9) in the topic, and a consumer happened to consume 5 of them before it went down (or before I killed the consumer). Then say I restart that consumer process. My questions are:

  1. If the auto.offset.reset is set to earliest, is it always going to start consuming from offset 0 ?

  2. If the auto.offset.reset is set to latest, is it going to start consuming from offset 5 ?

  3. Is the behaviour regarding this kind of scenario always deterministic ?

Please don't hesitate to comment if anything in my question is unclear.

Answer

serejja picture serejja · Sep 4, 2015

It is a bit more complex than you described.
The auto.offset.reset config kicks in ONLY if your consumer group does not have a valid offset committed somewhere (2 supported offset storages now are Kafka and Zookeeper), and it also depends on what sort of consumer you use.

If you use a high-level java consumer then imagine following scenarios:

  1. You have a consumer in a consumer group group1 that has consumed 5 messages and died. Next time you start this consumer it won't even use that auto.offset.reset config and will continue from the place it died because it will just fetch the stored offset from the offset storage (Kafka or ZK as I mentioned).

  2. You have messages in a topic (like you described) and you start a consumer in a new consumer group group2. There is no offset stored anywhere and this time the auto.offset.reset config will decide whether to start from the beginning of the topic (earliest) or from the end of the topic (latest)

One more thing that affects what offset value will correspond to earliest and latest configs is log retention policy. Imagine you have a topic with retention configured to 1 hour. You produce 5 messages, and then an hour later you post 5 more messages. The latest offset will still remain the same as in previous example but the earliest one won't be able to be 0 because Kafka will already remove these messages and thus the earliest available offset will be 5.

Everything mentioned above is not related to SimpleConsumer and every time you run it, it will decide where to start from using the auto.offset.reset config.

If you use Kafka version older than 0.9, you have to replace earliest, latest with smallest,largest.

Related questions

How can I get the last/end offset of a kafka topic partition?

I'm writing a kafka consumer using Java. I want to keep the real time of the message, so if there are too many messages waiting for consuming, such as 1000 or more, I should abandon the unconsumed messages and start consuming …

Read More
understanding consumer group id

I did fresh installation of Apache Kafka 0.10.1.0. I was able to send / receive messages on command prompt. While using Producer / Consumer Java Example, I am not able to know group.id parameter on Consumer Example. Let me know on how …

Read More
Kafka consumer offsets out of range with no configured reset policy for partitions

I'm receiving exception when start Kafka consumer. org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions{test-0=29898318} I'm using Kafka version 9.0.0 with Java 7.

Read More