Kafka - difference between Log end offset(LEO) vs High Watermark(HW)

apache-kafka kafka-consumer-api kafka-producer-api
Shankar picture Shankar · Aug 29, 2016 · Viewed 12.7k times · Source

What is the difference between LEO and HW in Replica ( Leader Replica)?

Will they contain the same number? I can understand HW is the last committed message offset.

When LEO will be updated and how?

Answer

Matthias J. Sax picture Matthias J. Sax · Aug 31, 2016

The high watermark indicated the offset of messages that are fully replicated, while the end-of-log offset might be larger if there are newly appended records to the leader partition which are not replicated yet.

Consumers can only consumer messages up to the high watermark.

See this blog post for more details: http://www.confluent.io/blog/hands-free-kafka-replication-a-lesson-in-operational-simplicity/

Related questions

Understanding Kafka Topics and Partitions

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: When a producer is producing a message - it will specify the topic it wants to send the message to, is …

Read More
Is key required as part of sending messages to Kafka?

KeyedMessage<String, byte[]> keyedMessage = new KeyedMessage<String, byte[]>(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed messages, will it still work with delete.retention.…

Read More
How to write a file to Kafka Producer

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps: Started zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties Started Server bin/kafka-server-start.sh config/server.properties Created …

Read More