Difference between retention configuration offsets.retention.minutes and log.retention.minutes

Question

What is the difference between the following two retention configurations?

• offsets.retention.minutes
• log.retention.minutes

I don't get how it differs or relate to each other. From my understanding, once the offset is removed, the record in the log is not accessible and vice versa. Is there something I have misunderstood?

Answer 1

The offset is a pointer to the most recent message that has been consumed by a consumer. So if you read 10 messages, the offset moves 10 places. offsets.retention.minutes allows you to move the offset back to the beginning if it isn't changed within a set period of time.

To visualise it, let's assume that we put the letters a to g (in that order) in a Kafka topic, all at different times. Before we start consuming the messages, the offset points to the oldest message:

OFFSET:    *
MESSAGES:  a b c d e f g

Now we consume 3 messages (a, b, c) so the offset moves:

OFFSET:          *
MESSAGES:  a b c d e f g

Now let's pretend we've set log.retention.minutes=10, and we put a and b into the topic 11 minutes ago, but the other messages were inserted more recently. We'd see:

OFFSET:          *
MESSAGES:      c d e f g

Now let's set offsets.retention.minutes=1, and pretend it has been 90 seconds since we last consumed anything. We'd see:

OFFSET:        *  
MESSAGES:      c d e f g

because c is now the oldest message on the topic (and the first that will be consumed).