Kafka - This server is not the leader for that topic-partition

apache-kafka apache-kafka-streams
Rednam Nagendra picture Rednam Nagendra · Dec 12, 2017 · Viewed 32.3k times · Source

I have two broker kafka 0.10.2.0 cluster.Replication factor is 2. I am running 1.0.0 kafka stream application against this Kafka. In my kafka stream application, producer config has retries = 10 and retry.backoff.ms = 100

After running few minutes, I observed following logs in Kakfa server.log. Due to this Kafka stream application is throwing 'NOT_LEADER_FOR_PARTITION' exception.

What may be the possible reason? Please help me.

[2017-12-12 10:26:02,583] ERROR [ReplicaFetcherThread-0-1], Error for partition [__consumer_offsets,22] to broker 1:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is not the leader for that topic-partition. (kafka.server.ReplicaFetcherThread)

Answer

Gal Shaboodi picture Gal Shaboodi · Dec 12, 2017

Each topic is served by one or multiple Brokers - one is leader and the remaining brokers are followers.

A producer needs to send new messages to the leader Broker which internally replicate the data to all followers.

I assume, that your producer client does not connect to the correct Broker, its connect to a follower instead of the leader, and this follower rejects your send request.

Try to run ./kafka-topics.sh --zookeeper localhost:2181 --topic your_topic --describe

Topic:your_topic    PartitionCount:3    ReplicationFactor:1 Configs:retention.ms=14400000
Topic: your_topic   Partition: 0    Leader: 2   Replicas: 2 Isr: 2
Topic: your_topic   Partition: 1    Leader: 0   Replicas: 0 Isr: 0
Topic: your_topic   Partition: 2    Leader: 1   Replicas: 1 Isr: 1

In this example you can see that your_topic have 3 partitions meaning all 3 brokers are leaders of that topic each on different partition, s.t broker 2 is leader on partition 0 and broker 0 and broker 1 are followers on partition 0.

Related questions

Kafka: Consumer API vs Streams API

I recently started learning Kafka and end up with these questions. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world. How Stream is different …

Read More
Kafka INVALID_FETCH_SESSION_EPOCH

We are using a kafka broker setup with a kafka streams application that runs using Spring cloud stream kafka. Although it seems to run fine, we do get the following error statements in our log: 2019-02-21 22:37:20,253 INFO kafka-coordinator-heartbeat-thread | anomaly-timeline …

Read More
Kafka Streams with Spring Boot

I want to work with Kafka Streams real time processing in my spring boot project. So I need Kafka Streams configuration or I want to use KStreams or KTable, but I could not find example on the internet. I did …

Read More