Can I ignore org.apache.kafka.common.errors.NotLeaderForPartitionExceptions?

exception-handling apache-kafka kafka-producer-api
Hector picture Hector · Apr 28, 2016 · Viewed 7.8k times · Source

My Apache Kafka producer (0.9.0.1) intermittently throws a

org.apache.kafka.common.errors.NotLeaderForPartitionException

My code that performs the Kafka send resembles this

final Future<RecordMetadata> futureRecordMetadata = KAFKA_PRODUCER.send(new ProducerRecord<String, String>(kafkaTopic, UUID.randomUUID().toString(), jsonMessage));

try {
    futureRecordMetadata.get();
} catch (final InterruptedException interruptedException) {
    interruptedException.printStackTrace();
    throw new RuntimeException("sendKafkaMessage(): Failed due to InterruptedException(): " + sourceTableName + " " + interruptedException.getMessage());
} catch (final ExecutionException executionException) {
    executionException.printStackTrace();
    throw new RuntimeException("sendKafkaMessage(): Failed due to ExecutionException(): " + sourceTableName + " " + executionException.getMessage());
}

I catch NotLeaderForPartitionException within the catch (final ExecutionException executionException) {} block.

Is it OK to ignore this particular exception?

Has my Kafka message been sent successfully?

Answer

Matthias J. Sax picture Matthias J. Sax · Apr 30, 2016

If you receive NotLeaderForPartitionException, your data was not written successfully.

Each topic partition is stored by one or multiple Brokers (with one leader; the remaining brokers are called followers) depending on your replication factor. A producer needs to send new messages to the leader Broker (data replication to followers happens internally).

Your producer client does not connect to the correct Broker, ie, to a follower instead of the leader (or to a broker that is not even a follower any longer), and this broker rejects your send request. This can happen if the leader changed but the producer still has outdated cached metadata about which broker is the leader for a partition.

Related questions

Catch multiple exceptions in one line (except block)

I know that I can do: try: # do something that may fail except: # do this if ANYTHING goes wrong I can also do this: try: # do something that may fail except IDontLikeYouException: # say please except YouAreTooShortException: # stand on a ladder …

Read More
Get connection string from App.config

var connection = ConnectionFactory.GetConnection( ConfigurationManager.ConnectionStrings["Test"] .ConnectionString, DataBaseProvider); And this is my App.config: <?xml version="1.0" encoding="utf-8" ?> <configuration> <connectionStrings> <add name="Test" connectionString="Data Source=.;Initial Catalog=OmidPayamak;Integrated Security=True" …

Read More
About catching ANY exception

How can I write a try/except block that catches all exceptions?

Read More