Topic can't be found when producing messages: UNKNOWN_TOPIC_OR_PARTITION

apache-kafka consumer producer
saloua picture saloua · Jan 9, 2017 · Viewed 28.7k times · Source

I have a two-nodes Kafka cluster (EC2 instances) where each node is used as a separate broker. When I run a producer on the leader instance with the following command:

kafka-console-producer.sh   --broker-list localhost:9092 --topic test

I get the following errors.

test message [2017-01-09 13:22:39,483] WARN Error while fetching metadata with correlation id 0 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13:22:39,562] WARN Error while fetching metadata with correlation id 1 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) [2017-01-09 13:22:39,663] WARN Error while fetching metadata with correlation id 2 : {test=UNKNOWN_TOPIC_OR_PARTITION} (org.apache.kafka.clients.NetworkClient) ...

Listing the topics with kafka-topics.sh shows that the topic exists.

The description of the topic with:

kafka-topics.sh --zookeeper localhost:2181 --describe --topic test

returns

 Topic:test PartitionCount:8    ReplicationFactor:1 Configs:
     Topic: test    Partition: 0    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 1    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 2    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 3    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 4    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 5    Leader: 2   Replicas: 2 Isr: 2
     Topic: test    Partition: 6    Leader: 1   Replicas: 1 Isr: 1
     Topic: test    Partition: 7    Leader: 2   Replicas: 2 Isr: 2

I am using Kafka 0.10.1.1.

server.properties file contains:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://0.0.0.0:9092
port=9092
host.name=kafka-node1(kafka-node1 for the second host)
advertised.host.name=kafka-node1(kafka-node2 for the second host)
advertised.port=9092

When I try to produce messages from the second host I get the message below:

WARN Got error produce response with correlation id 1 on topic-partition test-4, retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION (org.apache.kafka.clients.producer.internals.Sender) ....

Can anyone please help?

Answer

DHEERAJ KUMAR GUPTA picture DHEERAJ KUMAR GUPTA · May 1, 2021

If you are getting UNKNOWN_TOPIC_OR_PARTITION then add the below properties to server.properties file :

listeners=PLAINTEXT://host.name:port
advertised.listeners=PLAINTEXT://host.name:port

in my cases values were as follows:

listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://localhost:9092

Related questions

Kafka High Level Consumer Fetch All Messages From Topic Using Java API (Equivalent to --from-beginning)

I am testing the Kafka High Level Consumer using the ConsumerGroupExample code from the Kafka site. I would like to retrieve all the existing messages on the topic called "test" that I have in the Kafka server config. Looking at …

Read More
consumer.How to specify partition to read? [kafka]

I am introducing with kafka and I want to know how to specify partition when I consume messages from topic. I have found several picture like this: It means that 1 consumer can consume messages from several partitions but 1 partition can …

Read More
Kafka Consumer in C++

I have been searching for C++ kafka consumer. I came across following which are for C++ kafka but there is no consumer. https://github.com/adobe-research/libkafka (Only sample producer) https://github.com/edenhill/librdkafka/tree/master/src-cpp Does anyone …

Read More