error while starting kafka broker

apache-kafka apache-zookeeper
brain storm picture brain storm · Sep 29, 2016 · Viewed 35.2k times · Source

I was able to successfully set up zookeeper and one kafka broker yesterday. Everything worked as expected. I shut down kafka (ctrl + c) and then zookeeper.

Today I started zookeeper and when I started kafka (bin/kafka-server-start.sh config/server0.properties), I get the following error. I tried various remedies suggested (removing completely my kafka installation and doing it again from scratch). Still I get same error.

[2016-09-28 16:15:55,895] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.
        at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:305)
        at kafka.utils.ZkUtils.registerBrokerInZk(ZkUtils.scala:291)
        at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:70)
        at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:51)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:244)
        at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
        at kafka.Kafka$.main(Kafka.scala:67)
        at kafka.Kafka.main(Kafka.scala)
[2016-09-28 16:15:55,896] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)

All set up in mac

Answer

Shubham Jain picture Shubham Jain · Dec 19, 2017

I have faced the same issue while setting up the kafka and zookeeper multinode cluster. The cause of the issue is : As zookeeper ensemble takes some time to setup So when we start kafka brokers, all zookeeper nodes might not joined ensemble yet. So may be One kafka node connected to a zookeeper which is not yet in the ensemble. Another reason can be kafka broker is killed instead of stopped. When you kill your kafka process or it crashes, and it does not close its zookeeper connection correctly

To avoid this situation make sure your zookeeper ensemble is up and running before starting kafka brokers. Add proper waits between zookeeper and kafka broker services start and stop.

Related questions

List all kafka topics

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?

Read More
Is there a way to delete all the data from a topic or delete the topic before every run?

Is there a way to delete all the data from a topic or delete the topic before every run? Can I modify the KafkaConfig.scala file to change the logRetentionHours property? Is there a way the messages gets deleted as …

Read More
How to check if ZooKeeper is running or up from command prompt?

I exploring a few options to setup kafka and I knew that the Zookeeper has to be up and running to initiate a kafka. I would like to know how can I find the below. 1) hostname and port for my …

Read More