How to create topics in apache kafka?

apache-kafka kafka-consumer-api kafka-producer-api
Ratha picture Ratha · Apr 6, 2016 · Viewed 54k times · Source

What is the bestway to create topics in kafka?

  • How many replicas/partitions to be defined when we create topics?

In the new producer API, when i try to publish a message to a non existing topic , it first time fails and then successfully publishing.

  • I would like to know, the relationships between replica, partitions and the number of cluster nodes.
  • Do we need to create topic prior to publish messages?

Answer

ponkin picture ponkin · Apr 6, 2016

When you are starting your Kafka broker you can define set of properties in conf/server.properties file. This file is just key value property file. One of the properties is auto.create.topics.enable, if it's set to true (by default) Kafka will create topics automatically when you send messages to non-existing topics.

All config options you can find are defined here. IMHO, a simple rule for creating topics is the following: number of replicas cannot be more than the number of nodes that you have. Number of topics and partitions is unaffected by the number of nodes in your cluster

for example:

  • You have 9 node cluster
  • Your topic can have 9 partitions and 9 replicas or 18 partitions and 9 replicas or 36 partitions and 9 replicas and so on...

Related questions

Understanding Kafka Topics and Partitions

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: When a producer is producing a message - it will specify the topic it wants to send the message to, is …

Read More
Is key required as part of sending messages to Kafka?

KeyedMessage<String, byte[]> keyedMessage = new KeyedMessage<String, byte[]>(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed messages, will it still work with delete.retention.…

Read More
How to write a file to Kafka Producer

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps: Started zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties Started Server bin/kafka-server-start.sh config/server.properties Created …

Read More