Can a Kafka producer create topics and partitions?

apache-kafka messaging kafka-producer-api
smartwepa picture smartwepa · Apr 22, 2017 · Viewed 17.2k times · Source

currently I am evaluating different Messaging Systems. There is a question related to Apache Kafka which I could not answer myself.

Is it possible for a Kafka producer to create topics and partitions (on existing topics aswell) dynamically? If yes, is there any disadvantage that comes with it?

Thanks in Advance

Answer

user2122031 picture user2122031 · Apr 24, 2017

Updated:

The kafka broker has a property: auto.create.topics.enable

If you set that to true if the producer publishes a message to the topic with the new topic name it will automatically create a topic for you.

The Confluent Team recommends not doing this because the explosion of topics, depending on your environment can become unwieldy, and the topic creation will always have the same defaults when created. It's important to have a replication-factor of at least 3 to ensure durability of your topics in the event of disk failure.

Related questions

What's the purpose of Kafka's key/value pair-based messaging?

All of the examples of Kafka | producers show the ProducerRecord's key/value pair as not only being the same type (all examples show <String,String>), but the same value. For example: producer.send(new ProducerRecord<String, String&…

Read More
Kafka - Delayed Queue implementation using high level consumer

Want to implement a delayed consumer using the high level consumer api main idea: produce messages by key (each msg contains creation timestamp) this makes sure that each partition has ordered messages by produced time. auto.commit.enable=false (will …

Read More
Difference between stream processing and message processing

What is the basic difference between stream processing and traditional message processing? As people say that kafka is good choice for stream processing but essentially kafka is a messaging framework similar to ActivMQ, RabbitMQ etc. Why do we generally not …

Read More