How to set auto.create.topics.enable as default config on AWS MSK cluster

amazon-web-services apache-kafka aws-msk
rpassza picture rpassza · Apr 26, 2019 · Viewed 7.2k times · Source

Our AWS MSK cluster is not automatically creating topics when a producer first publishes a message to the topic. This is surprising because the default behaviour according to https://kafka.apache.org/documentation/#brokerconfigs should have auto.create.topics.enable = true.

I have attempted to set this configuration on my brokers but I am unable because the config is read-only for dynamic updates.

$ kafka-configs --bootstrap-server $KAFKA_BROKER --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true

Error while executing config command with args '--bootstrap-server $KAFKA_BROKER --entity-type brokers --entity-default --alter --add-config auto.create.topics.enable=true'
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidRequestException: Invalid config value for resource ConfigResource(type=BROKER, name=''): Cannot update these configs dynamically: Set(auto.create.topics.enable)

note: I replaced our broker's IP with $KAFKA_BROKER in the output

How can I configure my AWS MSK Kafka cluster to enable auto creation of topics?

Answer

Giorgos Myrianthous picture Giorgos Myrianthous · Apr 26, 2019

Update

You can now customise MSK configuration:

auto.create.topics.enable: Enables topic autocreation on the server.

Therefore, auto.create.topics.enable=true (defaults to false) should now do the trick as usual.

Although auto.create.topics.enable normally defaults to true, this is not the case for AWS MSK. According to these slides presented in Berlin's AWS Kafka Summit this February, the following are the default configuration parameters for Amazon MSK:

offsets.topic.replication.factor 3
transaction.state.log.replication.factor 3
transaction.state.log.min.isr 2
auto.create.topics.enable False
default.replication.factor 3
min.insync.replicas 2
unclean.leader.election.enable True
auto.leader.rebalance.enable True
authorizer.class.name kafka.security.auth.SimpleAclAuthorizer
group.initial.rebalance.delay.ms 3000
log.retention.hours 168

Note that even Kafka docs mention that the suggested production configuration should be set to false.

Related questions

Amazon Managed Streaming for Kafka- MSK features and performance

I'm evaluating AWS Managed Service Kafka (MSK) and I know that currently, it is in preview mode, hence might not have all features or proper documentation. I tried setting up the msk cluster and was validating whether msk could fulfill …

Read More
Kafka or SNS or something else?

Sorry if it is a newbie question. But I'm trying to understand what should I use. As far as I understand Kafka is : Apache Kafka is a distributed publish-subscribe messaging system. And SNS is also pub/sub system. My goal …

Read More
Permission denied (publickey) when SSH Access to Amazon EC2 instance

I want to use my Amazon ec2 instance but faced the following error: Permission denied (publickey). I have created my key pair and downloaded .pem file. Given: chmod 600 pem file. Then, this command ssh -i /home/kashif/serverkey.pem ubuntu@…

Read More