Purge Kafka Topic

apache-kafka purge
Peter Klipfel picture Peter Klipfel · Apr 29, 2013 · Viewed 196.5k times · Source

Is there a way to purge the topic in kafka?

I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error:

kafka.common.InvalidMessageSizeException: invalid message size

Increasing the fetch.size is not ideal here, because I don't actually want to accept messages that big.

Answer

steven appleyard picture steven appleyard · Apr 16, 2015

Temporarily update the retention time on the topic to one second:

kafka-topics.sh --zookeeper <zkhost>:2181 --alter --topic <topic name> --config retention.ms=1000

And in newer Kafka releases, you can also do it with kafka-configs --entity-type topics

kafka-configs.sh --zookeeper <zkhost>:2181 --entity-type topics --alter --entity-name <topic name> --add-config retention.ms=1000

then wait for the purge to take effect (about one minute). Once purged, restore the previous retention.ms value.

Related questions

List all kafka topics

I'm using kafka 0.10 without zookeeper. I want to get kafka topics list. This command is not working since we're not using zookeeper: bin/kafka-topics.sh --list --zookeeper localhost:2181. How can I get the same output without zookeeper?

Read More
Is there a way to delete all the data from a topic or delete the topic before every run?

Is there a way to delete all the data from a topic or delete the topic before every run? Can I modify the KafkaConfig.scala file to change the logRetentionHours property? Is there a way the messages gets deleted as …

Read More
Leader Not Available Kafka in Console Producer

I am trying to use Kafka. All configurations are done properly but when I try to produce message from console I keep getting the following error WARN Error while fetching metadata with correlation id 39 : {4-3-16-topic1=LEADER_NOT_AVAILABLE} (…

Read More