Make Kafka Topic Log Retention Permanent

apache-kafka apache-kafka-connect
user1077071 picture user1077071 · Sep 28, 2016 · Viewed 10.8k times · Source

I am writing log messages into a Kafka Topic and I want the retention of this topic to be permanent. I have seen in Kafka and Kafka Connect (_schemas, connect-configs, connect-status, connect-offsets, etc) that there are special topics that are not deleted by the log retention time. How do I enforce a topic to be like these other special topics? Is it the naming convention or some other properties?

Thanks

Answer

highlycaffeinated picture highlycaffeinated · Sep 28, 2016

If you want to retain all topics forever, you can set both log.retention.hours and log.retention.bytes to -1.

Related questions

Kafka Connect running out of heap space

After starting Kafka Connect (connect-standalone), my task fails immediately after starting with: java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) at java.nio.ByteBuffer.allocate(ByteBuffer.java:335) at org.apache.kafka.common.network.…

Read More
How to connect Kafka with Elasticsearch?

I am new in Kafka, I use kafka to collect netflow through logstash(it is ok), and I want to send the data to elasticsearch from kafka, but there are some problems. My question is how can I connect Kafka …

Read More
How can i get all group list with kafka0.10.x

when i use kafka0.8.x，i can get all group list with zookeeper by path /consumers/group_id, because offsets will commit to zookeeper in 0.8.x. But in version of 0.10.x, offsets commited to kafka broker like a path /brokers/…

Read More