Kafka set compression type at producer vs topic

apache-kafka kafka-producer-api
shants picture shants · Feb 7, 2018 · Viewed 16.2k times · Source

What's the difference between the following ways of enabling compression in kafka:

Approach 1: Create a topic using the command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --config compression.type=gzip --topic test

Approach 2: Set the property compression.type = gzip in Kafka Producer Client API.

I get better compression and higher throughput when using Approach 1.

If I use Approach 1, does it mean that the compression occurs at the broker end while in Approach 2, the messages are compressed at Producer end and then sent to broker?

Answer

amethystic picture amethystic · Feb 8, 2018

If I use Approach 1, does it mean that the compression occurs at the broker end?

It depends. If the producer does not set a compression.type or sets a different one, then the message will be compressed at the broker end. However, if producer also sets compression.type to gzip, no need to compress again at the broker end. Actually, there are other strict conditions that must be met to ensure no need to compress, although it's a little bit beyond of the scope.

in Approach 2, the messages are compressed at Producer end and then sent to broker?

Yes, records will be compressed before being sent to the broker if producer sets its compression.type config.

Related questions

How to check whether Kafka Server is running?

I want to ensure whether kafka server is running or not before starting production and consumption jobs. It is in windows environment and here's my kafka server's code in eclipse... Properties kafka = new Properties(); kafka.setProperty("broker.id", "1"); kafka.setProperty("…

Read More
Understanding Kafka Topics and Partitions

I am starting to learn Kafka for enterprise solution purposes. During my readings, some questions came to my mind: When a producer is producing a message - it will specify the topic it wants to send the message to, is …

Read More
Is key required as part of sending messages to Kafka?

KeyedMessage<String, byte[]> keyedMessage = new KeyedMessage<String, byte[]>(request.getRequestTopicName(), SerializationUtils.serialize(message)); producer.send(keyedMessage); Currently, I am sending messages without any key as part of keyed messages, will it still work with delete.retention.…

Read More