can a kafka consumer filter messages before polling all of them from a topic?

java apache-kafka filtering producer-consumer
mikej1688 picture mikej1688 · Jun 26, 2018 · Viewed 15.7k times · Source

It was said that consumers can only read the whole topic. No luck doing evaluations on brokers to filter messages.

It implies that we have to consume/receive all messages from a topic and filter them on the client side.

That's too much. I was wondering if we can filter and receive specific types of messages, based on somethings already passed to brokers, such as the msg keys or other things.

from the method, Consumer.poll(timeout), it seems no extra things we can do.

Answer

Mickael Maison picture Mickael Maison · Jun 26, 2018

No, with the Consumer you cannot only receive some messages from topics. The consumer fetches all messages in order.

If you don't want to filter messages in the Consumer, you could use a Streams job. For example, Streams would read from your topic and only push to another topic the messages the consumer is interested in. Then the consumer can subscribe to this new topic.

Related questions

How to Delete a topic in apache kafka

I need to delete a topic in kafka-0.8.2.2.3. I have used the below command for deleting the topic: bin/kafka-topics.sh --zookeeper localhost:2181 --delete --topic DummyTopic The command executed successfully but when I run a command to list the topics, …

Read More
How can I send large messages with Kafka (over 15MB)?

I send String-messages to Kafka V. 0.8 with the Java Producer API. If the message size is about 15 MB I get a MessageSizeTooLargeException. I have tried to set message.max.bytesto 40 MB, but I still get the exception. Small messages worked …

Read More
Java, How to get number of messages in a topic in apache kafka

I am using apache kafka for messaging. I have implemented the producer and consumer in Java. How can we get the number of messages in a topic?

Read More