I recently started learning Kafka and end up with these questions.
What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world.
How Stream is different as this also consumes from or produce messages to Kafka? and why is it needed as we can write our own consumer application using Consumer API and process them as needed or send them to Spark from the consumer application?
I did Google on this, but did not get any good answers for this. Sorry if this question is too trivial.
Update April 09, 2018: Nowadays you can also use ksqlDB, the event streaming database for Kafka, to process your data in Kafka. ksqlDB is built on top of Kafka's Streams API, and it too comes with first-class support for "streams" and "tables".
what is the difference between Consumer API and Streams API?
Kafka's Streams library (https://kafka.apache.org/documentation/streams/) is built on top of the Kafka producer and consumer clients. Kafka Streams is significantly more powerful and also more expressive than the plain clients.
It's much simpler and quicker to write a real-world application start to finish with Kafka Streams than with the plain consumer.
Here are some of the features of the Kafka Streams API, most of which are not supported by the consumer client (it would require you to implement the missing features yourself, essentially re-implementing Kafka Streams).
map
, filter
, reduce
as well as (2) an imperative style Processor API for e.g. doing complex event processing (CEP), and (3) you can even combine the DSL and the Processor API.See http://docs.confluent.io/current/streams/introduction.html for a more detailed but still high-level introduction to the Kafka Streams API, which should also help you to understand the differences to the lower-level Kafka consumer client.
Beyond Kafka Streams, you can also use the event streaming database ksqlDB to process your data in Kafka. ksqlDB is built on top of Kafka Streams. It supports essentially the same features as Kafka Streams, but you write streaming SQL instead of Java or Scala. Programmatically, you can interact with ksqlDB via a CLI or a REST API; it also has a native Java client in case you don't want to use REST.
So how is the Kafka Streams API different as this also consumes from or produce messages to Kafka?
Yes, the Kafka Streams API can both read data as well as write data to Kafka. It it supports Kafka transactions, so you can e.g. read one or more messages from one or more topic(s), optionally update processing state if you need to, and then write one or more output messages to one or more topics—all as one atomic operation.
and why is it needed as we can write our own consumer application using Consumer API and process them as needed or send them to Spark from the consumer application?
Yes, you could write your own consumer application -- as I mentioned, the Kafka Streams API uses the Kafka consumer client (plus the producer client) itself -- but you'd have to manually implement all the unique features that the Streams API provides. See the list above for everything you get "for free". It is thus rather a rare circumstance that a user would pick the plain consumer client rather than the more powerful Kafka Streams library.