Akka Stream Kafka vs Kafka Streams

scala akka-stream apache-kafka-streams typesafe stream-processing
nsanglar picture nsanglar · Aug 11, 2017 · Viewed 15.3k times · Source

I am currently working with Akka Stream Kafka to interact with kafka and I was wonderings what were the differences with Kafka Streams.

I know that the Akka based approach implements the reactive specifications and handles back-pressure, functionality that kafka streams seems to be lacking.

What would be the advantage of using kafka streams over akka streams kafka?

Answer

Frederic A. picture Frederic A. · Aug 11, 2017

Your question is very general, so I'll give a general answer from my point of view.

First, I've got two usage scenario:

  1. cases where I'm reading data from kafka, processing it and writing some output back to kafka, for these I'm using kafka streams exclusively.
  2. cases where either the data source or sink is not kafka, for those I'm using akka streams.

This already allows me to answer the part about back-pressure: for the 1st scenario above, there is a back-pressure mechanism in kafka streams.

Let's now only focus on the first scenario described above. Let's see what I would loose if I decided to stop using Kafka streams:

  • some of my stream processors stages need a persistent (distributed) state store, kafka streams provides it for me. It is something that akka streams doesn't provide.
  • scaling, kafka streams automatically balances the load as soon as a new instance of a stream processor is started, or as soon as one gets killed. This works inside the same JVM, as well as on other nodes: scaling up and out. This is not provided by akka streams.

Those are the biggest differences that matter to me, I'm hoping that it makes sense to you!

Related questions

How to get started with Akka Streams?

The Akka Streams library already comes with quite a wealth of documentation. However, the main problem for me is that it provides too much material - I feel quite overwhelmed by the number of concepts that I have to learn. …

Read More
How to create a Source that can receive elements later via a method call?

I would like to create a Source and later push elements on it, like in: val src = ... // create the Source here // and then, do something like this pushElement(x1, src) pushElement(x2, src) What is the recommended way to do …

Read More
How to read query parameters in akka-http?

I know akka-http libraries marshal and unmarshal to class type while processing request.But now, I need to read request-parameters of GET request. I tried parameter() method and It is returning ParamDefAux type but i need those values as strings …

Read More