Apache Kafka vs Apache Storm

apache-storm apache-kafka data-integration
Ananth Duari picture Ananth Duari · Feb 16, 2014 · Viewed 50.9k times · Source

Apache Kafka: Distributed messaging system
Apache Storm: Real Time Message Processing

How we can use both technologies in a real-time data pipeline for processing event data?

In terms of real time data pipeline both seems to me do the job identical. How can we use both the technologies on a data pipeline?

Answer

forhas picture forhas · Feb 16, 2014

You use Apache Kafka as a distributed and robust queue that can handle high volume data and enables you to pass messages from one end-point to another.

Storm is not a queue. It is a system that has distributed real time processing abilities, meaning you can execute all kind of manipulations on real time data in parallel.

The common flow of these tools (as I know it) goes as follows:

real-time-system --> Kafka --> Storm --> NoSql --> BI(optional)

So you have your real time app handling high volume data, sends it to Kafka queue. Storm pulls the data from kafka and applies some required manipulation. At this point you usually like to get some benefits from this data, so you either send it to some Nosql db for additional BI calculations, or you could simply query this NoSql from any other system.

Related questions

IBM MQ versus Apache Kafka

I am designing a new architecture big data where my client has as IBM MQ broker. We use to work with Kafka in big data architectures, because I suppose it is faster, reliable and stronger than the alternatives, and because …

Read More
Streamparse wordcount example

I have been wanting to use Apache Storm to stream from Kafka. I am more comfortable with Python, so I decided to use streamparse (https://github.com/Parsely/streamparse). The word count example is the introductory example. I have been …

Read More
difference between exactly-once and at-least-once guarantees

I'm studying distributed systems and referring to this old question: stackoverflow link I really can't understand the difference between exactly-once, at-least-once and at-most-once guarantees, I read these concepts in Kafka, Flink and Storm and Cassandra also. For instance someone says …

Read More