Creating a realtime push notification system for desktop/mobile/web apps using kafka as message broker
I have a use case where there needs to be a real-time communication between servers and clients following a pub/sub messaging pattern. Producers will be server in java, node etc and clients will be - java desktop apps, mobile apps (android/ios), browser(javascript).
I have explored many options discussed below but I am not able to come up with a powerful scalable solution.
Use case: The server will be publishing notifications/messages on various topics and all the clients (java/js/ios) subscribed to a set of topics will get these messages in real-time.
I followed 3 approaches to solve this problems 1> socketIo/socketcluster 2> explored mqtt protocol with mosquitto/rabbitmq as a broker. 3> explored kafka
The main goal is to make this architecture highly scalable with not only more than million simultaneous client connections but also more than million messages published and consumed per second.
The first approach is straightforward and it works but webSocket is not a scalable solution.
The second approach works but rabbitmq will create large number of queues (million queues for million clients) as it maintains queues for each client connected to it, also rabbitMq does not have a high message publish and consume rate, plus let's say we have a cluster of rabbitMq nodes then only one node is used for handling requests and others are used for high availability but not parallel consumption.
Thirdly I explored kafka which is known for it's benchmarks I created clients in java using kafka's high level java api which can be used to subscribe to a kafka topic and any message published to this topic gets delivered to the client in real-time.
So my question is how good it is to use kafka clients for real-time push notifications where all the java desktop apps (maybe a million) will include this kafka java client sdk and will be subscribed to certain topics, here I am treating each client as a consumer group.
Also one main problem here is that this kafka client is large in size due to its scala dependencies so using this client in android won't be a good option also I don't think it will work.
mqtt excels here as it has official phao clients for android, java, ios etc.
Also, I have not seen examples on web using kafka for pub/sub messaging with million consumers, mostly people are using it for data pipeline eg: real-time log processing, feeding data to HDFS, analytics engine etc, stream processing.
The main question is that how can I use mqtt protocol(which works well with android/ios/web/iot) with kafka as a message broker (which has a high publish/subscribe rate) and come up with a scalable solution to this problem.
My use case somehow also resembles to uber where there are million of android/ios devices (clients) and we can actually see real-time movement of all the cars in our location on map, does anyone have an idea that what is the architecture behind these real-time tracking of cars.
Answer
This article describes making a real-time chat system using Kafka and node.js. They also link to a git repo containing their example. Here is something important to note from the article:
In testing we noticed that there’s about 1sec of lag between posting a message and it appearing on all the other clients, which we figured out was due to how often Kafka commits messages to disk. Because Kafka assures that messages don’t get lost, they need to be written out to disk before they’re forwarded to subscribers. The developers have chosen to flush messages to disk every second, which explains the lag that we saw.
We think it’s an interesting way to do things, but it gets the job done. As they note, the focus is on throughput and not latency, so while it’s not ideally suited to this sort of usage, it gets the job done.