Looking for clarity on Event Hubs vs Topics in Azure Service Bus
I've been learning about Event Hubs and just want to get confirmation or correction on my perspective on Event Hubs? I’m used to leveraging retries, poison messages, at least once delivery and so on for normal enterprise messaging solutions, which Azure Service Bus Queues and Topics give me. It seems that Event Hubs is intended to provide a different tool for very high scale where you have to give up a little of the more “enterprise” features for much higher scale.
Am I thinking about this correctly? Are there additional specifics I need to consider as well? I realize there could be some functional overlap with Event Hubs and Topics, but I'm just looking to get some clarity on how to think of using Event Hubs.
Answer
If you have the choice it's almost always easier to write a system based around a full enterprise pubsub messaging system where you can mark single events as having been consumed, retry messages, and just about every other wonderful feature. If you've already accepted partitioning your message channel (which Azure Service Bus Topics appear to support) then you could in principle scale a more full featured messaging system to the degree you require. The issue is at what cost?
An Azure Service Bus Topic has a cost at high scale of approximately $0.20 per Million messages, Amazon SQS (somewhat similar) lists $0.50 per Million. If you host it yourself you'll likely need to set up a lot of RabbitMQ servers or even multiple clusters as you partition.
Azure Event Hub costs $0.028 per Million plus an amount per throughput unit, same for Amazon Kinesis. Apache Kafka has been benchmarked at 2 Million per second on 3 machines
At say 20,000 events per second sustained the difference between some Azure Topics and Azure Event Hub is in the range of a full time developer's salary. At 2 million per second sustained (which requires contacting MS), the difference is approaching $1M/month.
Basically use the partitioned stream|log / offset tracking systems when you either don't need all the useful features of a full messaging system, or when you don't need them enough to pay the ~10X premium. (Or can't use them because you can't scale the proper messaging system enough without heroic efforts).