Amazon Kinesis and guaranteed ordering

Dante picture Dante · Aug 31, 2016 · Viewed 14.2k times · Source

Amazon claims their Kinesis streaming product guarantees record ordering.

It provides ordering of records, as well as the ability to read and/or replay records in the same order (...)

Kinesis is composed of Streams that are themselves composed of one or more Shards. Records are stored in these Shards. We can write consumer applications that connect to a Shard and read/replay records in the order they were stored.

But can Kinesis guarantee, out of the box, ordering for the Stream itself without pushing ordering logic to the consumers? How can a consumer read records from multiple Shards of the same Stream, making sure the records are read in the same order they were added to the Stream?

Answer

Dante picture Dante · Sep 12, 2016

It seems this is not possible to achieve. Ordering is guaranteed on a shard level, but not across the all stream.

https://brandur.org/kinesis-order

So back to our original question: how can we guarantee that all records are consumed in the same order in which they’re produced? The answer is that we can’t, but that we shouldn’t let that unfortunate reality bother us too much. Once we’ve scaled our stream to multiple shards, there’s no mechanism that we can use to guarantee that records are consumed in order across the whole stream; only within a single shard.