How to decide total number of partition keys in AWS kinesis stream?

amazon-kinesis
shivba picture shivba · Jul 10, 2015 · Viewed 9.5k times · Source

In a producer-consumer web application, what should be the thought process to create a partition key for a kinesis stream shard. Suppose, I have a kinesis stream with 16 shards, how many partition keys should I create? Is it really dependent on the number of shards?

Answer

az3 picture az3 · Jul 13, 2015

Partition (or Hash) Key: starts from 1 up to 340282366920938463463374607431768211455. Lets say ~34020 * 10^34, I will omit 10^34 for ease...

If you have 30 shards, uniformly divided, each should cover 1134 * 10^34 hash keys. The coverage should be like this.

Shard-00: 0 - 1134 Shard-01: 1135 - 2268 Shard-03: 2269 - 3402 Shard-04: 3403 - 4536 ... Shard-28: 30619 - 31752 Shard-29: 31753 - 32886 Shard-30: 32887 - 34020

And if you have 3 consumer applications (listening to these 30 shards) each should listen 10 shards (optimum balanced).

This also explains Merge and Split operations on a Stream.

  • To merge 2 shards, they should cover adjacent hash keys. You cannot merge Shard-03 and Shard-29.
  • You can split any shard. If you split shard-00 in the middle, the distribution will like this;

Shard-31: 0 - 567 Shard-32: 568 - 1134 Shard-01: 1135 - 2268 Shard-03: 2269 - 3402 Shard-04: 3403 - 4536 ... Shard-28: 30619 - 31752 Shard-29: 31753 - 32886 Shard-30: 32887 - 34020

See, Shard-00 will no longer accept new data. The new records that are put in Kinesis stream with the same partition key range (as Shard-00) will be placed under Shard-31 or Shard-32.

While sending data to Kinesis (ie. producer side), you should not worry about "which shard the data goes to". Sending a random number (or uuid, or current timestamp in millis) would be best for scaling and distributing the data effectively on shards. Unless you are worried about the ordering of records in a single shard, it is best to choose a random number/constantly changing partition key for put_record request.

In Java you can use "putRecordsRequestEntry.setPartitionKey(Long.toString(System.currentTimeMillis()))" or "putRecordRequest.setPartitionKey(Long.toString(System.currentTimeMillis()))" can be examples.

Related questions

Why should I use Amazon Kinesis and not SNS-SQS?

I have a use case where there will be stream of data coming and I cannot consume it at the same pace and need a buffer. This can be solved using an SNS-SQS queue. I came to know the Kinesis …

Read More
Application report for application_ (state: ACCEPTED) never ends for Spark Submit (with Spark 1.2.0 on YARN)

I am running kinesis plus spark application https://spark.apache.org/docs/1.2.0/streaming-kinesis-integration.html I am running as below command on ec2 instance : ./spark/bin/spark-submit --class org.apache.spark.examples.streaming.myclassname --master yarn-cluster --num-executors 2 --driver-memory 1g --executor-memory 1g …

Read More
How to pause / resume a aws lambda function

For example I have lambda functions that consume messages from a KinesisStream. How do stop and resume the function so that I don't incur charges and I don't loose data in the stream. I know that if the events keep …

Read More