Multiple AWS Lambda functions on a Single DynamoDB Stream

witchking picture witchking · Jul 25, 2016 · Viewed 9.4k times · Source

I have a Lambda function to which multiple DynamoDB streams are configured as event sources and this is a part of a bigger pipeline. While doing my checks, I found some missing data in one of the downstream components. I want to write a simpler Lambda function which is configured as an event source to one of the earlier mentioned DynamoDB streams. This would cause one of my DynamoDB streams to have two Lambda functions reading from it. I was wondering if this is OK? Are both Lamdba functions guaranteed to receive all records placed in the stream and are there any resource (Read/Write throughput) limits I need to be aware of. Couldn't find any relevant documentation for this on the AWS website, but I did find this regarding processing of shards

To access a stream and process the stream records within, you must do the following:

  • Determine the unique Amazon Resource Name (ARN) of the stream that you want to access.
  • Determine which shard(s) in the stream contain the stream records that you are interested in.
  • Access the shard(s) and retrieve the stream records that you want.

Note No more than 2 processes at most should be reading from the same Streams shard at the same time. Having more than 2 readers per shard may result in throttling.

Not sure how the above relates to cases where Streams are configured as Event sources to Lambdas as opposed to manually reading from a Stream using the API.

Answer

hellomichibye picture hellomichibye · Jul 26, 2016

You can have multiple Lambdas using the same stream as an event source. They will not infer with each other. But as the documentation says: "Note No more than 2 processes at most should be reading from the same Streams shard at the same time. Having more than 2 readers per shard may result in throttling." So if you heavily utilize your streams you should not have more than two Lambdas connected to them.