DynamoDB Streams to S3

user3293898 picture user3293898 · Mar 10, 2016 · Viewed 8.2k times · Source

I am using Data Pipeline (DP) for daily backups of DynamoDB, however, I would like to do incremental backups of the data that is missed by DP runs (updates between DP runs). To accomplish that, I would like to use DynamoDB Streams + Lambda + S3 to bring real-time DynamoDB updates to S3. I understand how DynamoDB streams work, however, I am struggling with creating a Lambda function that writes to S3 and say rolls a file every hour.

Has anyone tried it?

Answer

AnubhavJain picture AnubhavJain · Aug 8, 2016

Its an hour job dude,What you need to do is

  1. Enable Dynamo DB update Stream and attach aws provided lambda function https://github.com/awslabs/lambda-streams-to-firehose
  2. Enable Firehose stream and use above function to stream outs records in firehose.
  3. Configure Firehose to dump the records to S3.

done.