SNS to Lambda vs SNS to SQS to Lambda
I'm trying to understand whether I need SQS in my workflow if someone can help explain. In my app, when an action is taken, it submits info to SNS topic which invokes LAMBDA to do some processing. This is working great as it is.
When I do research online, it seems that people are using SQS in this stack as well where SNS would put info on SQS and then SQS would then invoke LAMBDA.
I guess what I'm trying to understand is the need for SQS in this. What value doe that add or in other words, what am I losing by invoking my LAMBDA directly from SNS?
Answer
The only advantage of having a SQS in between SNS and Lambda is Reprocessing. Assume that the Lambda fails to process certain event for some reason (e.g. timeout or lack of memory footprint), you can increase the timeout (to max 5 minutes) or memory (to max of 1.5GB) and restart your polling and you can reprocess the older events.
This would not be possible in case of SNS to Lambda, wherein if Lambda fails the event is lost. And even if you configure DLQ you would still have to make provisions for reading that separately and processing the message
So if your events are critical and you don't want to miss out on them, then go for SNS - SQS - Lambda
Having said that SQS cannot trigger lambda like SNS. You will have to poll the SQS at frequent intervals
EDIT :
AWS announced SQS triggering Lambda support on 28 JUN 2018. So no need anymore for polling the queue at frequent intervals. New items to SQS can trigger Lambda