AWS - SQS Batch Size and Lambda approach

Wexoni picture Wexoni · Feb 7, 2019 · Viewed 11.6k times · Source

If I understand correctly, batch size setting on Lambda decides how many messages to take in one sweep from the SQS. Therefore this JSON (taken from the test Lambda SQS);

{
  "Records": [
    {
      "messageId": "19dd0b57-b21e-4ac1-bd88-01bbb068cb78",
      "receiptHandle": "MessageReceiptHandle",
      "body": "FAIL",
      "attributes": {
        "ApproximateReceiveCount": "1",
        "SentTimestamp": "1523232000000",
        "SenderId": "123456789012",
        "ApproximateFirstReceiveTimestamp": "1523232000001"
      },
      "messageAttributes": {
      },
      "md5OfBody": "7b270e59b47ff90a553787216d55d91d",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:eu-west-1:123456789012:MyQueue",
      "awsRegion": "eu-west-1"
    }
  ]
}

There is Records array. And if I set batch size to 5, then if there are 5 messages in SQS, they will be included in array. If there are 10 messages, then Lambda will be invoked twice, with 5 messages in each Record.

I am now confused a bit, as to what approach to take. My Lambda is fairly simple. It will be Axios POST request to external service. If it errors out, I will throw an error. I could even use axios-retry, and make retries fairly easy.

Should I use batch in my case? Naively lookin, all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

Contrary, I would have to iterate via all messages and attempt an Axios request. What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Answer

A.Khan picture A.Khan · Feb 7, 2019

all I need is 1 to 1. In other words, message arrives. Lambda takes it. If it errors, it will be retried automatically a bit later.

I don't think you need batch processing in above case.

What if the third of five messages fails, in that case I throw an error and Lambda is stopped. What happens to messages four and five? Are they resent to SQS and then again picked up for another execution?

Message(s) won't be deleted from the queue if your Lambda errors, based on the SQS visibility timeout and redrive policy configuration. SQS will trigger Lambda again according to the configuration. If you have configured DLQ then after maxReceiveCount is reached, failed messages will be added to the DLQ and removed from the main queue.