An error occurred (ThrottlingException) when calling the GetDeployment operation (reached max retries: 4): Rate exceeded

Question

With an increase in the number of Deployment Groups in AWS CodeDeploy, BitBucket Pipelines are starting to fail more often.

PIPELINE FAILED...

+ python ./_scripts/codedeploy_deploy.py
Failed to deploy application revision.
An error occurred (ThrottlingException) when calling the GetDeployment operation (reached max retries: 4): Rate exceeded

Is there any way to increase the value before rate limit or reduce the chance of occurrence?

AWS FORUM POST: https://forums.aws.amazon.com/thread.jspa?messageID=892511

Answer 1

Unfortunately, there is no way to increase the rate limit, as this is dynamically provisioned by the AWS API.

AWS SUPPORT:

This issue is not related to any concurrent deployment or any other resource related limit. This is a throttling issue, which cannot be changed.

Multiple API calls initiated at the same time gets throttled at our endpoints. The limit for each endpoint is varies and is dynamic, therefore it is not documented anywhere.

In this case, there are multiple calls for 'GetDeployment' API simultaneously hence the calls are getting throttled.

In such scenarios we recommend to implement error retries and exponential backoff between retries, so that the API calls are not simultaneous.

You can check the below link which explains how to implement it in our Code. - https://docs.aws.amazon.com/general/latest/gr/api-retries.html

I was able to implement an exponential back off to reduce the rate at which we are trying to get the deployment status and also increase the number of retries before deployment failure.

Make sure you are using the latest version of BOTO3 (boto3-1.9.108 botocore-1.12.108) which supports this new config system.

BOTO3 RETRY CONFIG: https://github.com/boto/botocore/issues/882#issuecomment-338846339

FORK: https://bitbucket.org/DJRavine/aws-codedeploy-bitbucket-pipelines-python/src/master/ GIST: https://gist.github.com/djravine/5007e2a7f726cebe14ea51c7ee54bf5d

PIPELINE SUCCESSFUL...

+ python ./_scripts/codedeploy_deploy.py
Deployment Created (Exponential back off 30s)
Deployment InProgress (Exponential back off 60s)
Deployment Succeeded

NOTE: I will update this post with more information as I revise the usage based on our deployments.