Rate exceeded in throttling is thrown on DescribeJobFlows
Several days ago all went well, but now, not just one, but three of our servers are throwing these exceptions:
{
"Error" => {
"Message" => "Rate exceeded",
"Code" => "Throttling",
"Type" => "Sender"
}
}
This happens when retrieving data from Amazon, while calling DescribeJobFlows on Amazon::Coral::ElasticMapReduceClient, and shows as NFO Exception Retriable invalid response.
Answer
Every AWS API has dynamic throttling protection. It's kinda sad but best way it to retry/sleep between API calls :(
There is a configuration for the boto3 client to change max_retries before throwing the exception. There is also the general reference on how to use retries.
Good luck!