Throttling S3 commands with aws cli
I'm running a backup script using AWS CLI to perform an S3 sync command every night on my MediaTemple server. This has run without fail for months, but I updated my Plesk installation and now every night, when the backup script runs, MediaTemple disables my server due to excessive usage. The limits I seem to be crossing are as follows:
RESOURCE INFO:
Packets per second limit: 35000
Packets per second detected: 42229.11667000000306870788
Bytes per second limit: 50000000
Bytes per second detected: 61801446.10000000149011611938
They also include a networking snapshot at the time they take the server offline which includes many open connections to Amazon IP addresses (9 at time of the snapshot).
Is there anything I can do to throttle the connections to AWS? Preferably I'm looking for an option within the AWS API (though I haven't seen anything useful in the documentation), but barring that, is there something I can do on my end to manage the connections at the network level?
Answer
As well as changing the max default connections and chunk size already mentions you can also set the max_bandwidth. This is very effective when uploading large single files.
aws configure set default.s3.max_bandwidth 50MB/s