Is it possible to perform a batch upload to amazon s3?

web-services amazon-web-services amazon-s3 cloud blob
flybywire picture flybywire · Feb 24, 2013 · Viewed 58k times · Source

Does amazon s3 support batch uploads? I have a job that needs to upload each night ~100K of files that can be up to 1G but is strongly skewed towards small files (90% are less than 100 bytes and 99% are less than 1000 bytes long).

Does the s3 API support uploading multiple objects in a single HTTP call?

All the objects must be available in S3 as individual objects. I cannot host them anywhere else (FTP, etc) or in another format (Database, EC2 local drive, etc). That is an external requirement that I cannot change.

Answer

RubyFanatic picture RubyFanatic · Jun 16, 2014

Alternatively, you can upload S3 via AWS CLI tool using the sync command.

aws s3 sync local_folder s3://bucket-name

You can use this method to batch upload files to S3 very fast.

Related questions

Service discovery vs load balancing

I am trying to understand in which scenario I should pick a service registry over a load balancer. From my understanding both solutions are covering the same functionality. For instance if we consider consul.io as a feature list we …

Read More
Amazon Product Advertising API signed request with Java

after many hours of tinkering and reading the whole internet several times I just can't figure out how to sign requests for use with the Product Advertising API. So far I managed to generate a client from the provided WSDL …

Read More
SOAP vs REST (differences)

I have read articles about the differences between SOAP and REST as a web service communication protocol, but I think that the biggest advantages for REST over SOAP are: REST is more dynamic, no need to create and update UDDI(…

Read More