AWS S3 copy files and folders between two buckets

amazon-s3 copy amazon-web-services
cnicolaou picture cnicolaou · Feb 15, 2012 · Viewed 126.9k times · Source

I have been on the lookout for a tool to help me copy content of an AWS S3 bucket into a second AWS S3 bucket without downloading the content first to the local file system.

I have tried to use the AWS S3 console copy option but that resulted in some nested files being missing.

I have tried to use Transmit app (by Panic). The duplicate command downloads the files first to the local system then uploads them back to the second bucket, which quite inefficient.

Answer

Layke picture Layke · Sep 12, 2013

Copy between S3 Buckets

AWS (just recently) released a command line interface for copying between buckets.

http://aws.amazon.com/cli/

$ aws s3 sync s3://mybucket-src s3://mybucket-target --exclude *.tmp
..

This will copy from one target bucket to another bucket.

See the documentation here : S3 CLI Documentation

Related questions

Is it possible to copy all files from one S3 bucket to another with s3cmd?

I'm pretty happy with s3cmd, but there is one issue: How to copy all files from one S3 bucket to another? Is it even possible? EDIT: I've found a way to copy files between buckets using Python with boto: …

Read More
Copy data from Amazon S3 to Redshift and avoid duplicate rows

I am copying data from Amazon S3 to Redshift. During this process, I need to avoid the same files being loaded again. I don't have any unique constraints on my Redshift table. Is there a way to implement this using …

Read More
Downloading an entire S3 bucket?

I noticed that there doesn't seem to be an option to download an entire S3 bucket from the AWS Management Console. Is there an easy way to grab everything in one of my buckets? I was thinking about making the …

Read More