AWS CLI moving file with wildcard (asterisk) in path

amazon-web-services amazon-s3 amazon-data-pipeline
BrainPermafrost picture BrainPermafrost · Jul 25, 2015 · Viewed 9.4k times · Source

I am attempting to move a file, from on s3 location to another, using an activity in a AWS data pipeline.

The command I am using is:

(aws s3 mv s3://foobar/Tagger/out//*/lastImage.txt s3://foobar/Tagger/testInput/lastImage.txt)

But I receive the following error:

A client error (404) occurred when calling the HeadObject operation: Key "Tagger/out//*/lastImage.txt" does not exist

But, if I replace the "*" with the specific directory name, it will work. The problem is I won't always know the name of the directory, so I was hoping I could use the "*" as a wild card.

Answer

shrikant picture shrikant · Aug 21, 2015

Wildcards in the AWS S3 CLI only work when using the --recursive flag.

So this should work for you:

aws s3 mv s3://foobar/Tagger/out/ s3://foobar/Tagger/testInput/ --recursive --exclude "*" --include "*/lastImage.txt"

Unfortunately, this will recreate the entire directory structure in your target location, and I'm not immediately sure that can be solved by just using the AWS CLI.

Related questions

Exporting a AWS Postgres RDS Table to AWS S3

I wanted to use AWS Data Pipeline to pipe data from a Postgres RDS to AWS S3. Does anybody know how this is done? More precisely, I wanted to export a Postgres Table to AWS S3 using data Pipeline. The …

Read More
Downloading an entire S3 bucket?

I noticed that there doesn't seem to be an option to download an entire S3 bucket from the AWS Management Console. Is there an easy way to grab everything in one of my buckets? I was thinking about making the …

Read More
Setting up FTP on Amazon Cloud Server

I am trying to set up FTP on Amazon Cloud Server, but without luck. I search over net and there is no concrete steps how to do it. I found those commands to run: $ yum install vsftpd $ ec2-authorize default …

Read More