How to select a file from aws s3 by using wild character

amazon-web-services amazon-s3 amazon-emr
user3858193 picture user3858193 · Apr 10, 2015 · Viewed 21.3k times · Source

I have many a files in s3 bucket and I want to copy those files which have start date of 2012. This below command copies all the file.

aws s3 cp s3://bp-dev/bp_source_input/ C:\Business_Panorama\nts\data\in --recursive  --include "201502_nts_*.xlsx"

Answer

bsnchan picture bsnchan · Apr 10, 2015

You may want to add the "--exclude" flag before your include filter.

The AWS CLI takes the filter "--include" to include it in your already existing search. Since all the files are being returned, you need to exclude all the files first, before including the 2015*.xlsx.

If you want files with only the format "201502_nts_*.xlsx", you can run aws s3 cp s3://bp-dev/bp_source_input/ C:\Business_Panorama\nts\data\in --recursive --exclude * --include "201502_nts_*.xlsx"

Related questions

How to handle fields enclosed within quotes(CSV) in importing data from S3 into DynamoDB using EMR/Hive

I am trying to use EMR/Hive to import data from S3 into DynamoDB. My CSV file has fields which are enclosed within double quotes and separated by comma. While creating external table in hive, I am able to specify …

Read More
Extremely slow S3 write times from EMR/ Spark

I'm writing to see if anyone knows how to speed up S3 write times from Spark running in EMR? My Spark Job takes over 4 hours to complete, however the cluster is only under load during the first 1.5 hours. I was …

Read More
Folder won't delete on Amazon S3

I'm trying to delete a folder created as a result of a MapReduce job. Other files in the bucket delete just fine, but this folder won't delete. When I try to delete it from the console, the progress bar next …

Read More