Set cache-control for entire S3 bucket automatically (using bucket policies?)
I need to set cache-control headers for an entire s3 bucket, both existing and future files and was hoping to do it in a bucket policy. I know I can edit the existing ones and I know how to specify them on put if I upload them myself but unfortunately the app that uploads them cannot set the headers as it uses s3fs to copy the files there.
Answer
There are now 3 ways to get this done: via the AWS Console, via the command line, or via the s3cmd command line tool.
AWS Console Instructions
This is now the recommended solution. It is straight forward, but it can take some time.
- Log in to AWS Management Console
- Go into S3 bucket
- Select all files by route
- Choose "More" from the menu
- Select "Change metadata"
- In the "Key" field, select "Cache-Control" from the drop down menu max-age=604800Enter (7 days) for Value
- Press "Save" button
(thanks to @biplob - please give him some love below)
AWS Command Line Solution
Originally, when I created this bucket policies were a no go, so I figured how to do it using aws-cli, and it is pretty slick. When researching I couldn't find any examples in the wild, so I thought I would post some of my solutions to help those in need.
NOTE: By default, aws-cli only copies a file's current metadata, EVEN IF YOU SPECIFY NEW METADATA.
To use the metadata that is specified on the command line, you need to add the '--metadata-directive REPLACE' flag. Here are a some examples.
For a single file
aws s3 cp s3://mybucket/file.txt s3://mybucket/file.txt --metadata-directive REPLACE \
--expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public
For an entire bucket (note --recursive flag):
aws s3 cp s3://mybucket/ s3://mybucket/ --recursive --metadata-directive REPLACE \
--expires 2034-01-01T00:00:00Z --acl public-read --cache-control max-age=2592000,public
A little gotcha I found, if you only want to apply it to a specific file type, you need to exclude all the files, then include the ones you want.
Only jpgs and pngs:
aws s3 cp s3://mybucket/ s3://mybucket/ --exclude "*" --include "*.jpg" --include "*.png" \
--recursive --metadata-directive REPLACE --expires 2034-01-01T00:00:00Z --acl public-read \
--cache-control max-age=2592000,public
Here are some links to the manual if you need more info:
- http://docs.aws.amazon.com/cli/latest/userguide/using-s3-commands.html
- http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html#options
Known Issues:
"Unknown options: --metadata-directive, REPLACE"
this can be caused by an out of date awscli - see @eliotRosewater's answer below
S3cmd tool
S3cmd is a "Command line tool for managing Amazon S3 and CloudFront services". While this solution requires a git pull it might be a simpler and more comprehensive solution.
For full instructions, see @ashishyadaveee11's post below
Hope it helps!