Google wont read my robots.txt on s3

amazon-s3 web-crawler robots.txt googlebot
suraj jain picture suraj jain · Oct 5, 2012 · Viewed 7.4k times · Source

As google is crawling our static content (stored on s3) we created a robots.txt in root directory (of the bucket).. as following 

User-agent: *
Disallow: /

Now since we wanted to remove exisiting content from google search .. we added the bucket in google webmaster a/c .. And requested for directory removal

Now the issue is that google wont recognize or read the robots.txt (which is need to future blocking of google block).. We r getting this message in webmaster tools

This site is not located at the top level for the domain. A robots.txt file is only valid when located in the highest-level directory and applies to all directories within the domain. The robots.txt file that applies to your site (if one exists) is located at http://s3.amazonaws.com/robots.txt. This page provides information on that file.

Answer

Ryan Parman picture Ryan Parman · Oct 6, 2012

Which URL did you give Google for your bucket? You need to use the DNS-style {bucket}.s3.amazonaws.com, instead of the path-style s3.amazonaws.com/{bucket}.

Related questions

Downloading an entire S3 bucket?

I noticed that there doesn't seem to be an option to download an entire S3 bucket from the AWS Management Console. Is there an easy way to grab everything in one of my buckets? I was thinking about making the …

Read More
Listing contents of a bucket with boto3

How can I see what's inside a bucket in S3 with boto3? (i.e. do an "ls")? Doing the following: import boto3 s3 = boto3.resource('s3') my_bucket = s3.Bucket('some/path/') returns: s3.Bucket(name='some/path/…

Read More
Setting up FTP on Amazon Cloud Server

I am trying to set up FTP on Amazon Cloud Server, but without luck. I search over net and there is no concrete steps how to do it. I found those commands to run: $ yum install vsftpd $ ec2-authorize default …

Read More