Python: Amazon S3 cannot get the bucket: says 403 Forbidden

Question

I have a bucket for my organization in Amazon S3 which looks like mydev.orgname

• I have a Java application that can connect to Amazon S3 with the credentials and can connect to S3, create, read files

• I have a requirement where a application reads the data from Python from same bucket. So I am using boto for this.

I do the following in oder to get the bucket

>>> import boto
>>> from boto.s3.connection import S3Connection
>>> from boto.s3.key import Key
>>> 
>>> conn = S3Connection('xxxxxxxxxxx', 'yyyyyyyyyyyyyyyyyyyyyy')
>>> conn
S3Connection:s3.amazonaws.com

Now when I try to get bucket I see error

>>> b = conn.get_bucket('mydev.myorg')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/boto/s3/connection.py", line 389, in get_bucket
    bucket.get_all_keys(headers, maxkeys=0)
  File "/Library/Python/2.7/site-packages/boto/s3/bucket.py", line 367, in get_all_keys
    '', headers, **params)
  File "/Library/Python/2.7/site-packages/boto/s3/bucket.py", line 334, in _get_all
    response.status, response.reason, body)
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>EEC05E43AF3E00F3</RequestId><HostId>v7HHmhJaLLQJZYkZ7sL4nqvJDS9yfrhfKQCgh4i8Tx+QsxKaub50OPiYrh3JjQbJ</HostId></Error>

But from the Java application everything seems to work.

Am I doing anything wrong here?

Answer 1

Giving the user a "stronger role" is not the correct solution. This is simply a problem with boto library usage. Clearly, you don't need extra permissions when using Java S3 library.

Correct way to use boto in this case is:

b = conn.get_bucket('my-bucket', validate=False)
k = b.get_key('my/cool/object.txt') # will send HEAD request to S3
...

Basically, boto by default (which is a mistake on their part IMHO), assumes you want to interact with S3 bucket. Granted, sometimes you do want that, but then you should use credentials that have permissions for S3 bucket operations. But a more popular use case is to interact with S3 objects, and in this case you don't need any special bucket-level permissions, hence the use of validate=False kwarg.