ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid

Question

s3cmd was installed from epel-testing repo by running:

yum --enablerepo epel-testing install s3cmd

Then I invoked the configuration tool with s3cmd --configure but I got this error:

Test access with supplied credentials? [Y/n] 
Please wait, attempting to list all buckets...
ERROR: Test failed: 400 (InvalidToken): The provided token is malformed or otherwise invalid.

Invoked as: /usr/bin/s3cmd --configureProblem: AttributeError: 'S3Error' object has no attribute 'find'
S3cmd:   1.5.0-beta1
python:   2.6.8 (unknown, Mar 14 2013, 09:31:22) 
[GCC 4.6.2 20111027 (Red Hat 4.6.2-2)]

Traceback (most recent call last):
  File "/usr/bin/s3cmd", line 2323, in <module>
    main()
  File "/usr/bin/s3cmd", line 2221, in main
    run_configure(options.config, args)
  File "/usr/bin/s3cmd", line 1704, in run_configure
    if e.find('403') != -1:
AttributeError: 'S3Error' object has no attribute 'find'

I'm sure the keys are correct.

Do you have any idea about this?

---

UPDATE Fri Mar 21 22:44:42 ICT 2014

Found some clues when running in debug mode.

With the same credentials, on the worked system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:07:18 +0000\n/'

On the failed system:

DEBUG: SignHeaders: 'GET\n\n\n\nx-amz-date:Fri, 21 Mar 2014 07:40:56 +0000\nx-amz-security-token:AQoDYXdzENb...\n/'

This security token was taken from the metadata:

# wget -O - -q 'http://169.254.169.254/latest/meta-data/iam/security-credentials/myrole'
{
  "Code" : "Success",
  "LastUpdated" : "2014-03-21T12:45:27Z",
  "Type" : "AWS-HMAC",
  "AccessKeyId" : "foo",
  "SecretAccessKey" : "bar",
  "Token" : "AQoDYXdzENb....",
  "Expiration" : "2014-03-21T19:18:02Z"
}

So the my question should be changed to: why does sometimes s3cmd add x-amz-security-token to the header when running --configure (I am not using --add-header)?

Answer 1

After installing s3cmd, setting up an IAM and attaching an "Amazon S3 Full Access" role - I too encountered the "The provided token is malformed or otherwise invalid" error. So next, I created an IAM and attached a policy with admin credentials (everything).

error appeared again using this new IAM.

From a github.com thread, @mdomsch mentioned some possible issues using EC2 with an embedded IAM roles - which is what my EC2 that was using the s3cmd had.

So next, I tried running.

s3cmd --access_key=xxxx --secret_key=xxxxxxxxxxxxx ls

using the IAM keys with the S3 policy and it worked.

So s3cmd takes as a default the credentials of embedded roles in the EC2. I am sure that if you spin up another EC2 with a role that has access to S3 - you will not get this "The provided token is malformed or otherwise invalid" error.

However - please DO NOT stick -access_key=xxxx --secret_key=xxxxxxxxxxxxx parameters into any script using s3cmd. Where ever possible - try to embed roles into EC2s when you fire them up (good security practice)

Anyway - to test - I fired up another EC2 w/out any embedded IAM roles, installed s3cmd, configured - and everything worked as expected using my IAM user that had the S3 policy.