How to specify AWS Access Key ID and Secret Access Key as part of a amazon s3n URL

hadoop amazon-web-services amazon-s3 mapreduce hadoop2
user3795951 picture user3795951 · Jul 24, 2014 · Viewed 41.6k times · Source

I am passing input and output folders as parameters to mapreduce word count program from webpage.

Getting below error:

HTTP Status 500 - Request processing failed; nested exception is java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or password (respectively) of a s3n URL, or by setting the fs.s3n.awsAccessKeyId or fs.s3n.awsSecretAccessKey properties (respectively).

Answer

RickH picture RickH · Jul 24, 2014

The documentation has the format: http://wiki.apache.org/hadoop/AmazonS3

 s3n://ID:SECRET@BUCKET/Path

Related questions

Deleting file/folder from Hadoop

I'm running an EMR Activity inside a Data Pipeline analyzing log files and I get the following error when my Pipeline fails: Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://10.208.42.127:9000/home/hadoop/temp-output-s3copy already exists …

Read More
How can I access S3/S3n from a local Hadoop 2.6 installation?

I am trying to reproduce an Amazon EMR cluster on my local machine. For that purpose, I have installed the latest stable version of Hadoop as of now - 2.6.0. Now I would like to access an S3 bucket, as I …

Read More
Loading data with Hive, S3, EMR, and Recover Partitions

SOLVED: See Update #2 below for the 'solution' to this issue. ~~~~~~~ In s3, I have some log*.gz files stored in a nested directory structure like: s3://($BUCKET)/y=2012/m=11/d=09/H=10/ I'm attempting to load these into Hive on Elastic …

Read More