SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] While getting spark context

authentication apache-spark yarn kerberos hadoop2
Avinash chavan picture Avinash chavan · Oct 9, 2017 · Viewed 7.2k times · Source

Installation Info :-

Hadoop version :- 2.6.5 Spark Version :- 2.1.0 And Kerberos

I am trying to get the spark context in yarn mode with kerberos authentication but below exception.

Code :-

public static void main(String[] args) {

        Configuration conf = new Configuration();

        conf.set("fs.defaultFS", "hdfs://localhost:9000");
        conf.set("hadoop.security.authentication", "kerberos");
        conf.set("hadoop.security.authorization", "true");

        UserGroupInformation.setConfiguration(conf);

        System.out.println("Security enabled " + UserGroupInformation.isSecurityEnabled());


        SparkConf sparkConf = new SparkConf().setAppName("Spark shell").setMaster("yarn");
        SparkSession  sparkSession = SparkSession.builder().config(sparkConf).getOrCreate();

        System.out.println(sparkSession.version() + " : " + sparkSession.sparkContext());
  }

Its prints security is enabled like:- Security enabled true

Exception :-

org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)

Please advise me to resolve this error.

Answer

Related questions

Locally reading S3 files through Spark (or better: pyspark)

I want to read an S3 file from my (local) machine, through Spark (pyspark, really). Now, I keep getting authentication errors like java.lang.IllegalArgumentException: AWS Access Key ID and Secret Access Key must be specified as the username or …

Read More
Git push requires username and password

I cloned a Git repository from my GitHub account to my PC. I want to work with both my PC and laptop, but with one GitHub account. When I try to push to or pull from GitHub using my PC, …

Read More
Git push results in "Authentication Failed"

I have been using Github for a little while and I have been fine with git add, git commit, and git push so far with no problems. Suddenly I am having an error that says: fatal: Authentication Failed In the …

Read More