Client cannot authenticate via:[TOKEN, KERBEROS]

Eric Fulton picture Eric Fulton · Sep 10, 2014 · Viewed 18.8k times · Source

I'm using YarnClient to programmatically start a job. The cluster i'm running on has been kerberos-ized.

Normal map reduce jobs submitted via "yarn jar examples.jar wordcount..." work.

The job i'm trying to submit programmatically, does not. I get this error:

14/09/04 21:14:29 ERROR client.ClientService: Error happened during application submit: Application application_1409863263326_0002 failed 2 times due to AM Container for appattempt_1409863263326_0002_000002 exited with exitCode: -1000 due to: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "yarn-c1-n1.clouddev.snaplogic.com/10.184.28.108"; destination host is: "yarn-c1-cdh.clouddev.snaplogic.com":8020; .Failing this attempt.. Failing the application. 14/09/04 21:14:29 ERROR client.YClient: Application submission failed

The code looks something like this:

ClientContext context = createContextFrom(args);
YarnConfiguration configuration = new YarnConfiguration();
YarnClient yarnClient = YarnClient.createYarnClient();
yarnClient.init(configuration);
ClientService client = new ClientService(context, yarnClient, new InstallManager(FileSystem.get(configuration)));
LOG.info(Messages.RUNNING_CLIENT_SERVICE);
boolean result = client.execute();

I had thought that perhaps adding something to the effect of:

yarnClient.getRMDelegationToken(new Text(InetAddress.getLocalHost().getHostAddress()));

Could perhaps assuage my woes, but that doesn't seem to help either. Any help would be greatly appreciated.

Answer

Eric Fulton picture Eric Fulton · Sep 16, 2014

Alright, well after hours and hours and hours we have this figured out. For all following generations of coders, forever plagued by hadoop's lack of documentation:

You must grab the tokens from UserGroupInformation object with a call to get credentials. Then you must set the tokens on the ContainerLaunchContext.