Connecting to Kerberrized HDFS , java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name;

java hadoop kerberos cloudera keytab
avinash patil picture avinash patil · Feb 10, 2016 · Viewed 10.2k times · Source

I am trying to connect to a Kerberized hdfs cluster , with below code , with same below code i am able to access hbase ofcourse with HBaseConfiguration,

Configuration config = new Configuration();
config.set("hadoop.security.authentication", "Kerberos");

UserGroupInformation.setConfiguration(config);
UserGroupInformation ugi = null;
ugi = UserGroupInformation.loginUserFromKeytabAndReturnUGI("me@EXAMPLE>COM","me.keytab");
model = ugi.doAs((PrivilegedExceptionAction<Map<String,Object>>) () -> { 
  testHadoop(hcb.gethDFSConfigBean());
  return null;
});

I have been able to successfully access Solr, Impala with same keytab and principal, I am getting this weird Failed to find service name for hdfs.

Please look at below stack trace

java.io.IOException: Failed on local exception: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name; Host Details : local host is: "Securonix-int3.local/10.0.4.36"; destination host is: "sobd189.securonix.com":8020; 
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
    at org.apache.hadoop.ipc.Client.call(Client.java:1472)
    at org.apache.hadoop.ipc.Client.call(Client.java:1399)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:752)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at com.sun.proxy.$Proxy10.getFileInfo(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1988)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1118)
    at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1400)
    at com.securonix.application.ui.uiUtil.SnyperUIUtil.lambda$main$4(SnyperUIUtil.java:1226)
    at com.securonix.application.ui.uiUtil.SnyperUIUtil$$Lambda$6/1620890840.run(Unknown Source)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at com.securonix.application.ui.uiUtil.SnyperUIUtil.main(SnyperUIUtil.java:1216)
Caused by: java.io.IOException: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
    at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:643)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:730)
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
    at org.apache.hadoop.ipc.Client.call(Client.java:1438)
    ... 23 more
Caused by: java.lang.IllegalArgumentException: Failed to specify server's Kerberos principal name
    at org.apache.hadoop.security.SaslRpcClient.getServerPrincipal(SaslRpcClient.java:322)
    at org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:231)
    at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
    at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
    at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:553)
    at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:368)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:722)
    at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:718)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:717)

After i enabled the debug code for Kerberos i got the below debug log when i call FileSystem.get(); Kerberor debug log :

Java config name: null Java config name: null Native config name: /etc/krb5.conf Native config name: /etc/krb5.conf Loaded from native config Loaded from native config 16/02/22 15:53:14 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Java config name: null Java config name: null Native config name: /etc/krb5.conf Native config name: /etc/krb5.conf Loaded from native config Loaded from native config

KdcAccessibility: reset >>> KdcAccessibility: reset KdcAccessibility: reset >>> KdcAccessibility: reset KeyTabInputStream, readName(): EXAMPLE.COM >>> KeyTabInputStream, readName(): EXAMPLE.COM KeyTabInputStream, readName(): securonix >>> KeyTabInputStream, readName(): securonix KeyTab: load() entry length: 55; type: 23 >>> KeyTab: load() entry length: 55; type: 23 KeyTabInputStream, readName(): EXAMPLE.COM >>> KeyTabInputStream, readName(): EXAMPLE.COM KeyTabInputStream, readName(): securonix >>> KeyTabInputStream, readName(): securonix KeyTab: load() entry length: 71; type: 18 >>> KeyTab: load() entry length: 71; type: 18 Looking for keys for: [email protected] Looking for keys for: [email protected] Added key: 18version: 1 Added key: 18version: 1 Added key: 23version: 1 Added key: 23version: 1 Looking for keys for: [email protected] Looking for keys for: [email protected] Added key: 18version: 1 Added key: 18version: 1 Added key: 23version: 1 Added key: 23version: 1 default etypes for default_tkt_enctypes: 18 18 16. default etypes for default_tkt_enctypes: 18 18 16. KrbAsReq creating message >>> KrbAsReq creating message KrbKdcReq send: kdc=sobd189.securonix.com TCP:88, timeout=30000, number of retries =3, #bytes=139 >>> KrbKdcReq send: kdc=sobd189.securonix.com TCP:88, timeout=30000, number of retries =3, #bytes=139 KDCCommunication: kdc=sobd189.securonix.com TCP:88, timeout=30000,Attempt =1, #bytes=139 >>> KDCCommunication: kdc=sobd189.securonix.com TCP:88, timeout=30000,Attempt =1, #bytes=139 DEBUG: TCPClient reading 639 bytes >>>DEBUG: TCPClient reading 639 bytes KrbKdcReq send: #bytes read=639 >>> KrbKdcReq send: #bytes read=639 KdcAccessibility: remove sobd189.securonix.com >>> KdcAccessibility: remove sobd189.securonix.com Looking for keys for: [email protected] Looking for keys for: [email protected] Added key: 18version: 1 Added key: 18version: 1 Added key: 23version: 1 Added key: 23version: 1 EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType >>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType KrbAsRep cons in KrbAsReq.getReply securonix

Interestingly when i use api of the file system like hdfs.exists()

 >>>KinitOptions cache name is /tmp/krb5cc_501
 >> Acquire default native Credentials
 default etypes for default_tkt_enctypes: 18 18 16.
 >>> Found no TGT's in LSA

Answer

Bryan Bende picture Bryan Bende · Mar 15, 2016

I believe the problem is that HDFS expects the Configuration to have a value for dfs.datanode.kerberos.principal, which is the principal of the datanodes, and it is missing in this case.

I had this same problem when I created a Configuration instance from only core-site.xml and forgot to add hdfs-site.xml. As soon as I added hdfs-site.xml it started working, and hdfs-site.xml had:

 <property>
      <name>dfs.datanode.kerberos.principal</name>
      <value>....</value>
 </property>

Hope this helps.

Related questions

Should I call ugi.checkTGTAndReloginFromKeytab() before every action on hadoop?

In my server application I'm connecting to Kerberos secured Hadoop cluster from my java application. I'm using various components like the HDFS file system, Oozie, Hive etc. On the application startup I do call UserGroupInformation.loginUserFromKeytabAndReturnUGI( ... ); This returns me UserGroupInformation …

Read More
Hadoop "Unable to load native-hadoop library for your platform" warning

I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable I'm running …

Read More
Hadoop cluster setup - java.net.ConnectException: Connection refused

I want to setup a hadoop-cluster in pseudo-distributed mode. I managed to perform all the setup-steps, including startuping a Namenode, Datanode, Jobtracker and a Tasktracker on my machine. Then I tried to run some exemplary programms and faced the java.…

Read More