Host and port to use to list a directory in hdfs

Tristan picture Tristan · Nov 24, 2014 · Viewed 11.7k times · Source

First of all, I'm using HortonWorks Sandbox as Hadoop dist, with no custom configuration at all.

Once connected on the sandbox, I'm able to list files of a HDFS directory doing :

[root@sandbox ~]# hadoop fs -ls hdfs:///user/guest

but if I try to specify a host and port I get only errors :

[root@sandbox ~]# hadoop fs -ls hdfs://localhost:8020/user/guest ls: Call From sandbox.hortonworks.com/10.0.2.15 to localhost:8020 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

[root@sandbox ~]# hadoop fs -ls hdfs://localhost:9000/user/guest ls: Call From sandbox.hortonworks.com/10.0.2.15 to localhost:9000 failed on connection exception: java.net.ConnectException: Connexion refusée; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused

Once I know the correct host and port to use, I'll be able to use them in my Java call :

Path pt = new Path("hdfs://host:port/user/guest/test-text-file.txt");

Answer

Ashrith picture Ashrith · Nov 25, 2014

Check the value of property fs.defaultFS in core-site.xml this contains the ip-address/hostname and port on which NameNode daemon should bind to when it start's up.

I see that you are using hortonworks sandbox, here is the property in core-site.xml and its located in /etc/hadoop/conf/core-site.xml

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://sandbox.hortonworks.com:8020</value>
</property>

So, you could try something like this:

hadoop fs -ls hdfs://sandbox.hortonworks.com:8020/user/guest 

Or you could also replace the ip address of sandbox.hortonworks.com from its respective entry in /etc/hosts, on my vm which looks something like this:

127.0.0.1       localhost.localdomain localhost
192.168.1.3 sandbox.hortonworks.com sandbox

So, I could try this as well:

hadoop fs -ls hdfs://192.168.1.3:8020/user/guest