How to find Hadoop hdfs directory on my system?

linux azure hadoop hdfs hortonworks-data-platform
N.. picture N.. · Apr 2, 2016 · Viewed 43.6k times · Source

How to find Hadoop HDFS directory on my system? I need this to run following command - 

hadoop dfs -copyFromLocal <local-dir> <hdfs-dir>

In this command I don't knon my hdfs-dir.

Not sure if its helpful or not but I ran following command and got this output - 

 hdfs dfs -ls
-rw-r--r--   3 popeye hdfs  127162942 2016-04-01 19:47 .

In hdfs-site.xml, I found following entry - 

<property>
      <name>dfs.datanode.data.dir</name>
      <value>/hadoop/hdfs/data</value>
      <final>true</final>
</property>

I tried to run following command but it gives error -

[root@sandbox try]# hdfs dfs -copyFromLocal 1987.csv /hadoop/hdfs/data
copyFromLocal: `/hadoop/hdfs/data': No such file or directory

FYI - I am doing all this on hortonworks sandbox on azure server.

Answer

BruceWayne picture BruceWayne · Apr 3, 2016

Your approach is wrong or may be understanding is wrong

dfs.datanode.data.dir, is where you want to store your data blocks

If you type hdfs dfs -ls / you will get list of directories in hdfs. Then you can transfer files from local to hdfs using -copyFromLocal or -put to a particular directory or using -mkdir you can create new directory

Refer below link for more information

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html

Related questions

Linux python3 - Can't open lib 'SQL Server'

I am trying to connect to an Microsoft Azure SQL server database. This is how i am trying to connect: conn = pyodbc.connect('DRIVER={SQL Server};SERVER=%s' % (self.config.get("Sql", "DataSource")), user= self.config.get("Sql", "UserId"), password=…

Read More
How do I find all files containing specific text on Linux?

I'm trying to find a way to scan my entire Linux system for all files containing a specific string of text. Just to clarify, I'm looking for text within the file, not in the file name. When I was looking …

Read More
How to change permissions for a folder and its subfolders/files in one step?

I would like to change permissions of a folder and all its sub folders and files in one step (command) in Linux. I have already tried the below command but it works only for the mentioned folder: chmod 775 /opt/lampp/…

Read More