How to get hadoop put to create directories if they don't exist

hadoop hdfs cloudera put biginsights
owly picture owly · May 7, 2014 · Viewed 45.4k times · Source

I have been using Cloudera's hadoop (0.20.2). With this version, if I put a file into the file system, but the directory structure did not exist, it automatically created the parent directories:

So for example, if I had no directories in hdfs and typed:

hadoop fs -put myfile.txt /some/non/existing/path/myfile.txt

It would create all of the directories: some, non, existing and path and put the file in there.

Now, with a newer offering of hadoop (2.2.0) this auto creation of directories is not happening. The same command above yields:

put: ` /some/non/existing/path/': No such file or directory

I have a workaround to just do hadoop fs -mkdir first, for every put, but this is not going to perform well.

Is this configurable? Any advice?

Answer

art-vybor picture art-vybor · Oct 7, 2014

Now you should use hadoop fs -mkdir -p <path>

Related questions

Why does "hadoop fs -mkdir" fail with Permission Denied?

I am using Cloudera on a VM machine that I am playing around with. Unfortunately I am having issues copying data to the HDFS, I am getting the following: [cloudera@localhost ~]$ hadoop fs -mkdir input mkdir: Permission denied: user=cloudera, …

Read More
Find port number where HDFS is listening

I want to access hdfs with fully qualified names such as : hadoop fs -ls hdfs://machine-name:8020/user I could also simply access hdfs with hadoop fs -ls /user However, I am writing test cases that should work on different distributions(…

Read More
issue Running Spark Job on Yarn Cluster

I want to run my spark Job in Hadoop YARN cluster mode, and I am using the following command: spark-submit --master yarn-cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --class com.dc.analysis.jobs.AggregationJob sparkanalitic.jar param1 param2 param3 I am …

Read More