hdfs copy multiple files to same target directory

hadoop copy hdfs subdirectory cp
R. Sluiter picture R. Sluiter · Dec 16, 2016 · Viewed 12.2k times · Source

I learned that if you want to copy multiple files from one hadoop folder to another hadoop folder you can better create one big 'hdfs dfs -cp' statement with lots of components, instead of creating several hdfs dfs -cp statements. With 'better' I mean that it will improve the overal time it takes to copy files: one command is quicker than several separate -cp commands run after each other.

When I do this and my target directory is the same for all files that I want to copy I get a warning.

I'm executing the following command:

hdfs dfs -cp -f /path1/file1 /pathx/target /path2/file2 /pathx/target /path3/file3 /pathx/target

After executing it I get the following warning returned:

cp: `/pathx/target' to `/pathx/target/target': is a subdirectory of itself

Although I get this weird warning the copy itself succeeds like it should. Is this a bug or am I missing something?

Answer

Luca Natali picture Luca Natali · Dec 16, 2016

Try to use the following syntax:

hadoop fs -cp /path1/file1 /path2/file2 path3/file3 /pathx/target

Related questions

How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through …

Read More
Could not find uri with key dfs.encryption.key.provider.uri to create a keyProvider in HDFS encryption for CDH 5.4

CDH Version: CDH5.4.5 Issue: When HDFS Encryption is enabled using KMS available in Hadoop CDH 5.4 , getting error while putting file into encryption zone. Steps: Steps for Encryption of Hadoop as follows: Creating a key [SUCCESS] [tester@master ~]$ hadoop key create …

Read More
Hadoop "Unable to load native-hadoop library for your platform" warning

I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable I'm running …

Read More