Hadoop copy a directory?

hadoop hdfs
rakeshr picture rakeshr · Jan 17, 2011 · Viewed 72.1k times · Source

Is there an HDFS API that can copy an entire local directory to the HDFS? I found an API for copying files but is there one for directories?

Answer

Spike Gronim picture Spike Gronim · Jan 17, 2011

Use the Hadoop FS shell. Specifically:

$ hadoop fs -copyFromLocal /path/to/local hdfs:///path/to/hdfs

If you want to do it programmatically, create two FileSystems (one Local and one HDFS) and use the FileUtil class

Related questions

Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through …

Read More
hadoop copy a local file system folder to HDFS

I need to copy a folder from local file system to HDFS. I could not find any example of moving a folder(including its all subfolders) to HDFS $ hadoop fs -copyFromLocal /home/ubuntu/Source-Folder-To-Copy HDFS-URI

Read More