Transfer file out from HDFS

hadoop hdfs data-transfer
dipeshtech picture dipeshtech · Aug 29, 2012 · Viewed 29.7k times · Source

I want to transfer files out from HDFS to local filesystem of a different server which is not in hadoop cluster but in the network.

I could have done: 

hadoop fs -copyToLocal <src> <dest>
and then scp/ftp <toMyFileServer>.

As the data is huge and due to limited space on local filesystem of hadoop gateway machine, I wanted to avoid this and sent data directly to my file server.

Please help with some pointers on how to handle this issue.

Answer

cabad picture cabad · Jun 5, 2013

This is the simplest way to do it:

ssh <YOUR_HADOOP_GATEWAY> "hdfs dfs -cat <src_in_HDFS> " > <local_dst>

It works for binary files too.

Related questions

Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through …

Read More
hadoop copy a local file system folder to HDFS

I need to copy a folder from local file system to HDFS. I could not find any example of moving a folder(including its all subfolders) to HDFS $ hadoop fs -copyFromLocal /home/ubuntu/Source-Folder-To-Copy HDFS-URI

Read More