Moving files in Hadoop using the Java API?

java hadoop hdfs
Wanderer picture Wanderer · Apr 1, 2011 · Viewed 26.1k times · Source

I want to move files around in HDFS using the Java APIs. I cannot figure out a way to do this. The FileSystem class only seems to want to allow moving to and from the local file system.. but I want to keep them in HDFS and move them there.

Am I missing something basic? The only way I can figure to do it is to read it from the input stream and write it back out... and then delete the old copy (yuck).

thanks

Answer

bajafresh4life picture bajafresh4life · Apr 1, 2011

Use FileSystem.rename():

public abstract boolean rename(Path src, Path dst) throws IOException

Renames Path src to Path dst. Can take place on local fs or remote DFS.

Parameters:
src - path to be renamed
dst - new path after rename
Returns:
true if rename is successful
Throws:
IOException - on failure

Related questions

How to list all files in a directory and its subdirectories in hadoop hdfs

I have a folder in hdfs which has two subfolders each one has about 30 subfolders which,finally,each one contains xml files. I want to list all xml files giving only the main folder's path. Locally I can do this …

Read More
Write a file in hdfs with Java

I want to create a file in HDFS and write data in that. I used this code: Configuration config = new Configuration(); FileSystem fs = FileSystem.get(config); Path filenamePath = new Path("input.txt"); try { if (fs.exists(filenamePath)) { fs.delete(filenamePath, …

Read More
Reading HDFS and local files in Java

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths with the prefix file:// and HDFS paths with the prefix hdfs:// and write some code as the following Configuration configuration = …

Read More