HDFS Reduced Replication Factor

hadoop replication hdfs
Carl Sagan picture Carl Sagan · Jul 23, 2013 · Viewed 9.2k times · Source

I've reduced the replication factor from 3 to 1, yet do not see any activity from the namenode or between datanodes to remove overly-replicated HDFS file blocks. Is there a way to monitor or force the replication job?

Answer

Charles Menguy picture Charles Menguy · Jul 23, 2013

Changing dfs.replication will only apply to new files you create, but will not modify the replication factor for the already existing files.

To change replication factor for files that already exist, you could run the following command which will be run recursively on all files in HDFS:

hadoop dfs -setrep -w 1 -R /

Related questions

Data Replication error in Hadoop

I am implementing the Hadoop Single Node Cluster on my machine by following Michael Noll's tutorial and have come across data replication error: Here's the full error message: > hadoop@laptop:~/hadoop$ bin/hadoop dfs -copyFromLocal > tmp/testfiles testfiles &…

Read More
How to track which data block is in which data node in hadoop?

If a data block is replicated, in which data node will it be replicated to? Is there any tool to show where the replicated blocks are present?

Read More
How do you retrieve the replication factor info in Hdfs files?

I have set the replication factor for my file as follows: hadoop fs -D dfs.replication=5 -copyFromLocal file.txt /user/xxxx When a NameNode restarts, it makes sure under-replicated blocks are replicated. Hence the replication info for the file is …

Read More