How to track which data block is in which data node in hadoop?

hadoop replication
Varshith picture Varshith · Jun 16, 2011 · Viewed 23.6k times · Source

If a data block is replicated, in which data node will it be replicated to? Is there any tool to show where the replicated blocks are present?

Answer

user188041 picture user188041 · Jun 16, 2011

If you know the filename, you can look this up through the DFS browser.

Go to your namenode web interface, say "browse the filesystem" and navigate to the file you're interested in. In the bottom of the page, there will be a list of all blocks in the file, and where each of those blocks is located.

NOTE: It looks like this when you click on an actual file within the HDFS filesystem.

ss#1

Alternatively, you could run:

hadoop fsck / -files -blocks -locations

Which will report on all blocks and all their locations.

Related questions

Data Replication error in Hadoop

I am implementing the Hadoop Single Node Cluster on my machine by following Michael Noll's tutorial and have come across data replication error: Here's the full error message: > hadoop@laptop:~/hadoop$ bin/hadoop dfs -copyFromLocal > tmp/testfiles testfiles &…

Read More
How do you retrieve the replication factor info in Hdfs files?

I have set the replication factor for my file as follows: hadoop fs -D dfs.replication=5 -copyFromLocal file.txt /user/xxxx When a NameNode restarts, it makes sure under-replicated blocks are replicated. Hence the replication info for the file is …

Read More
HDFS Reduced Replication Factor

I've reduced the replication factor from 3 to 1, yet do not see any activity from the namenode or between datanodes to remove overly-replicated HDFS file blocks. Is there a way to monitor or force the replication job?

Read More