The way to check a HDFS directory's size?

hadoop command-line directory hdfs
Cheng picture Cheng · Jun 28, 2011 · Viewed 161.2k times · Source

I know du -sh in common Linux filesystems. But how to do that with HDFS?

Answer

Matt D picture Matt D · Jun 28, 2011

Prior to 0.20.203, and officially deprecated in 2.6.0:

hadoop fs -dus [directory]

Since 0.20.203 (dead link) 1.0.4 and still compatible through 2.6.0:

hdfs dfs -du [-s] [-h] URI [URI …]

You can also run hadoop fs -help for more info and specifics.

Related questions

Hadoop "Unable to load native-hadoop library for your platform" warning

I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable I'm running …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
connect to host localhost port 22: Connection refused

While installing hadoop in my local machine , i got following error ssh -vvv localhost OpenSSH_5.5p1, OpenSSL 1.0.0e-fips 6 Sep 2011 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to localhost [127.0.0.1] port 22. …

Read More