Find out actual disk usage in HDFS

hadoop hdfs bigdata diskspace
Aliaxander picture Aliaxander · Aug 7, 2015 · Viewed 18.7k times · Source

Is there a way to find out how much space is consumed in HDFS? I used 

hdfs dfs -df

but it seems to be not relevant cause after deleting huge amount of data with

hdfs dfs -rm -r -skipTrash

the previous comand displays changes not at once but after several minutes (I need up-to-date disk usage info).

Answer

Balduz picture Balduz · Aug 7, 2015

To see the space consumed by a particular folder try:

hadoop fs -du -s /folder/path

And if you want to see the usage, space consumed, space available, etc. of the whole HDFS:

hadoop dfsadmin -report

Related questions

How to copy data from one HDFS to another HDFS?

I have two HDFS setup and want to copy (not migrate or move) some tables from HDFS1 to HDFS2. How to copy data from one HDFS to another HDFS? Is it possible via Sqoop or other command line?

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through …

Read More