hadoop fs -rm -skipTrash doesn't work

hadoop rm distcp
Evgenii picture Evgenii · Aug 9, 2017 · Viewed 14.9k times · Source

I copied some files from a directory to directory using

hadoop distcp -Dmapreduce.job.queuename=adhoc /user/comverse/data/$CURRENT_DATE_NO_DASH_*/*rcr.gz /apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin04/file_rcr/

I stopped the scipt before it finished and the remained a lot of .distcp.tmp.attempt and files that fnished moving in the dst directory

Now I want to clean the dst directory. After running 

hadoop fs -rm -skipTrash /apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin04/file_mta/*

most of the files were deleted, but some remained(at least that's what HUE shows). The strange thing is, every time I run hadoop fs -rm -skipTrash, according to HUE, the number of remaining files changes to more or less.

I tried 

hadoop fs -ls /apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin04/file_mta/

and saw that some of the files that should be deleted were still there. Then I run

hadoop fs -rm -skipTrash /apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin04/file_mta/*

a dozen more times and there were always more files to delete(There still are). What is happening?

ALSO

Each time I refresh the page in hue, the number of files grows. HALP.

EDIT

It seems that stopping distcp in the command line doesn't actually kill the job. That was the reason.

Answer

R Palanivel-Tamilnadu India picture R Palanivel-Tamilnadu India · Aug 11, 2017

You could use this "-R":

This remove all the file from your hdfs location.

hadoop fs -rm -R -skipTrash /apps/hive/warehouse/arstel.db/fair_usage/fct_evkuzmin04/file_mta/*

Related questions

Hadoop "Unable to load native-hadoop library for your platform" warning

I'm currently configuring hadoop on a server running CentOs. When I run start-dfs.sh or stop-dfs.sh, I get the following error: WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable I'm running …

Read More
Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
connect to host localhost port 22: Connection refused

While installing hadoop in my local machine , i got following error ssh -vvv localhost OpenSSH_5.5p1, OpenSSL 1.0.0e-fips 6 Sep 2011 debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to localhost [127.0.0.1] port 22. …

Read More