Wildcard in Hadoop's FileSystem listing API calls

java hadoop hdfs wildcard
snooze92 picture snooze92 · Jul 9, 2014 · Viewed 15.2k times · Source

tl;dr: To be able to use wildcards (globs) in the listed paths, one simply has to use globStatus(...) instead of listStatus(...).

Answer

Mukesh S picture Mukesh S · Jul 9, 2014

Instead of listStatus you can try hadoops globStatus. Hadoop provides two FileSystem method for processing globs:

public FileStatus[] globStatus(Path pathPattern) throws IOException
public FileStatus[] globStatus(Path pathPattern, PathFilter filter) throws IOException

An optional PathFilter can be specified to restrict the matches further.

For more description you can check Hadoop:Definitive Guide here

Hope it helps..!!!

Related questions

How to list all files in a directory and its subdirectories in hadoop hdfs

I have a folder in hdfs which has two subfolders each one has about 30 subfolders which,finally,each one contains xml files. I want to list all xml files giving only the main folder's path. Locally I can do this …

Read More
Write a file in hdfs with Java

I want to create a file in HDFS and write data in that. I used this code: Configuration config = new Configuration(); FileSystem fs = FileSystem.get(config); Path filenamePath = new Path("input.txt"); try { if (fs.exists(filenamePath)) { fs.delete(filenamePath, …

Read More
Reading HDFS and local files in Java

I want to read file paths irrespective of whether they are HDFS or local. Currently, I pass the local paths with the prefix file:// and HDFS paths with the prefix hdfs:// and write some code as the following Configuration configuration = …

Read More