Get a few lines of HDFS data

hadoop hdfs
Unmesha SreeVeni picture Unmesha SreeVeni · Feb 28, 2014 · Viewed 50.9k times · Source

I am having a 2 GB data in my HDFS.

Is it possible to get that data randomly. Like we do in the Unix command line 

cat iris2.csv |head -n 50

Answer

Viacheslav Rodionov picture Viacheslav Rodionov · Feb 28, 2014

Native head

hadoop fs -cat /your/file | head

is efficient here, as cat will close the stream as soon as head will finish reading all the lines.

To get the tail there is a special effective command in hadoop:

hadoop fs -tail /your/file

Unfortunately it returns last kilobyte of the data, not a given number of lines.

Related questions

Spark - load CSV file as DataFrame?

I would like to read a CSV in spark and convert it as DataFrame and store it in HDFS with df.registerTempTable("table_name") I have tried: scala> val df = sqlContext.load("hdfs:///csv/file/dir/file.csv") Error …

Read More
How to copy file from HDFS to the local file system

How to copy file from HDFS to the local file system . There is no physical location of a file under the file , not even directory . how can i moved them to my local for further validations.i am tried through …

Read More
hadoop copy a local file system folder to HDFS

I need to copy a folder from local file system to HDFS. I could not find any example of moving a folder(including its all subfolders) to HDFS $ hadoop fs -copyFromLocal /home/ubuntu/Source-Folder-To-Copy HDFS-URI

Read More