Count lines in large files

linux mapreduce
Dnaiel picture Dnaiel · Oct 3, 2012 · Viewed 109.9k times · Source

I commonly work with text files of ~20 Gb size and I find myself counting the number of lines in a given file very often.

The way I do it now it's just cat fname | wc -l, and it takes very long. Is there any solution that'd be much faster?

I work in a high performance cluster with Hadoop installed. I was wondering if a map reduce approach could help.

I'd like the solution to be as simple as one line run, like the wc -l solution, but not sure how feasible it is.

Any ideas?

Answer

P.P picture P.P · Oct 3, 2012

Try: sed -n '$=' filename

Also cat is unnecessary: wc -l filename is enough in your present way.

Related questions

How do I find all files containing specific text on Linux?

I'm trying to find a way to scan my entire Linux system for all files containing a specific string of text. Just to clarify, I'm looking for text within the file, not in the file name. When I was looking …

Read More
How to change permissions for a folder and its subfolders/files in one step?

I would like to change permissions of a folder and all its sub folders and files in one step (command) in Linux. I have already tried the below command but it works only for the mentioned folder: chmod 775 /opt/lampp/…

Read More
How can I recursively find all files in current and subfolders based on wildcard matching?

How can I recursively find all files in current and subfolders based on wildcard matching?

Read More