How to count lines fast?

Jader Dias picture Jader Dias · May 23, 2011 · Viewed 22.6k times · Source

I tried unxutils' wc -l but it crashed for 1GB files. I tried this C# code

long count = 0;
using (StreamReader r = new StreamReader(f))
{
    string line;
    while ((line = r.ReadLine()) != null)
    {
        count++;
    }
}

return count;

It reads a 500MB file in 4 seconds

var size = 256;
var bytes = new byte[size];
var count = 0;
byte query = Convert.ToByte('\n');
using (var stream = File.OpenRead(file))
{
    int many;
    do
    {
        many = stream.Read(bytes, 0, size);
        count += bytes.Where(a => a == query).Count();                    
    } while (many == size);
}

Reads in 10 seconds

var count = 0;
int query = (int)Convert.ToByte('\n');
using (var stream = File.OpenRead(file))
{
    int current;
    do
    {
        current = stream.ReadByte();
        if (current == query)
        {
            count++;
            continue;
        }
    } while (current!= -1);
}

Takes 7 seconds

Is anything faster I haven't tried yet?

Answer

Jader Dias picture Jader Dias · May 23, 2011

File.ReadLines was introduced in .NET 4.0

var count = File.ReadLines(file).Count();

works in 4 seconds, the same time as the first code snippet