How to read a large (1 GB) txt file in .NET?

c# .net streamreader
Jeevan Bhatt picture Jeevan Bhatt · Nov 25, 2010 · Viewed 83.2k times · Source

I have a 1 GB text file which I need to read line by line. What is the best and fastest way to do this?

private void ReadTxtFile()
{            
    string filePath = string.Empty;
    filePath = openFileDialog1.FileName;
    if (string.IsNullOrEmpty(filePath))
    {
        using (StreamReader sr = new StreamReader(filePath))
        {
            String line;
            while ((line = sr.ReadLine()) != null)
            {
                FormatData(line);                        
            }
        }
    }
}

In FormatData() I check the starting word of line which must be matched with a word and based on that increment an integer variable.

void FormatData(string line)
{
    if (line.StartWith(word))
    {
        globalIntVariable++;
    }
}

Answer

TalentTuner picture TalentTuner · Nov 25, 2010

If you are using .NET 4.0, try MemoryMappedFile which is a designed class for this scenario.

You can use StreamReader.ReadLine otherwise.

Related questions

How to Find And Replace Text In A File With C#

My code so far StreamReader reading = File.OpenText("test.txt"); string str; while ((str = reading.ReadLine())!=null) { if (str.Contains("some text")) { StreamWriter write = new StreamWriter("test.txt"); } } I know how to find the text, but I have no idea …

Read More
Reading large text files with streams in C#

I've got the lovely task of working out how to handle large files being loaded into our application's script editor (it's like VBA for our internal product for quick macros). Most files are about 300-400 KB which is fine loading. …

Read More
An elegant way to consume (all bytes of a) BinaryReader?

Is there an elegant to emulate the StreamReader.ReadToEnd method with BinaryReader? Perhaps to put all the bytes into a byte array? I do this: read1.ReadBytes((int)read1.BaseStream.Length); ...but there must be a better way.

Read More