what's the fastest way to scan a very large file in java?

java performance string-search
chacko picture chacko · Feb 3, 2011 · Viewed 10.9k times · Source

Imagine I have a very large text file. Performance really matters.

All I want to do is to scan it to look for a certain string. Maybe I want to count how many I have of those, but it really is not the point.

The point is: what's the fastest way ?

I don't care about maintainance it needs to be fast.

Fast is key.

Answer

Joel picture Joel · Feb 3, 2011

For a one off search use a Scanner, as suggested here

A simple technique that could well be considerably faster than indexOf() is to use a Scanner, with the method findWithinHorizon(). If you use a constructor that takes a File object, Scanner will internally make a FileChannel to read the file. And for pattern matching it will end up using a Boyer-Moore algorithm for efficient string searching.

Related questions

Why is processing a sorted array faster than processing an unsorted array?

Here is a piece of C++ code that shows some very peculiar behavior. For some strange reason, sorting the data miraculously makes the code almost six times faster: #include <algorithm> #include <ctime> #include <iostream> …

Read More
How can I read a large text file line by line using Java?

I need to read a large text file of around 5-6 GB line by line using Java. How can I do this quickly?

Read More
Convert Set to List without creating new List

I am using this code to convert a Set to a List: Map<String, List<String>> mainMap = new HashMap<>(); for (int i=0; i < something.size(); i++) { Set<String> set = getSet(...); //returns …

Read More