Reading and processing big text file of 25GB

java file file-io large-files
user1142292 picture user1142292 · Jan 11, 2012 · Viewed 36.3k times · Source

I have to read a big text file of, say, 25 GB and need to process this file within 15-20 minutes. This file will have multiple header and footer section.

I tried CSplit to split this file based on header, but it is taking around 24 to 25 min to split it to a number of files based on header, which is not acceptable at all.

I tried sequential reading and writing by using BufferReader and BufferWiter along with FileReader and FileWriter. It is taking more than 27 min. Again, it is not acceptable.

I tried another approach like get the start index of each header and then run multiple threads to read file from specific location by using RandomAccessFile. But no luck on this.

How can I achieve my requirement?

Possible duplicate of:

Read large files in Java

Answer

xikkub picture xikkub · Jan 11, 2012

Try using a large buffer read size (for example, 20MB instead of 2MB) to process your data quicker. Also don't use a BufferedReader because of slow speeds and character conversions.

This question has been asked before: Read large files in Java

Related questions

How do I create a Java string from the contents of a file?

I've been using the idiom below for some time now. And it seems to be the most wide-spread, at least on the sites I've visited. Is there a better/different way to read a file into a string in Java? …

Read More
How do I save a String to a text file using Java?

In Java, I have text from a text field in a String variable called "text". How can I save the contents of the "text" variable to a file?

Read More
What is simplest way to read a file into String?

I am trying to read a simple text file into a String. Of course there is the usual way of getting the input stream and iterating with readLine() and reading contents into String. Having done this hundreds of times in …

Read More