java : writing large files?

java java-io bufferedwriter
Ashika Umanga Umagiliya picture Ashika Umanga Umagiliya · Jan 7, 2010 · Viewed 14.6k times · Source

Greetings , I get huge number of records from database and write into a file.I was wondering what the best way to write huge files. (1Gb - 10Gb).

Currently I am using BufferedWriter

BufferedWriter mbrWriter=new BufferedWriter(new FileWriter(memberCSV));
while(done){
 //do writings
}
mbrWriter.close();

Answer

BalusC picture BalusC · Jan 7, 2010

If you really insist using Java for this, then the best way would be to write immediately as soon as the data comes in and thus not to collect all the data from ResultSet into Java's memory first. You would need at least that much of free memory in Java otherwise.

Thus, do e.g.

while (resultSet.next()) {
    writer.write(resultSet.getString("columnname"));
    // ...
}

That said, most decent DB's ships with builtin export-to-CSV capabilities which are undoubtely way more efficient than you could ever do in Java. You didn't mention which one you're using, but if it was for example MySQL, you could have used the LOAD DATA INFILE for this. Just refer the DB-specific documentation. Hope this gives new insights.

Related questions

How to get the current working directory in Java?

I want to access my current working directory using java. My code : String current = new java.io.File( "." ).getCanonicalPath(); System.out.println("Current dir:"+current); String currentDir = System.getProperty("user.dir"); System.out.println("Current dir using System:" +currentDir); Output: …

Read More
Getting java.net.SocketTimeoutException: Connection timed out in android

I'm relatively new to android development. I'm developing an android application where i'm sending request to web server and parsing json objects. Frequently i'm getting java.net.SocketTimeoutException: Connection timed out exception while communicating with the server. Some times it …

Read More
Create a directory if it does not exist and then create the files in that directory as well

Condition is if directory exists it has to create files in that specific directory with out creating a new directory. The below code only creating a file with new directory but not for existing directory . For example the directory name …

Read More