Groovy: Appending to large files

groovy file-writing
Eqbal picture Eqbal · Nov 5, 2010 · Viewed 16.2k times · Source

How do I append to large files efficiently. I have a process that has to continually append to a file and as the file size grows the performance seems to slow down as well. Is there anyway to specify a large buffer size with the append

Answer

robbbert picture robbbert · Nov 5, 2010

While Don's approach is valid, in general (it will throw an exception, though, because of a syntax error, and you need to flush() a BufferedOutputStream), I've been planning to elaborate further (which takes its time).

Groovy does not provide special objects for I/O operations. Thus, you would use Java's FileOutputStream (for writing bytes) or FileWriter (for writing strings). Both provide a constructor that takes a boolean append parameter.
For both, there exist decorators (BufferedOutputStream and BufferedWriter), which are buffered. "Buffering", in this scope, means, that contents are not necessarily written instantly to the underlying stream, and thus, there's a potential for I/O optimization.

Don already provided a sample for the BufferedOutputStream, and here's one for the BufferedWriter:

File file = new File("foo")
if (file.exists()) {
    assert file.delete()
    assert file.createNewFile()
}

boolean append = true
FileWriter fileWriter = new FileWriter(file, append)
BufferedWriter buffWriter = new BufferedWriter(fileWriter)

100.times { buffWriter.write "foo" }

buffWriter.flush()
buffWriter.close()

While Groovy does not provide its own I/O objects, the Groovy JDK (GDK) enhances several Java types by adding convenience methods. In the scope of I/O outputting, the OutputStream and the File types are relevant.

So, finally, you can work with those the "Groovy way":

new File("foo").newOutputStream().withWriter("UTF-8") { writer ->
    100.times { writer.write "foo" + it }
}

EDIT: As per your further inquiry:

None of the GDK methods allows for setting a buffer size.

The above "Groovy" code will overwrite the file if called repeatedly. - In contrast, the following piece of code will append the string to an existing file and, thus, can be called repeatedly:

new File("foo").withWriterAppend("UTF-8") { it.write("bar") }

Related questions

Hibernate: Automatically creating/updating the db tables based on entity classes

I have the following entity class (in Groovy): import javax.persistence.Entity import javax.persistence.Id import javax.persistence.GeneratedValue import javax.persistence.GenerationType @Entity public class ServerNode { @Id @GeneratedValue(strategy = GenerationType.AUTO) Long id String firstName String lastName } and …

Read More
How do I get the output of a shell command executed using into a variable from Jenkinsfile (groovy)?

I have something like this on a Jenkinsfile (Groovy) and I want to record the stdout and the exit code in a variable in order to use the information later. sh "ls -l" How can I do this, especially as …

Read More
Converting a string to int in Groovy

I have a String that represents an integer value and would like to convert it to an int. Is there a groovy equivalent of Java's Integer.parseInt(String)?

Read More