How to use a BufferedWriter in Python?

philgiese picture philgiese · Jan 10, 2012 · Viewed 7k times · Source

I am facing the following problem: I'm trying to implement a simulator for supply chains. These will produce a lot of EPCIS Events (events that occur at RFID readers). These events should then be written to a csv file in order to load them into any database and run analytical algorithms on them.

The simulator is implemented using python and works fine. What I'm now trying to do is buffered writing of the events to the file in order to decrease the time, that is needed to access the disk. Browsing the python documentation I stumbled upon the io.BufferedWriter, which sounds exactly like the thing I was looking for. Anyway, I can't quite get it to work.

Here's what I did so far. I implemented my CsvWriter class, that inherits from RawIOBase and manages the file handle. When it is instantiated, it will create a BufferedWriter, handing in itself, as the raw parameter (might that already be a problem?)

class CsvWriter(AbstractWriter):

    def __init__(self, filename):    
        self.filename = filename
        self.file = self.openFile()
        self.buffer = BufferedWriter(self, settings.WRITE_THRESHOLD)

When I know want to write something I call write_buffered buffered, which looks like that:

def write_buffered(self, data_dict):    
        self.buffer.write(b';'.join(map(str, data_dict.values())) + '\n')

The actual write methods which (as I figured) needs to be implemented on the CsvWriter class itself looks like that:

def write(self, data):
        if self.file.closed:
            self.file = self.openFile()

        return self.file.write(data)

The problem is, that when I try to run the simulator, I get the following error:

IOError: raw write() returned invalid length -1 (should have been between 0 and 78)

Do any of you have a clue for me how to fix this?

Answer