Writing a BytesIO object to a file, 'efficiently'

python io bytesio
Kalabaaz picture Kalabaaz · Aug 20, 2016 · Viewed 14.9k times · Source

So a quick way to write a BytesIO object to a file would be to just use:

with open('myfile.ext', 'wb') as f:
    f.write(myBytesIOObj.getvalue())
myBytesIOObj.close()

However, if I wanted to iterate over the myBytesIOObj as opposed to writing it in one chunk, how would I go about it? I'm on Python 2.7.1. Also, if the BytesIO is huge, would it be a more efficient way of writing by iteration?

Thanks

Answer

tdelaney picture tdelaney · Aug 20, 2016

shutil has a utility that will write the file efficiently. It copies in chunks, defaulting to 16K. Any multiple of 4K chunks should be a good cross platform number. I chose 131072 rather arbitrarily because really the file is written to the OS cache in RAM before going to disk and the chunk size isn't that big of a deal.

import shutil

myBytesIOObj.seek(0)
with open('myfile.ext', 'wb') as f:
    shutil.copyfileobj(myBytesIOObj, f, length=131072)

BTW, there was no need to close the file object at the end. with defines a scope, and the file object is defined inside that scope. The file handle is therefore closed automatically on exit from the with block.

Related questions

Unicode (UTF-8) reading and writing to files in Python

I'm having some brain failure in understanding reading and writing text to a file (Python 2.4). # The string, which has an a-acute in it. ss = u'Capit\xe1n' ss8 = ss.encode('utf8') repr(ss), repr(ss8) ("u'Capit\xe1n'", "'Capit\…

Read More
Load data from txt with pandas

I am loading a txt file containig a mix of float and string data. I want to store them in an array where I can access each element. Now I am just doing import pandas as pd data = pd.read_…

Read More
StringIO in Python3

I am using Python 3.2.1 and I can't import the StringIO module. I use io.StringIO and it works, but I can't use it with numpy's genfromtxt like this: x="1 3\n 4.5 8" numpy.genfromtxt(io.StringIO(x)) I get the following error: …

Read More