_pickle in python3 doesn't work for large data saving

python pickle
Jake0x32 picture Jake0x32 · Apr 17, 2015 · Viewed 29.8k times · Source

I am trying to apply _pickle to save data onto disk. But when calling _pickle.dump, I got an error

OverflowError: cannot serialize a bytes object larger than 4 GiB

Is this a hard limitation to use _pickle? (cPickle for python2)

Answer

Eric Levieil picture Eric Levieil · Apr 17, 2015

Not anymore in Python 3.4 which has PEP 3154 and Pickle 4.0
https://www.python.org/dev/peps/pep-3154/

But you need to say you want to use version 4 of the protocol:
https://docs.python.org/3/library/pickle.html

pickle.dump(d, open("file", 'w'), protocol=4)

Related questions

How can I use pickle to save a dict?

I have looked through the information that the Python docs give, but I'm still a little confused. Could somebody post sample code that would write a new file then use pickle to dump a dictionary into it?

Read More
Saving and loading objects and using pickle

I´m trying to save and load objects using pickle module. First I declare my objects: >>> class Fruits:pass ... >>> banana = Fruits() >>> banana.color = 'yellow' >>> banana.value = 30 After that …

Read More
Serializing class instance to JSON

I am trying to create a JSON string representation of a class instance and having difficulty. Let's say the class is built like this: class testclass: value1 = "a" value2 = "b" A call to the json.dumps is made like this: …

Read More