pickle - putting more than 1 object in a file?

python file pickle
newnewbie picture newnewbie · Mar 17, 2013 · Viewed 48.7k times · Source

I have got a method which dumps a number of pickled objects (tuples, actually) into a file.

I do not want to put them into one list, I really want to dump several times into the same file. My problem is, how do I load the objects again? The first and second object are just one line long, so this works with readlines. But all the others are longer. naturally, if I try 

myob = cpickle.load(g1.readlines()[2])

where g1 is the file, I get an EOF error because my pickled object is longer than one line. Is there a way to get just my pickled object?

Answer

Martin Atkins picture Martin Atkins · Mar 17, 2013

If you pass the filehandle directly into pickle you can get the result you want.

import pickle

# write a file
f = open("example", "w")
pickle.dump(["hello", "world"], f)
pickle.dump([2, 3], f)
f.close()

f = open("example", "r")
value1 = pickle.load(f)
value2 = pickle.load(f)
f.close()

pickle.dump will append to the end of the file, so you can call it multiple times to write multiple values.

pickle.load will read only enough from the file to get the first value, leaving the filehandle open and pointed at the start of the next object in the file. The second call will then read the second object, and leave the file pointer at the end of the file. A third call will fail with an EOFError as you'd expect.

Although I used plain old pickle in my example, this technique works just the same with cPickle.

Related questions

Why do I get "Pickle - EOFError: Ran out of input" reading an empty file?

I am getting an interesting error while trying to use Unpickler.load(), here is the source code: open(target, 'a').close() scores = {}; with open(target, "rb") as file: unpickler = pickle.Unpickler(file); scores = unpickler.load(); if not isinstance(scores, dict): …

Read More
How to save a list to a file and read it as a list type?

Say I have the list score=[1,2,3,4,5] and it gets changed whilst my program is running. How could I save it to a file so that next time the program is run I can access the changed list as a list …

Read More
What is the difference between rb and r+b modes in file objects

I am using pickle module in Python and trying different file IO modes: # works on windows.. "rb" with open(pickle_f, 'rb') as fhand: obj = pickle.load(fhand) # works on linux.. "r" with open(pickle_f, 'r') as fhand: obj = …

Read More