more efficient way to pickle a string

python numpy pickle space-efficiency
gatoatigrado picture gatoatigrado · Mar 30, 2009 · Viewed 9.8k times · Source

The pickle module seems to use string escape characters when pickling; this becomes inefficient e.g. on numpy arrays. Consider the following

z = numpy.zeros(1000, numpy.uint8)
len(z.dumps())
len(cPickle.dumps(z.dumps()))

The lengths are 1133 characters and 4249 characters respectively.

z.dumps() reveals something like "\x00\x00" (actual zeros in string), but pickle seems to be using the string's repr() function, yielding "'\x00\x00'" (zeros being ascii zeros).

i.e. ("0" in z.dumps() == False) and ("0" in cPickle.dumps(z.dumps()) == True)

Answer

Benjamin Peterson picture Benjamin Peterson · Mar 30, 2009

Try using a later version of the pickle protocol with the protocol parameter to pickle.dumps(). The default is 0 and is an ASCII text format. Ones greater than 1 (I suggest you use pickle.HIGHEST_PROTOCOL). Protocol formats 1 and 2 (and 3 but that's for py3k) are binary and should be more space conservative.

Related questions

best way to preserve numpy arrays on disk

I am looking for a fast way to preserve large numpy arrays. I want to save them to the disk in a binary format, then read them back into memory relatively fastly. cPickle is not fast enough, unfortunately. I found …

Read More
Convert numpy array type and values from Float64 to Float32

I am trying to convert threshold array(pickle file of isolation forest from scikit learn) of type from Float64 to Float32 for i in range(len(tree.tree_.threshold)): tree.tree_.threshold[i] = tree.tree_.threshold[i].astype(np.float32) ​ …

Read More
Python pickle protocol choice?

I an using python 2.7 and trying to pickle an object. I am wondering what the real difference is between the pickle protocols. import numpy as np import pickle class Data(object): def __init__(self): self.a = np.zeros((100, 37000, 3), dtype=np.…

Read More