more efficient way to pickle a string

Posted by gatoatigrado on Stack Overflow See other posts from Stack Overflow or by gatoatigrado
Published on 2009-03-30T01:48:21Z Indexed on 2010/05/09 21:48 UTC
Read the original article Hit count: 235

Filed under:
|
|
|

The pickle module seems to use string escape characters when pickling; this becomes inefficient e.g. on numpy arrays. Consider the following

z = numpy.zeros(1000, numpy.uint8)
len(z.dumps())
len(cPickle.dumps(z.dumps()))

The lengths are 1133 characters and 4249 characters respectively.

z.dumps() reveals something like "\x00\x00" (actual zeros in string), but pickle seems to be using the string's repr() function, yielding "'\x00\x00'" (zeros being ascii zeros).

i.e. ("0" in z.dumps() == False) and ("0" in cPickle.dumps(z.dumps()) == True)

© Stack Overflow or respective owner

Related posts about python

Related posts about pickle