Accessing items from a dictionary using pickle efficiently in Python

Posted by user248237 on Stack Overflow See other posts from Stack Overflow or by user248237
Published on 2010-12-22T19:44:36Z Indexed on 2010/12/22 19:54 UTC
Read the original article Hit count: 192

Filed under:
|
|

I have a large dictionary mapping keys (which are strings) to objects. I pickled this large dictionary and at certain times I want to pull out only a handful of entries from it. The dictionary has usually thousands of entries total. When I load the dictionary using pickle, as follows:

from cPickle import *
# my dictionary from pickle, containing thousands of entries
mydict = open(load('mypickle.pickle'))
# accessing only handful of entries here
for entry in relevant_entries:
  # find relevant entry
  value = mydict[entry]

I notice that it can take up to 3-4 seconds to load the entire pickle, which I don't need, since I access only a tiny subset of the dictionary entries later on (shown above.)

How can I make it so pickle only loads those entries that I have from the dictionary, to make this faster?

Thanks.

© Stack Overflow or respective owner

Related posts about python

Related posts about pickle