*large* python dictionary with persistence storage for quick look-ups

python persistence object-persistence
Abhi picture Abhi · Aug 7, 2012 · Viewed 9.7k times · Source

I have a 400 million lines of unique key-value info that I would like to be available for quick look ups in a script. I am wondering what would be a slick way of doing this. I did consider the following but not sure if there is a way to disk map the dictionary and without using a lot of memory except during dictionary creation.

  1. pickled dictionary object : not sure if this is an optimum solution for my problem
  2. NoSQL type dbases : ideally want something which has minimum dependency on third party stuff plus the key-value are simply numbers. If you feel this is still the best option, I would like to hear that too. May be it will convince me.

Please let me know if anything is not clear.

Thanks! -Abhi

Answer

Sam Mussmann picture Sam Mussmann · Aug 7, 2012

If you want to persist a large dictionary, you are basically looking at a database.

Python comes with built in support for sqlite3, which gives you an easy database solution backed by a file on disk.

Related questions

How to store a dictionary on a Django Model?

I need to store some data in a Django model. These data are not equal to all instances of the model. At first I thought about subclassing the model, but I’m trying to keep the application flexible. If I …

Read More
Why can't environmental variables set in python persist?

I was hoping to write a python script to create some appropriate environmental variables by running the script in whatever directory I'll be executing some simulation code, and I've read that I can't write a script to make these env …

Read More
Easiest way to persist a data structure to a file in python?

Let's say I have something like this: d = { "abc" : [1, 2, 3], "qwerty" : [4,5,6] } What's the easiest way to progammatically get that into a file that I can load from python later? Can I somehow save it as python source (from within a python …

Read More