in-memory database in Python

python sql database in-memory-database
Claudiu picture Claudiu · Jun 15, 2010 · Viewed 32.7k times · Source

I'm doing some queries in Python on a large database to get some stats out of the database. I want these stats to be in-memory so other programs can use them without going to a database.

I was thinking of how to structure them, and after trying to set up some complicated nested dictionaries, I realized that a good representation would be an SQL table. I don't want to store the data back into the persistent database, though. Are there any in-memory implementations of an SQL database that supports querying the data with SQL syntax?

Answer

Tim Post picture Tim Post · Jun 15, 2010

SQLite3 might work. The Python interface does support the in-memory implementation that the SQLite3 C API offers.

From the spec:

You can also supply the special name :memory: to create a database in RAM.

It's also relatively cheap with transactions, depending on what you are doing. To get going, just:

import sqlite3
conn = sqlite3.connect(':memory:')

You can then proceed like you were using a regular database.

Depending on your data - if you can get by with key/value (strings, hashes, lists, sets, sorted sets, etc) - Redis might be another option to explore (as you mentioned that you wanted to share with other programs).

Related questions

Python: Number of rows affected by cursor.execute("SELECT ...)

How can I access the number of rows affected by: cursor.execute("SELECT COUNT(*) from result where server_state='2' AND name LIKE '"+digest+"_"+charset+"_%'")

Read More
sqlalchemy filter multiple columns

How do I combine two columns and apply filter? For example, I want to search in both the "firstname" and "lastname" columns at the same time. Here is how I have been doing it if searching only one column: query = …

Read More
SQLAlchemy: how to filter date field?

Here is model: class User(Base): ... birthday = Column(Date, index=True) #in database it's like '1987-01-17' ... I want to filter between two dates, for example to choose all users in interval 18-30 years. How to implement it …

Read More