Fastest, non-memory-based, multi-process key-value store for Node.js
What is the fastest non-memory key-value store for Node.js supporting multiple processes?
I need to store simple key-value string/string pairs (not documents or JSON, just strings).
Here are some examples (there would be millions of those):
- 12345678 – abcdefghijklmnopabcdefghijklmnop
- 86358098 – ahijklmnopbcdefgahijklmnopbcdefg
- abcdefghijklmnopabcdefghijklmnop - 12345678
- ahijklmnopbcdefgahijklmnopbcdefg - 86358098
I have tried:
- Redis: it's really fast and does everything I need, but consumes too much RAM.
- LevelDB: it's fast and not too heavy on RAM, but only single-process.
A workaround for LevelDB is multilevel, which exposes a single LevelDB process though HTTP.
But that of course comes at a cost; I need something fast.
Is there any key-value store that:
- supports Node.js or has bindings for it;
- stores string/string pairs;
- supports multiple processes;
- does not entirely reside in memory;
- is fast?
I only care about reading. Fast multi-process reading is necessary, but not writing.
I'm happy with the current speed of LevelDB, just not with the fact that it is single-process.
Additional details:
- I'm talking about some 50 million key/value pairs, with keys and values between 8 and 500 chars.
- The code will run on a regular Linux server.
- Memory usage should be limited to a few gigabytes (4GB is fine, 8GB is acceptable)
- Reading will happen way more than writing; actually, I could do without writing.
- Speed is more important than anything (given memory and multi-process constraint are respected).
Answer
I would suggest to have a look at LMDB (which is the most efficient engine for OpenLDAP, and used in a number of other open-source projects).
LMDB is an embedded key/value store, with a Berkeley-DB or LevelDB like API, does not have to store everything in memory, and can support access from multiple processes. There are Node.js bindings: