Are there any tools to estimate index size in MongoDB?

mongodb indexing b-tree mongodb-indexes
jpredham picture jpredham · Dec 22, 2011 · Viewed 7.5k times · Source

I'm looking for a tool to get a decent estimate of how large a MongoDB index will be based on a few signals like:

  • How many documents in my collection
  • The size of the indexed field(s)
  • The size of the _id I'm using if not ObjectId
  • Geo/Non-geo

Has anyone stumbled across something like this? I can imagine it would be extremely useful given Mongo's performance degradation once it hits the memory wall and documents start getting paged out to disk. If I have a functioning database and want to add another index, the only way I'll know if it will be too big is to actually add it.

It wouldn't need to be accurate down to the bit, but with some assumptions about B-Trees and the index implementation I'm sure it could be reasonable enough to be helpful.

If this doesn't exist already I'd like to build and open source it, so if I've missed any required parameters for this calculation please include in your answer.

Answer

Tyler Brock picture Tyler Brock · Dec 22, 2011

I just spoke with some of the 10gen engineers and there isn't a tool but you can do a back of the envelope calculation that is based on this formula:

2 * [ n * ( 18 bytes overhead + avg size of indexed field + 5 or so bytes of conversion fudge factor ) ]

Where n is the number of documents you have.

The overhead and conversion padding are mongo specific but the 2x comes from the b-tree data structure being roughly half full (but having allocated 100% of the space a full tree would require) in the worst case.

I'd explain more but I'm learning about it myself at the moment. This presentation will have more details: http://www.10gen.com/presentations/mongosp-2011/mongodb-internals

Related questions

Can I do a MongoDB "starts with" query on an indexed subdocument field?

I'm trying to find documents where a field starts with a value. Table scans are disabled using notablescan. This works: db.articles.find({"url" : { $regex : /^http/ }}) This doesn't: db.articles.find({"source.homeUrl" : { $regex : /^http/ }}) I get the error: error: { "$…

Read More
MongoDB querying performance for over 5 million records

We've recently hit the >2 Million records for one of our main collections and now we started to suffer for major performance issues on that collection. They documents in the collection have about 8 fields which you can filter by using …

Read More
How to create indexes in MongoDB via .NET

I've programmatically created a new document collection using the MongoDB C# driver. At this point I want to create and build indexes programmatically. How can I do that?

Read More