Why is PyMongo count_documents is slower than count?

mongodb mongodb-query pymongo pymongo-3.x
Threegirl picture Threegirl · Sep 8, 2018 · Viewed 10.5k times · Source

In db['TF'] I have about 60 million records.

I need to get the quantity of the records.

If I run db['TF'].count(), it returns at once.

If I run db['TF'].count_documents({}), that is a such long time before I get the result.

However, the count method will be deprecated.

So, how can I get the quantity quickly when using count_documents? Is there some arguments I missed?

I have read the doc and code, but nothing found.

Thanks a lot!

Answer

Amit Wagner picture Amit Wagner · Sep 8, 2018

This is not about PyMongo but Mongo itself.

count is a native Mongo function. It doesn't really count all the documents. Whenever you insert or delete a record in Mongo, it caches the total number of records in the collection. Then when you run count, Mongo will return that cached value.

count_documents uses a query object, which means that it has to loop through all the records in order to get the total count. Because you're not passing any parameters, it will have to run over all 60 million records. This is why it is slow.

based on @Stennie comment

You can use estimated_document_count() in PyMongo 3.7+ to return the fast count based on collection metadata. The original count() was deprecated because the behaviour differed (estimated vs actual count) based on whether query criteria was provided. The newer driver API is more intentional about the outcome

Related questions

Query Mongodb on month, day, year... of a datetime

I'm using mongodb and I store datetime in my database in this way for a date "17-11-2011 18:00" I store: date = datetime.datetime(2011, 11, 17, 18, 0) db.mydatabase.mycollection.insert({"date" : date}) I would like to do a request like that month = 11 db.…

Read More
How to convert a pymongo.cursor.Cursor into a dict?

I am using pymongo to query for all items in a region (actually it is to query for all venues in a region on a map). I used db.command(SON()) before to search in a spherical region, which can …

Read More
How can I use 'Not Like' operator in MongoDB

I can use the SQL Like Operator using pymongo, db.test.find({'c':{'$regex':'ttt'}}) But how can I use Not Like Operator? I tried db.test.find({'c':{'$not':{'$regex':'ttt'}}) but got error: OperationFailure: $not …

Read More