Bulk insert performance in MongoDB for large collections

performance mongodb insert mongodb-java
mich8bsp picture mich8bsp · Jun 9, 2015 · Viewed 7.3k times · Source

I'm using the BulkWriteOperation (java driver) to store data in large chunks. At first it seems to be working fine, but when the collection grows in size, the inserts can take quite a lot of time.

Currently for a collection of 20M documents, bulk insert of 1000 documents could take about 10 seconds.

Is there a way to make inserts independent of collection size? I don't have any updates or upserts, it's always new data I'm inserting.

Judging from the log, there doesn't seem to be any issue with locks. Each document has a time field which is indexed, but it's linearly growing so I don't see any need for mongo to take the time to reorganize the indexes.

I'd love to hear some ideas for improving the performance

Thanks

Answer

glytching picture glytching · Jun 23, 2017

You believe that the indexing does not require any document reorganisation and the way you described the index suggests that a right handed index is ok. So, indexing seems to be ruled out as an issue. You could of course - as suggested above - definitively rule this out by dropping the index and re running your bulk writes.

Aside from indexing, I would …

  • Consider whether your disk can keep up with the volume of data you are persisting. More details on this in the Mongo docs
  • Use profiling to understand what’s happening with your writes

Related questions

MySQL vs MongoDB 1000 reads

I have been very excited about MongoDb and have been testing it lately. I had a table called posts in MySQL with about 20 million records indexed only on a field called 'id'. I wanted to compare speed with MongoDB and …

Read More
When to use CouchDB over MongoDB and vice versa

I am stuck between these two NoSQL databases. In my project I will be creating a database within a database. For example, I need a solution to create dynamic tables. So users can create tables with columns and rows. I …

Read More
Fastest way to remove duplicate documents in mongodb

I have approximately 1.7M documents in mongodb (in future 10m+). Some of them represent duplicate entry which I do not want. Structure of document is something like this: { _id: 14124412, nodes: [ 12345, 54321 ], name: "Some beauty" } Document is duplicate if it has at …

Read More