What is the ideal bulk size formula in ElasticSearch?

elasticsearch elasticsearch-bulk-api
shyos picture shyos · Aug 28, 2013 · Viewed 31.4k times · Source

I believe there should be a formula to calculate bulk indexing size in ElasticSearch. Probably followings are the variables of such a formula.

  • Number of nodes
  • Number of shards/index
  • Document size
  • RAM
  • Disk write speed
  • LAN speed

I wonder If anyone know or use a mathematical formula. If not, how people decide their bulk size? By trial and error?

Answer

moliware picture moliware · Aug 28, 2013

There is no golden rule for this. Extracted from the doc:

There is no “correct” number of actions to perform in a single bulk call. You should experiment with different settings to find the optimum size for your particular workload.

Related questions

How to use Bulk API to store the keywords in ES by using Python

I have to store some message in ElasticSearch integrate with my python program. Now what I try to store the message is: d={"message":"this is message"} for index_nr in range(1,5): ElasticSearchAPI.addToIndex(index_nr, d) print d That …

Read More
How do I update multiple items in ElasticSearch?

Let's say I have a tag type in an ElasticSearch index, with the following mapping: { "tag": { "properties": { "tag": {"type": "string", "store": "yes"}, "aliases": {"type": "string"} } } } Each entry is a tag, and an array of aliases to that tag. Here is …

Read More
Validation Failed: 1: no requests added in bulk indexing ElasticSearch

I have a JSON file and I need to index it on ElasticSearch server. JSOIN file looks like this: { "sku": "1", "vbid": "1", "created": "Sun, 05 Oct 2014 03:35:58 +0000", "updated": "Sun, 06 Mar 2016 12:44:48 +0000", "type": "Single", "downloadable-duration": "perpetual", "online-duration": "365 days", "book-format": "ePub", "build-status": "In Inventory", "description": "On 7 …

Read More