Elasticsearch OutOfMemoryError Java heap space

Question

I am running an 8 cores, 32g RAM elasticsearch node with 5 shards, 400 million (small) documents.

Everything works great until I run an agg search, then shards start failing with:

java.lang.OutOfMemoryError: Java heap space

I have changed heap size with: export ES_HEAP_SIZE=16g (also ES_MAX_MEM and ES_MIN_MEM to same)

also changed the yml file for elasticsearch:

bootstrap.mlockall: true

and even (recommended by install documents):

sudo sysctl -w vm.max_map_count=262144

Restart service and still no no impact, still java.lang.OutOfMemoryError: Java heap space

Any other suggestions? other than don't run agg queries?

query is:

https://localhost:9200/my_index_name/_search?search_type=count
{
  "aggs": {
    "distinct_hostname": {
      "cardinality": {
        "field": "hostname"
      }
    }
  }
}

Answer 1

I think I have discovered the error. I was using 'service' to run elasticsearch and therefore my environment variables got stripped. I had to update the /etc/default/elasticsearch file with the correcct env variables (specifically the ES_HEAP_SIZE=16g).

So far it's running well and app is not erroring.