Elastic Search HTTPConnectionPool(host='127.0.0.1', port=9200): Max retries exceeded

Question

I have an Elastic Search server which I am querying both from a live website and through a Django management command. The management command runs using celery beat at 3am GMT to synchronise data from an outside service. Sometimes (but not every time) this command is run elastic search appears to crash and I get the following error in my error log.

    [09/Jan/2014 08:03:46] ERROR [django.request:212] Internal Server Error: /
    Traceback (most recent call last):
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/django/core/handlers/base.py",
line 115, in get_response
        response = callback(request, *callback_args, **callback_kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/django/views/generic/base.py",
line 68, in view
        return self.dispatch(request, *args, **kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/django/views/generic/base.py",
line 86, in dispatch
        return handler(request, *args, **kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/django/views/generic/base.py",
line 153, in get
        context = self.get_context_data(**kwargs)
      File
"/srv/www/site.co.uk/clothes_comparison/clothes_comparison/views.py",
line 56, in get_context_data
        fields=['id', 'name', 'price', 'images', 'advertiser']
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py",
line 96, in decorate
        return func(*args, query_params=query_params, **kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py",
line 512, in multi_get
        'GET', ['_mget'], {'docs': docs}, query_params=query_params)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/pyelasticsearch/client.py",
line 238, in send_request
        **({'data': request_body} if body else {}))
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/requests/sessions.py",
line 347, in get
        return self.request('GET', url, **kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/requests/sessions.py",
line 335, in request
        resp = self.send(prep, **send_kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/requests/sessions.py",
line 438, in send
        r = adapter.send(request, **kwargs)
      File
"/srv/www/site.co.uk/env/local/lib/python2.7/site-packages/requests/adapters.py",
line 327, in send
        raise ConnectionError(e)
    ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=9200): Max
retries exceeded with url: /_mget (Caused by <class 'socket.error'>: [Errno 111]
Connection refused)

I am using pyelasticsearch to connect to Elastic Search with the following code in my settings.py file:

try:
    ES_CON
except NameError:
    ES_CON = None

if not ES_CON:
    ES_CON = ElasticSearch(ELASTICSEARCH_URI)

Any help would be greatly appreciated.

Answer 1

I would suggest using the official Elasticsearch python client: elasticsearch-py which has reliable connection handling, is thread safe etc. It is also faster (according to the author, who is part of the Elasticsearch team).

You can then have your es = Elasticsearch() either at the top of your tasks.py, or in e.g. core.helpers and import es from there.