How to reindex ElasticSearch quickly?

elasticsearch pyes elasticsearch-plugin
diemacht picture diemacht · Jun 9, 2013 · Viewed 13.6k times · Source

I have an ElasticSearch index with around 200M documents, total index size of 90Gb.

I changed mapping, so I would like ElasticSearch to re-index all the documents.

I wrote a script that creates a new index (with the new mapping), then goes over all the documents in the old index and puts then into the new one.

It seems to work, but the problem is that it works extremely slowly. It started with 300 documents / minute two days ago, and now the speed is 150 documents/minute.

The script runs on a machine within the same network the elastic search machines in.

With such speed it will require a month for the re-index to finish.

Does anybody know about some faster technique to re-index an elastic search index?

Answer

diemacht picture diemacht · Jun 11, 2013

Answered in the google groups:

Option A: Use bulk index operations.

Option B: Use the re-index plug-in that runs inside ES machine: https://github.com/karussell/elasticsearch-reindex

Related questions

Elasticsearch python API: Delete documents by query

I see that the following API will do delete by query in Elasticsearch - http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-delete-by-query.html But I want to do the same with the elastic search bulk API, even though …

Read More
Querying Multi Level Nested fields on Elastic Search

I'm new to Elastic Search and to the non-SQL paradigm. I've been following ES tutorial, but there is one thing I couldn't put to work. In the following code (I'me using PyES to interact with ES) I create a single …

Read More
Elasticsearch clients for python, no solution

I am having a very bad week having chosen elasticsearch with graylog2. I am trying to run queries against the data in ES using Python. I have tried following clients. ESClient - Very weird results, I think its not maintained, …

Read More