How reliable is ElasticSearch as a primary datastore against factors like write loss, data availability
I am working on a project with a requirement of coming up with a generic dashboard where a users can do different kinds of grouping, filtering and drill down on different fields. For this we are looking for a search store that allows slice and dice of data.
There would be multiple sources of data and would be storing it in the Search Store. There may be some pre-computation required on the source data which can be done by an intermediate components.
I have looked through several blogs to understand whether ES can be used reliably as a primary datastore too. It mostly depends on the use-case we are looking for. Some of the information about the use case that we have :
- Around 300 million record each year with 1-2 KB.
- Assuming storing 1 year data, we are today with 300 GB but use-case can go up to 400-500 GB given growth of data.
- As of now not sure, how we will push data, but roughly, it can go up to ~2-3 million records per 5 minutes.
- Search request are low, but requires complex queries which can search data for last 6 weeks to 6 months.
- document will be indexed across almost all the fields in document.
Some blogs say that it is reliable enough to use as a primary data store -
- http://chrisberkhout.com/blog/elasticsearch-as-a-primary-data-store/
- http://highscalability.com/blog/2014/1/6/how-hipchat-stores-and-indexes-billions-of-messages-using-el.html
- https://karussell.wordpress.com/2011/07/13/jetslide-uses-elasticsearch-as-database/
And some blogs say that ES have few limitations -
- https://www.found.no/foundation/elasticsearch-as-nosql/
- https://www.found.no/foundation/crash-elasticsearch/
- http://www.quora.com/Why-should-I-NOT-use-ElasticSearch-as-my-primary-datastore
Has anyone used Elastic Search as the sole truth of data without having a primary storage like PostgreSQL, DynamoDB or RDS? I have looked up that ES has certain issues like split brains and index corruption where there can be a problem with the data loss. So, I am looking to know if anyone has used ES and have got into any troubles with the data
Thanks.
Answer
Short answer: it depends on your use case, but you probably don't want to use it as a primary store.
Longer answer: You should really understand all of the possible issues that can come up around resiliency and data loss. Elastic has some great documentation of these issues which you should really understand before using it as a primary data store. In addition Aphyr's post on the topic is a good resource.
If you understand the risks you are taking and you believe that those risks are acceptable (e.g. because small data loss is not a problem for your application) then you should feel free to go ahead and try it.