What are some use cases for using Elasticsearch versus standard sql queries?
I'm just getting started with Elasticsearch and one of the main use cases I've seen is its scalability with searches on large data sets, but besides this when would you want to use it over just creating sql queries with a traditional RDBMS?
Answer
There are two primary Elasticsearch use cases:
- Text search
You want Elasticsearch when you're doing a lot of text search, where traditional RDBMS databases are not performing really well (poor configuration, acts as a black-box, poor performance). Elasticsearch is highly customizable, extendable through plugins. You can build robust search without much knowledge quite fast.
- Logging and analysis
Another edge case is that a lot people use Elasticsearch to store logs from various sources (to centralize them), so they can analyze them and make sense out of it. In this case, Kibana becomes handy. It lets you connect to Elasticsearch cluster and create visualisations straight away. For instance, Loggly is built using Elasticsearch and Kibana.
Keep in mind, that you wouldn't want to use Elasticsearch as your primary data storage. Reasons here: How reliable is ElasticSearch as a primary datastore against factors like write loss, data availability
Update
I felt like the second part is no longer edgy, it's actually what Elastic as a company has been doing really well in past year. With current DevOps movement, CI/CD pipelines, increasing amount of metrics from various sources, ELK became a defacto choice for infrastructure monitoring, it's no longer just a distributed RESTful text-search engine. It has an amazing set of products:
- Logstash (tons of data inputs)
- Beats
- Filebeat
- Metricbeat
- Packetbeat
- Winlogbeat
- Kibana
- Graph
- Timelion
- X-Pack (premium)
- Alerts
- Reporting
- Security
- Machine Learning
- Cross data center metrics
An ecosystem, built by community, is growing around ELK stack that expands current features, few of them worth mentioning:
- ElastAlert
- Search Guard