Simplest way to get data from AWS mysql RDS to AWS Elasticsearch?

amazon-web-services elasticsearch logstash amazon-elasticsearch
Sam Fen picture Sam Fen · Apr 28, 2017 · Viewed 11.9k times · Source

I have data in an AWS RDS, and I would like to pipe it over to an AWS ES instance, preferably updating once an hour, or similar.

On my local machine, with a local mysql database and Elasticsearch database, it was easy to set this up using Logstash.

Is there a "native" AWS way to do the same thing? Or do I need to set up an EC2 server and install Logstash on it myself?

Answer

Val picture Val · Apr 29, 2017

You can achieve the same thing with your local Logstash, simply point your jdbc input to your RDS database and the elasticsearch output to your AWS ES instance. If you need to run this regularly, then yes, you'd need to setup a small instance to run Logstash on it.

A more "native" AWS solution to achieve the same thing would include the use of Amazon Kinesis and AWS Lambda.

Here's a good article explaining how to connect it all together, namely:

  • how to stream RDS data into a Kinesis Stream
  • configuring a Lambda function to handle the stream
  • push the data to your AWS ES instance

Related questions

Elasticsearch tried to parse field [null] as object, but found a concrete value

How can I change mapping to resolve these error, new to elastic stack ` [logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.08", :_type=>"doc", :_routing=>nil}, #], :response=&…

Read More
Proper access policy for Amazon Elastic Search Cluster

I've recently started using the new Amazon Elasticsearch Service and I can't seem to figure out the access policy I need so that I can only access the services from my EC2 instances that have a specific IAM role assigned …

Read More
How to access Kibana from Amazon elasticsearch service?

I created Amazon elasticsearch service and populated data into it using logstash, which has been installed on an EC2 instance. On the Amazon elasticservice console page, there will be a link to access Kibana. search-cluster_name-XXXXXXXXXXXXXXXXXXX.region_name.es.amazonaws.…

Read More