How to use Elasticsearch with MongoDB?

Question

I have gone through many blogs and sites about configuring Elasticsearch for MongoDB to index Collections in MongoDB but none of them were straightforward.

Please explain to me a step by step process for installing elasticsearch, which should include:

• configuration
• run in the browser

I am using Node.js with express.js, so please help accordingly.

Answer 1

This answer should be enough to get you set up to follow this tutorial on Building a functional search component with MongoDB, Elasticsearch, and AngularJS.

If you're looking to use faceted search with data from an API then Matthiasn's BirdWatch Repo is something you might want to look at.

So here's how you can setup a single node Elasticsearch "cluster" to index MongoDB for use in a NodeJS, Express app on a fresh EC2 Ubuntu 14.04 instance.

Make sure everything is up to date.

sudo apt-get update

Install NodeJS.

sudo apt-get install nodejs
sudo apt-get install npm

Install MongoDB - These steps are straight from MongoDB docs. Choose whatever version you're comfortable with. I'm sticking with v2.4.9 because it seems to be the most recent version MongoDB-River supports without issues.

Import the MongoDB public GPG Key.

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10

Update your sources list.

echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list

Get the 10gen package.

sudo apt-get install mongodb-10gen

Then pick your version if you don't want the most recent. If you are setting your environment up on a windows 7 or 8 machine stay away from v2.6 until they work some bugs out with running it as a service.

apt-get install mongodb-10gen=2.4.9

Prevent the version of your MongoDB installation being bumped up when you update.

echo "mongodb-10gen hold" | sudo dpkg --set-selections

Start the MongoDB service.

sudo service mongodb start

Your database files default to /var/lib/mongo and your log files to /var/log/mongo.

Create a database through the mongo shell and push some dummy data into it.

mongo YOUR_DATABASE_NAME
db.createCollection(YOUR_COLLECTION_NAME)
for (var i = 1; i <= 25; i++) db.YOUR_COLLECTION_NAME.insert( { x : i } )

Now to Convert the standalone MongoDB into a Replica Set.

First Shutdown the process.

mongo YOUR_DATABASE_NAME
use admin
db.shutdownServer()

Now we're running MongoDB as a service, so we don't pass in the "--replSet rs0" option in the command line argument when we restart the mongod process. Instead, we put it in the mongod.conf file.

vi /etc/mongod.conf

Add these lines, subbing for your db and log paths.

replSet=rs0
dbpath=YOUR_PATH_TO_DATA/DB
logpath=YOUR_PATH_TO_LOG/MONGO.LOG

Now open up the mongo shell again to initialize the replica set.

mongo DATABASE_NAME
config = { "_id" : "rs0", "members" : [ { "_id" : 0, "host" : "127.0.0.1:27017" } ] }
rs.initiate(config)
rs.slaveOk() // allows read operations to run on secondary members.

Now install Elasticsearch. I'm just following this helpful Gist.

Make sure Java is installed.

sudo apt-get install openjdk-7-jre-headless -y

Stick with v1.1.x for now until the Mongo-River plugin bug gets fixed in v1.2.1.

wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.1.1.deb
sudo dpkg -i elasticsearch-1.1.1.deb

curl -L http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
sudo mv *servicewrapper*/service /usr/local/share/elasticsearch/bin/
sudo rm -Rf *servicewrapper*
sudo /usr/local/share/elasticsearch/bin/service/elasticsearch install
sudo ln -s `readlink -f /usr/local/share/elasticsearch/bin/service/elasticsearch` /usr/local/bin/rcelasticsearch

Make sure /etc/elasticsearch/elasticsearch.yml has the following config options enabled if you're only developing on a single node for now:

cluster.name: "MY_CLUSTER_NAME"
node.local: true

Start the Elasticsearch service.

sudo service elasticsearch start

Verify it's working.

curl http://localhost:9200

If you see something like this then you're good.

{
  "status" : 200,
  "name" : "Chi Demon",
  "version" : {
    "number" : "1.1.2",
    "build_hash" : "e511f7b28b77c4d99175905fac65bffbf4c80cf7",
    "build_timestamp" : "2014-05-22T12:27:39Z",
    "build_snapshot" : false,
    "lucene_version" : "4.7"
  },
  "tagline" : "You Know, for Search"
}

Now install the Elasticsearch plugins so it can play with MongoDB.

bin/plugin --install com.github.richardwilly98.elasticsearch/elasticsearch-river-mongodb/1.6.0
bin/plugin --install elasticsearch/elasticsearch-mapper-attachments/1.6.0

These two plugins aren't necessary but they're good for testing queries and visualizing changes to your indexes.

bin/plugin --install mobz/elasticsearch-head
bin/plugin --install lukas-vlcek/bigdesk

Restart Elasticsearch.

sudo service elasticsearch restart

Finally index a collection from MongoDB.

curl -XPUT localhost:9200/_river/DATABASE_NAME/_meta -d '{
  "type": "mongodb",
  "mongodb": {
    "servers": [
      { "host": "127.0.0.1", "port": 27017 }
    ],
    "db": "DATABASE_NAME",
    "collection": "ACTUAL_COLLECTION_NAME",
    "options": { "secondary_read_preference": true },
    "gridfs": false
  },
  "index": {
    "name": "ARBITRARY INDEX NAME",
    "type": "ARBITRARY TYPE NAME"
  }
}'

Check that your index is in Elasticsearch

curl -XGET http://localhost:9200/_aliases

Check your cluster health.

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

It's probably yellow with some unassigned shards. We have to tell Elasticsearch what we want to work with.

curl -XPUT 'localhost:9200/_settings' -d '{ "index" : { "number_of_replicas" : 0 } }'

Check cluster health again. It should be green now.

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'

Go play.