replica set config is invalid or we are not a member of it, running in kubernetes

Question

I have posted this problem from here

I am running sharded mongodb in a kubernetes environment, with 3 shards, and 3 instances on each shard. for some reasons, my mongodb instance have been rescheduled to another machine.

the problem is when a mongodb instance have ben rescheduled to another instance, its replica config will be invalidated. resulting to this error below.

            > rs.status()
            {
                "state" : 10,
                "stateStr" : "REMOVED",
                "uptime" : 2110,
                "optime" : Timestamp(1448462710, 6),
                "optimeDate" : ISODate("2015-11-25T14:45:10Z"),
                "ok" : 0,
                "errmsg" : "Our replica set config is invalid or we are not a member of it",
                "code" : 93
            }
            >

this is the config

            > rs.config().members
            [
                {
                    "_id" : 0,
                    "host" : "mongodb-shard2-service:27038",
                    "arbiterOnly" : false,
                    "buildIndexes" : true,
                    "hidden" : false,
                    "priority" : 1,
                    "tags" : {

                    },
                    "slaveDelay" : 0,
                    "votes" : 1
                },
                {
                    "_id" : 1,
                    "host" : "shard2-slave2-service:27039",
                    "arbiterOnly" : false,
                    "buildIndexes" : true,
                    "hidden" : false,
                    "priority" : 1,
                    "tags" : {

                    },
                    "slaveDelay" : 0,
                    "votes" : 1
                },
                {
                    "_id" : 2,
                    "host" : "shard2-slave1-service:27033",
                    "arbiterOnly" : false,
                    "buildIndexes" : true,
                    "hidden" : false,
                    "priority" : 1,
                    "tags" : {

                    },
                    "slaveDelay" : 0,
                    "votes" : 1
                }
            ]

and a sample of db.serverStatus() of a rescheduled mongodb instance

            > db.serverStatus()
            {
                "host" : "mongodb-shard2-master-ofgrb",
                "version" : "3.0.7",
                "process" : "mongod",
                "pid" : NumberLong(8),

I hope I am making sense.. because, I will be using this in live production very soon.. thank you!!

Answer 1

Sorry for the delay. Here is a post that describes in detail how to bring up mongodb: https://medium.com/google-cloud/mongodb-replica-sets-with-kubernetes-d96606bd9474#.x197hr2ps

One Service and one single-replica ReplicationController per instance is the current approach for a stateful application where each instance needs a stable, predictable identity. With this approach, it is also straightforward to allocate a PersistentVolume for each pod.

Other solutions are possible, such as the sidecar approach shown in that example and the custom seed provider in the Cassandra example, but are a little more complex.

Kubernetes 1.2 will provide a means of setting the hostname (as seen by the container) for each pod. Kubernetes 1.3 will add a new controller to create the instances.