Redis sentinel not connect to master: Error: READONLY You can't write against a read only slave?

Justin picture Justin · Mar 15, 2017 · Viewed 15.1k times · Source

I have this redis replication with sentinel architect:

5 servers: .1, .2, .3, .4 and .5

  • .1: nodejs app, redis master, redis sentinel
  • .2: nodejs app, redis slave
  • .3: nodejs app, redis slave
  • .4: redis sentinel
  • .5: redis sentinel

Sentinel handle failover as expected. redis-cli -h x.y.z.5 -p 26379 sentinel get-master-addr-by-name mymaster is always return .1 server.

My code to connect to this replication (I use ioredis):

let sentinels = [
  { host: process.env.REDIS_SENTINEL_1, port: process.env.REDIS_PORT },
  { host: process.env.REDIS_SENTINEL_2, port: process.env.REDIS_PORT },
  { host: process.env.REDIS_SENTINEL_3, port: process.env.REDIS_PORT }
]
store = new Redis({
  name: 'mymaster',
  sentinels: sentinels
})
store.on('ready', () => {
  store.set('foo', new Date())
  store.get('foo', function (err, result) {
    console.log('redis: get foo: ', result)
  });
});

Then, .1 nodejs app run ok, but in .2 and .3, the error occurs with this log:

 events.js:161
       throw er; // Unhandled 'error' event
       ^

 Error: READONLY You can't write against a read only slave.

P/s: my redis servers are binding in both server IP and 127.0.0.1. Because, without 127.0.0.1, I meet this error:

Error: Redis connection to 127.0.0.1:6379 failed - connect ECONNREFUSED 127.0.0.1:6379

So, I guess that, my app isn't connect to redis server via sentinel, but it connect directly with localhost.

Any help are appriciate!

Update: log when I turn on the DEBUG=ioredis:*:

2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis status[x.y.z.5:26379]: connecting -> connect
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis status[x.y.z.5:26379]: connect -> ready
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:connection send 1 commands in offline queue
2|MyApp     | Wed, 15 Mar 2017 12:58:42 GMT ioredis:redis write command[0] -> sentinel(get-master-addr-by-name,mymaster)
2|MyApp     | events.js:161
2|MyApp     |       throw er; // Unhandled 'error' event
2|MyApp     |       ^
2|MyApp     | Error: READONLY You can't write against a read only slave.
2|MyApp     |     at JavascriptReplyParser._parseResult (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:43:16)
2|MyApp     |     at JavascriptReplyParser.try_parsing (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:114:21)
2|MyApp     |     at JavascriptReplyParser.run (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:126:22)
2|MyApp     |     at JavascriptReplyParser.execute (/home/demo/demo_api/source/node_modules/redis/lib/parsers/javascript.js:107:10)
2|MyApp     |     at Socket.<anonymous> (/home/demo/demo_api/source/node_modules/redis/index.js:131:27)
2|MyApp     |     at emitOne (events.js:96:13)
2|MyApp     |     at Socket.emit (events.js:189:7)
2|MyApp     |     at readableAddChunk (_stream_readable.js:176:18)
2|MyApp     |     at Socket.Readable.push (_stream_readable.js:134:10)
2|MyApp     |     at TCP.onread (net.js:551:20)
2|MyApp     | [nodemon] app crashed - waiting for file changes before starting...
2|MyApp     | sentinel production
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[localhost:6379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[x.y.z.5:26379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster)
2|MyApp     | sentinel production
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[localhost:6379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis status[x.y.z.5:26379]: [empty] -> connecting
2|MyApp     | Wed, 15 Mar 2017 12:58:43 GMT ioredis:redis queue command[0] -> sentinel(get-master-addr-by-name,mymaster)

Update 2: one more thing, I run the app in my local machine with the production environment and everything work like charm!!!

Answer

Justin picture Justin · Mar 16, 2017

Yeah, finally found.

It's my mistake when an old code use the redis module and try to connect directly to localhost.

Replace it to ioredis instance and it work like charm :)