HA cluster - Pacemaker - OFFLINE nodes status
I'm using Pacemaker + Corosync in Centos7 Create Cluster using these commands:
pcs cluster auth pcmk01-cr pcmk02-cr -u hacluster -p passwd
pcs cluster setup --name my_cluster pcmk01-cr pcmk02-cr
[pcmk01]# pcs cluster start --all
[root@pcmk01 /]# pcs cluster cib clust_cfg
[pcmk01]# pcs -f clust_cfg property set stonith-enabled=false
[root@pcmk01 /]# pcs -f clust_cfg property set no-quorum-policy=ignore
[root@pcmk01 /]# pcs -f clust_cfg resource defaults resource-stickiness=200
When I check the status of cluster I see strange and diffrent behavior between nodes, it's looks like the nodes doesn't know each other.
pcs status on NODE1:
[root@rvpcmk01 ~]# pcs status
Cluster name: my_cluster
Stack: corosync
Current DC: rvpcmk01-cr (version 1.1.15-11.el7_3.2-e174ec8) - partition WITHOUT quorum
Last updated: Mon Feb 6 15:18:18 2017 Last change: Mon Feb 6 15:03:03 2017 by root via cibadmin on rvpcmk01-cr
2 nodes and 0 resources configured
Online: [ rvpcmk01-cr ]
OFFLINE: [ rvpcmk02-cr ]
No resources
PCSD Status:
rvpcmk01-cr: Online
rvpcmk02-cr: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
pcs status on NODE2:
[root@rvpcmk02 ~]# pcs status
Cluster name: RV_cluster
Stack: corosync
Current DC: rvpcmk02-cr (version 1.1.15-11.el7_3.2-e174ec8) - partition WITHOUT quorum
Last updated: Mon Feb 6 15:19:53 2017 Last change: Mon Feb 6 15:04:12 2017 by root via crm_attribute on rvpcmk02-cr
2 nodes and 0 resources configured
Online: [ rvpcmk02-cr ]
OFFLINE: [ rvpcmk01-cr ]
No resources
PCSD Status:
rvpcmk01-cr: Offline
rvpcmk02-cr: Online
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: active/enabled
I also run this command on both nodes:
NODE1:
[root@rvpcmk01 ~]# pcs status corosync
Membership information
----------------------
Nodeid Votes Name
1 1 rvpcmk01-cr (local)
NODE2:
[root@rvpcmk02 ~]# pcs status corosync
Membership information
----------------------
Nodeid Votes Name
2 1 rvpcmk02-cr (local)
As I know both nodes should be appear in this above status.
Can you please help & advise what i'm missing here? why nodes seems not know one on each other?
here is also my /etc/hosts files on both servers:
[root@rvpcmk01 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.17.235.109 rvpcmkvip
172.17.235.43 rvpcmk01
172.17.235.44 rvpcmk02
172.17.235.75 rvpcmk01-cr
172.17.235.106 rvpcmk02-cr
172.17.235.119 rvpcmk01-drbd
172.17.235.46 rvpcmk02-drbd
[root@rvpcmk02 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
172.17.235.109 rvpcmkvip
172.17.235.43 rvpcmk01
172.17.235.44 rvpcmk02
172.17.235.75 rvpcmk01-cr
172.17.235.106 rvpcmk02-cr
172.17.235.119 rvpcmk01-drbd
172.17.235.46 rvpcmk02-drbd
I check the authorization (that for sure was authorized when I start to configure my cluster and now I can see that there is a problem but I don't understand what it is and what is the root cause for it:
[root@rvpcmk02 drbd.d]# pcs cluster auth rvpcmk01-cr rvpcmk02-cr -u hacluster -p passwd --debug
Running: /usr/bin/ruby -I/usr/lib/pcsd/ /usr/lib/pcsd/pcsd-cli.rb auth
--Debug Input Start--
{"username": "hacluster", "local": false, "nodes": ["rvpcmk01-cr", "rvpcmk02-cr"], "password": "passwd", "force": false}
--Debug Input End--
Return Value: 0
--Debug Output Start--
{
"status": "ok",
"data": {
"auth_responses": {
"rvpcmk01-cr": {
"status": "noresponse"
},
"rvpcmk02-cr": {
"status": "ok",
"token": "e340f461-12ef-4701-a1a6-ef44439dda94"
}
},
"sync_successful": true,
"sync_nodes_err": [
"rvpcmk01-cr"
],
"sync_responses": {
"rvpcmk01-cr": {
"status": "error"
},
"rvpcmk02-cr": {
"status": "ok",
"result": {
"tokens": "accepted"
}
}
}
},
"log": [
"I, [2017-02-06T17:58:16.146090 #27368] INFO -- : PCSD Debugging enabled\n",
"D, [2017-02-06T17:58:16.146225 #27368] DEBUG -- : Did not detect RHEL 6\n",
"I, [2017-02-06T17:58:16.146304 #27368] INFO -- : Running: /usr/sbin/corosync-cmapctl totem.cluster_name\n",
"I, [2017-02-06T17:58:16.146375 #27368] INFO -- : CIB USER: hacluster, groups: \n",
"D, [2017-02-06T17:58:16.156239 #27368] DEBUG -- : [\"totem.cluster_name (str) = RV-cluster\\n\"]\n",
"D, [2017-02-06T17:58:16.156390 #27368] DEBUG -- : Duration: 0.009834697s\n",
"I, [2017-02-06T17:58:16.156525 #27368] INFO -- : Return Value: 0\n",
"I, [2017-02-06T17:58:16.157212 #27368] INFO -- : SRWT Node: rvpcmk02-cr Request: check_auth\n",
"I, [2017-02-06T17:58:16.160119 #27368] INFO -- : SRWT Node: rvpcmk01-cr Request: check_auth\n",
"I, [2017-02-06T17:58:16.162104 #27368] INFO -- : No response from: rvpcmk01-cr request: /check_auth, exception: Connection refused - connect(2)\n",
"I, [2017-02-06T17:58:16.240477 #27368] INFO -- : No response from: rvpcmk01-cr request: /auth, exception: Connection refused - connect(2)\n",
"I, [2017-02-06T17:58:16.387075 #27368] INFO -- : Running: /usr/sbin/pcs status nodes corosync\n",
"I, [2017-02-06T17:58:16.387208 #27368] INFO -- : CIB USER: hacluster, groups: \n",
"D, [2017-02-06T17:58:16.670689 #27368] DEBUG -- : [\"Corosync Nodes:\\n\", \" Online: rvpcmk02-cr \\n\", \" Offline: rvpcmk01-cr \\n\"]\n",
"D, [2017-02-06T17:58:16.670889 #27368] DEBUG -- : Duration: 0.28344494s\n",
"I, [2017-02-06T17:58:16.671033 #27368] INFO -- : Return Value: 0\n",
"I, [2017-02-06T17:58:16.671978 #27368] INFO -- : Sending config 'tokens' version 36 1f36b1c29146694381cd47b96c01d65876ba8db9 to nodes: rvpcmk02-cr, rvpcmk01-cr\n",
"I, [2017-02-06T17:58:16.672428 #27368] INFO -- : SRWT Node: rvpcmk01-cr Request: set_configs\n",
"I, [2017-02-06T17:58:16.673819 #27368] INFO -- : SRWT Node: rvpcmk02-cr Request: set_configs\n",
"I, [2017-02-06T17:58:16.681322 #27368] INFO -- : No response from: rvpcmk01-cr request: /set_configs, exception: Connection refused - connect(2)\n",
"I, [2017-02-06T17:58:16.783061 #27368] INFO -- : Sending config response from rvpcmk01-cr: {\"status\"=>\"error\"}\n",
"I, [2017-02-06T17:58:16.783169 #27368] INFO -- : Sending config response from rvpcmk02-cr: {\"status\"=>\"ok\", \"result\"=>{\"tokens\"=>\"accepted\"}}\n"
]
}
--Debug Output End--
Error: Unable to communicate with rvpcmk01-cr
rvpcmk02-cr: Authorized
Error: Unable to synchronize and save tokens on nodes: rvpcmk01-cr. Are they authorized?
If you need some other info please tell me what and I will provide.
Answer
add the following firewall rules on all cluster nodes or flush iptables rull
# firewall-cmd --permanent --add-service=high-availability
# firewall-cmd --add-service=high-availability
and then from rvpcmk02-cr (which is DC node) try to re-auth you cluster nodes
# pcs cluster auth rvpcmk01-cr rvpcmk02-cr -u hacluster -p passwd --debug