etcdctl throws Error: context deadline exceeded error

Uday Vishwakarma picture Uday Vishwakarma · Dec 6, 2018 · Viewed 6.9k times · Source

I'm trying to create a one node etcd cluster on AWS using coreos cloud-config. I have created a Route53 recordset with value etcd.uday.com which has a alias to the ELB which points to the ec2 instance. Etcd is running successfully but when I run the etcd member list command I get below error

ETCDCTL_API=3 etcdctl member list \
--endpoints=https://etcd.udayvishwakarma.com:2379 \
--cacert=./ca.pem \
--cert=etcd-client.pem \
--key=etcd-client-key.pem

Error: context deadline exceeded

However, it lists members when --insecure-skip-tls-verify flag is added to the etcdctl member list command. I have generated certificated using cfssl using below configs

ca.json

{
"CN": "Root CA",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "UK",
      "L": "London",
      "O": "Kubernetes",
      "OU": "CA"
    }
  ],
  "ca": {
    "expiry": "87658h"
  }
}

ca.config

 {
  "signing": {
    "default": {
      "expiry": "2190h"
    },
    "profiles": {
      "client": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "server": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "peer": {
        "expiry": "8760h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      },
      "ca": {
        "usages": [
          "signing",
          "digital signature",
          "cert sign",
          "crl sign"
        ],
        "expiry": "26280h",
        "is_ca": true
      }
    }
  }
}

etcd-member.json

 {
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "hosts":[
    "etcd.uday.com"
  ],
  "names": [
    {
      "O": "Kubernetes"
    }
  ]
}

etcd-client.json

 {
  "CN": "etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "hosts":[
    "etcd.uday.com"
  ],
  "names": [
    {
      "O": "Kubernetes"
    }
  ]
}



     cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -hostname="etcd.uday.com" \
          -config=ca-config.json -profile=peer \
etcd-member.json | cfssljson -bare etcd-member

    cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -hostname="etcd.uday.com" \
    -config=ca-config.json -profile=client\
 etcd-client.json | cfssljson -bare etcd-client

My etcd-member.service systemd unit cloudconfig is as below

    units:
  - name: etcd-member.service
    drop-ins:
    - name: aws-etcd-cluster.conf
      content: |
        [Service]
        Environment=ETCD_USER=etcd
        Environment=ETCD_NAME=%H
        Environment=ETCD_IMAGE_TAG=v3.1.12
        Environment=ETCD_SSL_DIR=/etc/etcd/ssl

        Environment=ETCD_CA_FILE=/etc/ssl/certs/ca.pem
        Environment=ETCD_CERT_FILE=/etc/ssl/certs/etcd-client.pem
        Environment=ETCD_KEY_FILE=/etc/ssl/certs/etcd-client-key.pem
        Environment=ETCD_CLIENT_CERT_AUTH=true
        Environment=ETCD_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem
        Environment=ETCD_PEER_CA_FILE=/etc/ssl/certs/ca.pem
        Environment=ETCD_PEER_CERT_FILE=/etc/ssl/certs/etcd-member.pem
        Environment=ETCD_PEER_KEY_FILE=/etc/ssl/certs/etcd-member-key.pem
        Environment=ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/ca.pem


        Environment=ETCD_INITIAL_CLUSTER_STATE=new
        Environment=ETCD_INITIAL_CLUSTER=%H=https://%H:2380
        Environment=ETCD_DATA_DIR=/var/lib/etcd3
        Environment=ETCD_LISTEN_CLIENT_URLS=https://%H:2379,https://127.0.0.1:2379
        Environment=ETCD_ADVERTISE_CLIENT_URLS=https://%H:2379
        Environment=ETCD_LISTEN_PEER_URLS=https://%H:2380
        Environment=ETCD_INITIAL_ADVERTISE_PEER_URLS=https://%H:2380
        PermissionsStartOnly=true
        Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid"
        ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid
        ExecStartPre=/usr/bin/sed -i 's/^ETCDCTL_ENDPOINT.*$/ETCDCTL_ENDPOINT=https:\/\/%H:2379/' /etc/environment
        ExecStartPre=/usr/bin/mkdir -p /var/lib/etcd3
        ExecStartPre=/usr/bin/chown -R etcd:etcd /var/lib/etcd3
        ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid
    enable: true
    command: start

Is cert generation wrong or something I have missed?

Answer

Junaid picture Junaid · Jun 26, 2019

The certificates are generated for etcd.uday.com. You are trying to connect using etcd.udayvishwakarma.com while certificate is valid for etcd.uday.com. Change endpoint on etcdctl from etcd.udayvishwakarma.com to etcd.uday.com.