"kubectl exec" results in "error: unable to upgrade connection: Unauthorized"
I tried kubectl exec on a k8s 1.6.4 RBAC-enabled cluster and the error returned was: error: unable to upgrade connection: Unauthorized. docker exec on the same container succeeds. Otherwise, kubectl is working. kubectl tunnels through an SSH connection but I don't think this is the issue.
kubelet authn is enabled but not authz. The docs say that authz is AlwaysAllow by default, so I have left it this way.
I have a feeling that it is similar to this issue. But the error message is a tad different.
Thanks in advance!
Verbose logs for the kubectl exec command:
I0614 16:50:11.003677 64104 round_trippers.go:398] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true
I0614 16:50:11.003705 64104 round_trippers.go:398] curl -k -v -XPOST -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.6.4 (darwin/amd64) kubernetes/d6f4332" https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true
I0614 16:50:11.169474 64104 round_trippers.go:417] POST https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true 401 Unauthorized in 165 milliseconds
I0614 16:50:11.169493 64104 round_trippers.go:423] Response Headers:
I0614 16:50:11.169497 64104 round_trippers.go:426] Date: Wed, 14 Jun 2017 08:50:11 GMT
I0614 16:50:11.169500 64104 round_trippers.go:426] Content-Length: 12
I0614 16:50:11.169502 64104 round_trippers.go:426] Content-Type: text/plain; charset=utf-8
I0614 16:50:11.169506 64104 round_trippers.go:417] POST https://localhost:6443/api/v1/namespaces/monitoring/pods/alertmanager-main-0/exec?command=%2Fbin%2Fls&container=alertmanager&container=alertmanager&stderr=true&stdout=true 401 Unauthorized in 165 milliseconds
I0614 16:50:11.169509 64104 round_trippers.go:423] Response Headers:
I0614 16:50:11.169512 64104 round_trippers.go:426] Date: Wed, 14 Jun 2017 08:50:11 GMT
I0614 16:50:11.169545 64104 round_trippers.go:426] Content-Length: 12
I0614 16:50:11.169548 64104 round_trippers.go:426] Content-Type: text/plain; charset=utf-8
F0614 16:50:11.169635 64104 helpers.go:119] error: unable to upgrade connection: Unauthorized
Answer
This is an RTFM moment... The solution was basically to follow all the steps on this page for authn, authz, or both.
I had omitted --kubelet-client-certificate and --kubelet-client-key which resulted in the error. Without these flags, kube-apiserver will fail to authenticate with kubelet when you do a kubectl exec.
My original attempt to configure authn was by reading the docs for the kubelet daemon (ie. not the one above). Hence the grave omission.