How fix this error "watch chan error: etcdserver: mvcc: required revision has been compacted"?

Question

I have a kubernetes cluster with two nodes, and two nodes for etcd,also I am using authentication for my kubernetes. When I run this command: kubectl get cs I get this output:

NAME                 STATUS      MESSAGE                                                                                        ERROR
scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: getsockopt: connection refused   
controller-manager   Healthy     ok                                                                                             
etcd-1               Healthy     {"health": "true"}                                                                             
etcd-0               Healthy     {"health": "true"}   

when I check logs of my kube-apiserver I get this error:

E0516 22:28:14.312855       1 watcher.go:188] watch chan error: etcdserver: mvcc: required revision has been compacted

what's this error for and how can I fix this?

also I get this warning for my kube-controller-manager:

W0516 22:28:14.314137       1 reflector.go:323] k8s.io/kubernetes/pkg/controller/garbagecollector/graph_builder.go:192: watch of <nil> ended with: etcdserver: mvcc: required revision has been compacted

and in for my kube-scheduler:

W0516 10:43:38.097940       1 reflector.go:323] k8s.io/kubernetes/plugin/pkg/scheduler/factory/factory.go:451: watch of *v1.Pod ended with: too old resource version: 2153673 (2154157)

my kubernetes version is 1.6.3

I really appreciate your help:)

Answer 1

update etcd version to 3.0.11 or later

https://github.com/kubernetes/kubernetes/issues/45506