Kube flannel in CrashLoopBackOff status
We just start to create our cluster on kubernetes.
Now we try to deploy tiller but we have en error:
NetworkPlugin cni failed to set up pod "tiller-deploy-64c9d747bd-br9j7_kube-system" network: open /run/flannel/subnet.env: no such file or directory
After that I call:
kubectl get pods --all-namespaces -o wide
And got response:
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
kube-system coredns-78fcdf6894-ksdvt 1/1 Running 2 7d 192.168.0.4 kube-master <none>
kube-system coredns-78fcdf6894-p4l9q 1/1 Running 2 7d 192.168.0.5 kube-master <none>
kube-system etcd-kube-master 1/1 Running 2 7d 10.168.209.20 kube-master <none>
kube-system kube-apiserver-kube-master 1/1 Running 2 7d 10.168.209.20 kube-master <none>
kube-system kube-controller-manager-kube-master 1/1 Running 2 7d 10.168.209.20 kube-master <none>
kube-system kube-flannel-ds-amd64-42rl7 0/1 CrashLoopBackOff 2135 7d 10.168.209.17 node5 <none>
kube-system kube-flannel-ds-amd64-5fx2p 0/1 CrashLoopBackOff 2164 7d 10.168.209.14 node2 <none>
kube-system kube-flannel-ds-amd64-6bw5g 0/1 CrashLoopBackOff 2166 7d 10.168.209.15 node3 <none>
kube-system kube-flannel-ds-amd64-hm826 1/1 Running 1 7d 10.168.209.20 kube-master <none>
kube-system kube-flannel-ds-amd64-thjps 0/1 CrashLoopBackOff 2160 7d 10.168.209.16 node4 <none>
kube-system kube-flannel-ds-amd64-w99ch 0/1 CrashLoopBackOff 2166 7d 10.168.209.13 node1 <none>
kube-system kube-proxy-d6v2n 1/1 Running 0 7d 10.168.209.13 node1 <none>
kube-system kube-proxy-lcckg 1/1 Running 0 7d 10.168.209.16 node4 <none>
kube-system kube-proxy-pgblx 1/1 Running 1 7d 10.168.209.20 kube-master <none>
kube-system kube-proxy-rnqq5 1/1 Running 0 7d 10.168.209.14 node2 <none>
kube-system kube-proxy-wc959 1/1 Running 0 7d 10.168.209.15 node3 <none>
kube-system kube-proxy-wfqqs 1/1 Running 0 7d 10.168.209.17 node5 <none>
kube-system kube-scheduler-kube-master 1/1 Running 2 7d 10.168.209.20 kube-master <none>
kube-system kubernetes-dashboard-6948bdb78-97qcq 0/1 ContainerCreating 0 7d <none> node5 <none>
kube-system tiller-deploy-64c9d747bd-br9j7 0/1 ContainerCreating 0 45m <none> node4 <none>
We have some flannel pods in CrashLoopBackOff status. For example kube-flannel-ds-amd64-42rl7.
When I call:
kubectl describe pod -n kube-system kube-flannel-ds-amd64-42rl7
I've got status Running:
Name: kube-flannel-ds-amd64-42rl7
Namespace: kube-system
Priority: 0
PriorityClassName: <none>
Node: node5/10.168.209.17
Start Time: Wed, 22 Aug 2018 16:47:10 +0300
Labels: app=flannel
controller-revision-hash=911701653
pod-template-generation=1
tier=node
Annotations: <none>
Status: Running
IP: 10.168.209.17
Controlled By: DaemonSet/kube-flannel-ds-amd64
Init Containers:
install-cni:
Container ID: docker://eb7ee47459a54d401969b1770ff45b39dc5768b0627eec79e189249790270169
Image: quay.io/coreos/flannel:v0.10.0-amd64
Image ID: docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac
Port: <none>
Host Port: <none>
Command:
cp
Args:
-f
/etc/kube-flannel/cni-conf.json
/etc/cni/net.d/10-flannel.conflist
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 22 Aug 2018 16:47:24 +0300
Finished: Wed, 22 Aug 2018 16:47:24 +0300
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/etc/cni/net.d from cni (rw)
/etc/kube-flannel/ from flannel-cfg (rw)
/var/run/secrets/kubernetes.io/serviceaccount from flannel-token-9wmch (ro)
Containers:
kube-flannel:
Container ID: docker://521b457c648baf10f01e26dd867b8628c0f0a0cc0ea416731de658e67628d54e
Image: quay.io/coreos/flannel:v0.10.0-amd64
Image ID: docker-pullable://quay.io/coreos/flannel@sha256:88f2b4d96fae34bfff3d46293f7f18d1f9f3ca026b4a4d288f28347fcb6580ac
Port: <none>
Host Port: <none>
Command:
/opt/bin/flanneld
Args:
--ip-masq
--kube-subnet-mgr
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Thu, 30 Aug 2018 10:15:04 +0300
Finished: Thu, 30 Aug 2018 10:15:08 +0300
Ready: False
Restart Count: 2136
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_NAME: kube-flannel-ds-amd64-42rl7 (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/etc/kube-flannel/ from flannel-cfg (rw)
/run from run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from flannel-token-9wmch (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
run:
Type: HostPath (bare host directory volume)
Path: /run
HostPathType:
cni:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType:
flannel-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-flannel-cfg
Optional: false
flannel-token-9wmch:
Type: Secret (a volume populated by a Secret)
SecretName: flannel-token-9wmch
Optional: false
QoS Class: Guaranteed
Node-Selectors: beta.kubernetes.io/arch=amd64
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Pulled 51m (x2128 over 7d) kubelet, node5 Container image "quay.io/coreos/flannel:v0.10.0-amd64" already present on machine
Warning BackOff 1m (x48936 over 7d) kubelet, node5 Back-off restarting failed container
here kube-controller-manager.yaml:
apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
component: kube-controller-manager
tier: control-plane
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- command:
- kube-controller-manager
- --address=127.0.0.1
- --allocate-node-cidrs=true
- --cluster-cidr=192.168.0.0/24
- --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
- --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
- --controllers=*,bootstrapsigner,tokencleaner
- --kubeconfig=/etc/kubernetes/controller-manager.conf
- --leader-elect=true
- --node-cidr-mask-size=24
- --root-ca-file=/etc/kubernetes/pki/ca.crt
- --service-account-private-key-file=/etc/kubernetes/pki/sa.key
- --use-service-account-credentials=true
image: k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 8
httpGet:
host: 127.0.0.1
path: /healthz
port: 10252
scheme: HTTP
initialDelaySeconds: 15
timeoutSeconds: 15
name: kube-controller-manager
resources:
requests:
cpu: 200m
volumeMounts:
- mountPath: /etc/ssl/certs
name: ca-certs
readOnly: true
- mountPath: /etc/kubernetes/controller-manager.conf
name: kubeconfig
readOnly: true
- mountPath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
name: flexvolume-dir
- mountPath: /etc/pki
name: etc-pki
readOnly: true
- mountPath: /etc/kubernetes/pki
name: k8s-certs
readOnly: true
hostNetwork: true
priorityClassName: system-cluster-critical
volumes:
- hostPath:
path: /etc/ssl/certs
type: DirectoryOrCreate
name: ca-certs
- hostPath:
path: /etc/kubernetes/controller-manager.conf
type: FileOrCreate
name: kubeconfig
- hostPath:
path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec
type: DirectoryOrCreate
name: flexvolume-dir
- hostPath:
path: /etc/pki
type: DirectoryOrCreate
name: etc-pki
- hostPath:
path: /etc/kubernetes/pki
type: DirectoryOrCreate
name: k8s-certs
status: {}
OS is CentOS Linux release 7.5.1804
logs from one of pods:
# kubectl logs --namespace kube-system kube-flannel-ds-amd64-5fx2p
main.go:475] Determining IP address of default interface
main.go:488] Using interface with name eth0 and address 10.168.209.14
main.go:505] Defaulting external address to interface address (10.168.209.14)
kube.go:131] Waiting 10m0s for node controller to sync
kube.go:294] Starting kube subnet manager
kube.go:138] Node controller sync successful
main.go:235] Created subnet manager: Kubernetes Subnet Manager - node2
main.go:238] Installing signal handlers
main.go:353] Found network config - Backend type: vxlan
vxlan.go:120] VXLAN config: VNI=1 Port=0 GBP=false DirectRouting=false
main.go:280] Error registering network: failed to acquire lease: node "node2" pod cidr not assigned
main.go:333] Stopping shutdownHandler...
Where error is?
Answer
For flannel to work correctly, you must pass --pod-network-cidr=10.244.0.0/16 to kubeadm init.