Kubernetes - Pod Remains in ContainerCreating Status
I am new to all things Kubernetes so still have much to learn.
Have created a two node Kubernetes cluster and both nodes (master and worker) are ready to do work which is good:
[monkey@k8s-dp1 nginx-test]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-dp1 Ready master 2h v1.9.1
k8s-dp2 Ready <none> 2h v1.9.1
Also, all Kubernetes Pods look okay:
[monkey@k8s-dp1 nginx-test]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-k8s-dp1 1/1 Running 0 2h
kube-system kube-apiserver-k8s-dp1 1/1 Running 0 2h
kube-system kube-controller-manager-k8s-dp1 1/1 Running 0 2h
kube-system kube-dns-86cc76f8d-9jh2w 3/3 Running 0 2h
kube-system kube-proxy-65mtx 1/1 Running 1 2h
kube-system kube-proxy-wkkdm 1/1 Running 0 2h
kube-system kube-scheduler-k8s-dp1 1/1 Running 0 2h
kube-system weave-net-6sbbn 2/2 Running 0 2h
kube-system weave-net-hdv9b 2/2 Running 3 2h
However, if I try to create a new deployment in the cluster, the deployment gets created but its pod fails to go into the appropriate RUNNING state. e.g.
[monkey@k8s-dp1 nginx-test]# kubectl apply -f https://k8s.io/docs/tasks/run-application/deployment.yaml
deployment "nginx-deployment" created
[monkey@k8s-dp1 nginx-test]# kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
default nginx-deployment-569477d6d8-f42pz 0/1 ContainerCreating 0 5s
default nginx-deployment-569477d6d8-spjqk 0/1 ContainerCreating 0 5s
kube-system etcd-k8s-dp1 1/1 Running 0 3h
kube-system kube-apiserver-k8s-dp1 1/1 Running 0 3h
kube-system kube-controller-manager-k8s-dp1 1/1 Running 0 3h
kube-system kube-dns-86cc76f8d-9jh2w 3/3 Running 0 3h
kube-system kube-proxy-65mtx 1/1 Running 1 2h
kube-system kube-proxy-wkkdm 1/1 Running 0 3h
kube-system kube-scheduler-k8s-dp1 1/1 Running 0 3h
kube-system weave-net-6sbbn 2/2 Running 0 2h
kube-system weave-net-hdv9b 2/2 Running 3 2h
I am not sure how to figure out what the problem is but if I for example do a kubectl get ev, I can see the following suspect event:
<invalid> <invalid> 1 nginx-deployment-569477d6d8-f42pz.15087c66386edf5d Pod
Warning FailedCreatePodSandBox kubelet, k8s-dp2 Failed create pod sandbox.
But I don't know where to go from here. I can also see that the nginx docker image itself never appears in docker images.
How do I find out more about the problem? Am I missing something fundamental in the kubernetes setup?
--- NEW INFO ---
For background info in case it helps...
Kubernetes nodes are running on CentOS 7 VMs hosted on Windows 10 hyper-v.
--- NEW INFO ---
Running kubectl describe pods shows the following Warning:
Warning NetworkNotReady 1m kubelet, k8s-dp2 network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
--- NEW INFO ---
Switched off the Hyper-v VMs running Kubernetes for the night after my day job hours were over and on my return to the office this morning, I powered up the Kubernetes VMs once again to carry on and, for about 15 mins, the command:
kubectl get pods --all-namespaces was still showing ContainerCreating for those nginx pods the same as yesterday but, right now, the command is now showing all pods as Running including the nginx pods... i.e. the problem solved itself after a full reboot of both master and worker node VMs.
I now did another full reboot again and all pods are showing as Running which is good.
Answer
Use kubectl describe pod <name> to see more info