Running MongoDB on Kubernetes Minikube with local persistent storage
I am currently trying to reproduce this tutorial on Minikube:
http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html
I updated the configuration files to use a hostpath as a persistent storage on minikube node.
kind: PersistentVolume
apiVersion: v1
metadata:
name: pv0001
labels:
type: local
spec:
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/tmp"
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: myclaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
---
apiVersion: v1
kind: Service
metadata:
name: mongo
labels:
name: mongo
spec:
ports:
- port: 27017
targetPort: 27017
clusterIP: None
selector:
role: mongo
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
spec:
serviceName: "mongo"
replicas: 3
template:
metadata:
labels:
role: mongo
environment: test
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo
command:
- mongod
- "--replSet"
- rs0
- "--smallfiles"
- "--noprealloc"
ports:
- containerPort: 27017
volumeMounts:
- name: myclaim
mountPath: /data/db
- name: mongo-sidecar
image: cvallance/mongo-k8s-sidecar
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=test"
volumeClaimTemplates:
- metadata:
name: myclaim
Which result in the following:
kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM REASON AGE
pv0001 1Gi RWO Retain Available 17s
pvc-134a6c0f-1565-11e7-9cf1-080027f4d8c3 1Gi RWO Delete Bound default/myclaim 11s
kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES AGE
myclaim Bound pvc-134a6c0f-1565-11e7-9cf1-080027f4d8c3 1Gi RWO 14s
kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.0.0.1 <none> 443/TCP 3d
mongo None <none> 27017/TCP 53s
kubectl get pod
No resources found.
kubectl describe service mongo
Name: mongo
Namespace: default
Labels: name=mongo
Selector: role=mongo
Type: ClusterIP
IP: None
Port: <unset> 27017/TCP
Endpoints: <none>
Session Affinity: None
No events.
kubectl get statefulsets
NAME DESIRED CURRENT AGE
mongo 3 0 4h
kubectl describe statefulsets mongo
Name: mongo
Namespace: default
Image(s): mongo,cvallance/mongo-k8s-sidecar
Selector: environment=test,role=mongo
Labels: environment=test,role=mongo
Replicas: 0 current / 3 desired
Annotations: <none>
CreationTimestamp: Thu, 30 Mar 2017 18:23:56 +0200
Pods Status: 0 Running / 0 Waiting / 0 Succeeded / 0 Failed
No volumes.
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------------- -------
1s 1s 4 {statefulset } WarningFailedCreate pvc: myclaim-mongo-0, error: PersistentVolumeClaim "myclaim-mongo-0" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
1s 1s 4 {statefulset } WarningFailedCreate pvc: myclaim-mongo-1, error: PersistentVolumeClaim "myclaim-mongo-1" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
1s 0s 4 {statefulset } WarningFailedCreate pvc: myclaim-mongo-2, error: PersistentVolumeClaim "myclaim-mongo-2" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
kubectl get ev | grep mongo
29s 1m 15 mongo StatefulSet Warning FailedCreate {statefulset } pvc: myclaim-mongo-0, error: PersistentVolumeClaim "myclaim-mongo-0" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
29s 1m 15 mongo StatefulSet Warning FailedCreate {statefulset } pvc: myclaim-mongo-1, error: PersistentVolumeClaim "myclaim-mongo-1" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
29s 1m 15 mongo StatefulSet Warning FailedCreate {statefulset } pvc: myclaim-mongo-2, error: PersistentVolumeClaim "myclaim-mongo-2" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]
kubectl describe pvc myclaim
Name: myclaim
Namespace: default
StorageClass: standard
Status: Bound
Volume: pvc-134a6c0f-1565-11e7-9cf1-080027f4d8c3
Labels: <none>
Capacity: 1Gi
Access Modes: RWO
No events.
minikube version: v0.17.1
It seems that the service is not able to load pods, which makes it complicated to debug with kubectl logs. Is there something wrong with the way I am creating a persistent volume on my node ?
Thanks a lot
Answer
TL; DR
In the situation described in the question the problem was that the Pods for the StatefulSet did not start up at all therefore the Service had no targets. The reason for not starting up was:
WarningFailedCreate pvc: myclaim-mongo-0, error: PersistentVolumeClaim "myclaim-mongo-0" is invalid: [spec.accessModes: Required value: at least 1 access mode is required, spec.resources[storage]: Required value]`
And since the volume by default is defined as required the Pod won't start without it. So edit the StatefulSet's volumeClaimTemplate to have:
volumeClaimTemplates:
- metadata:
name: myclaim
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 1Gi
(There is no need to create the PersistentVolumeClaim manually.)
More general solution
If can't connect a Service try this command:
kubectl describe service myservicename
And if you see something like this in the output:
Endpoints: <none>
That means there are no targets (usually Pods) running or the targets are not ready. To find out which one is the case do:
kubectl describe endpoint myservicename
It will list all endpoints, ready or not. If not ready, investigate the readinessProbe in the Pod. If doesn't exist then try to find out why by looking at the StatefulSet (Deployment, ReplicaSet, ReplicationController, etc) itself for messages (the Events section):
kubectl describe statefulset mystatefulsetname
This information is available if you do:
kubectl get ev | grep something
If you are sure they are running and ready then the labels on the Pods and the Service do not match up.