timeout expired waiting for volumes to attach or mount for pod

Question

I install kubernetes on ubuntu on baremetal. I deploy 1 master and 3 worker. and then deploy rook and every thing work fine.but when i want to deploy a wordpress on it , i get this error

Unable to mount volumes for pod "wordpress-mysql-b78774f44-lxtfv_default(ffb4ff12-553e-11e9-a229-52540076d16c)": timeout expired waiting for volumes to attach or mount for pod "default"/"wordpress-mysql-b78774f44-lxtfv". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-nj8xw]

#kubectl describe pods wordpress-mysql-b78774f44-lxtfv

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  7m11s (x4 over 7m18s)  default-scheduler  pod has unbound immediate PersistentVolumeClaims (repeated 3 times)
  Normal   Scheduled         7m5s                   default-scheduler  Successfully assigned default/wordpress-mysql-b78774f44-lxtfv to worker3
  Warning  FailedMount       2m46s (x2 over 5m1s)   kubelet, worker3   Unable to mount volumes for pod "wordpress-mysql-b78774f44-lxtfv_default(ffb4ff12-553e-11e9-a229-52540076d16c)": timeout expired waiting for volumes to attach or mount for pod "default"/"wordpress-mysql-b78774f44-lxtfv". list of unmounted volumes=[mysql-persistent-storage]. list of unattached volumes=[mysql-persistent-storage default-token-nj8xw]
  Normal   Pulled            107s                   kubelet, worker3   Container image "mysql:5.6" already present on machine
  Normal   Created           104s                   kubelet, worker3   Created container mysql
  Normal   Started           101s                   kubelet, worker3   Started container mysql


and my pods is running
#kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
wordpress-595685cc49-2bmzk        1/1     Running   4          15m
wordpress-mysql-b78774f44-lxtfv   1/1     Running   0          15m

my pv and pvc

#kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                    STORAGECLASS      REASON   AGE
persistentvolume/pvc-0f6722a0-553f-11e9-a229-52540076d16c   20Gi       RWO            Delete           Bound      default/wp-pv-claim      rook-ceph-block            15m
persistentvolume/pvc-e9797517-553b-11e9-a229-52540076d16c   20Gi       RWO            Delete           Released   default/wp-pv-claim      rook-ceph-block            37m
persistentvolume/pvc-ff52a22e-553e-11e9-a229-52540076d16c   20Gi       RWO            Delete           Bound      default/mysql-pv-claim   rook-ceph-block            16m

NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/mysql-pv-claim   Bound    pvc-ff52a22e-553e-11e9-a229-52540076d16c   20Gi       RWO            rook-ceph-block   16m
persistentvolumeclaim/wp-pv-claim      Bound    pvc-0f6722a0-553f-11e9-a229-52540076d16c   20Gi       RWO   

     rook-ceph-block   15m

my wordpress yaml file

# cat wordpress.yaml 
apiVersion: v1
kind: Service
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  ports:
  - port: 80
  selector:
    app: wordpress
    tier: frontend
  type: LoadBalancer
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: wp-pv-claim
  labels:
    app: wordpress
spec:
  storageClassName: rook-ceph-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: frontend
    spec:
      containers:
      - image: wordpress:4.6.1-apache
        name: wordpress
        env:
        - name: WORDPRESS_DB_HOST
          value: wordpress-mysql
        - name: WORDPRESS_DB_PASSWORD
          value: changeme
        ports:
        - containerPort: 80
          name: wordpress
        volumeMounts:
        - name: wordpress-persistent-storage
          mountPath: /var/www/html
      volumes:
      - name: wordpress-persistent-storage
        persistentVolumeClaim:
          claimName: wp-pv-claim
and my mysql yaml file
cat mysql.yaml 
apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  ports:
    - port: 3306
  selector:
    app: wordpress
    tier: mysql
  clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-pv-claim
  labels:
    app: wordpress
spec:
  storageClassName: rook-ceph-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: wordpress-mysql
  labels:
    app: wordpress
spec:
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: wordpress
        tier: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: changeme
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim

Answer 1

It looks like your deployment has progressed successfully! The error you see occurred 2m46s from the time that you did the describe. This could be due to ceph provisioning the PV/PVC, or some other situation in which your PVC is not ready to be mounted. However, according to the events, your container eventually successfully mounts and is started. All the pod health checks pass and the Kube scheduler progresses it into the READY state.

Is there any evidence that your container and the mount is not working properly after start?