Linux mount fails with error Transport endpoint not connected

Question

From time to time for reasons unknown, the Amazon S3 Fuse mount on a linux server fails throughout the day. The only resolution is to umount and then mount the directory again. I tried writing the following shell script which when manually unmounted it worked and remounted but I learned there must be some other "state" when a link fails but is not actually unmounted.

Original error:

[root@app3 mnt]# cd s3fs
[root@app3 s3fs]# ls
ls: cannot access amazon: Transport endpoint is not connected
amazon
[root@app3 s3fs]# umount amazon 
[root@app3 s3fs]# mount amazon/

Shell script attempt to check mount and remount if failed (worked in manual tests but failed):

   #!/bin/bash

   cat /etc/mtab | grep /mnt/$1 >/dev/null
   if [ "$?" -eq "0" ]; then
        echo /mnt/$1 is mounted.
   else
        echo /mnt/$1 is not mounted at this time.
        echo remounting now...
        umount /mnt/$1
        mount /mnt/$1
   fi

---

1. Why would the shell script work if I manually unmount the directory and run test, but when transport endpoint fails the test returns true and remount doesn't happen?
2. What is the best way to solve this?

Answer 1

I know this is old but it might help others facing this issue. We had a similar problem with our bucket being unmounted randomly and getting the 'Transport endpoint is not connected' error.

Instead of using "cat /etc/mtab", I use "df -hT" and it works with my script. The problem is it gets stuck in this weird state, of being half unmounted and the "mtab" still sees it as mounted; but I still don't know why.

This is the code I'm using:

#!/bin/bash

if [ $(df -hT | grep -c s3fs) != 1 ]
then 
        # unmount it first
        umount /path/to/mounted/bucket;

        # remount it
        /usr/local/bin/s3fs bucket-name /path/to/mount/bucket -o noatime -o allow_other;

        echo "s3fs is down";

        # maybe send email here to let you know it went down
fi

Also make sure you run your script as root, otherwise it won't be able to unmount/remount.