S3FS - Recursive CHOWN/CHMOD takes a LONG time

amazon-s3 fuse s3fs
Jafo picture Jafo · Jun 9, 2015 · Viewed 7.3k times · Source

Any recursive chown or chmod command on an s3fs mount takes a long time when you have a few directories (about 70) each with quite a few files.

Either of these commands are likely to take almost 24 hours. I have to do this or the Apache process cannot access these files/directories. The command on a normal mount takes about 20 seconds.

Mounting with:

/storage -o noatime -o allow_other -o use_cache=/s3fscache -o default_acl=public-read-write

In /etc/fuse.conf:

user_allow_other

Using latest version: 1.78

Any thoughts on how to do this faster?

Answer

Jafo picture Jafo · Jul 7, 2015

After a while, I found it just better to parallel the processes in order to speed it up. Example:

find /s3fsmount/path/to/somewhere -print | xargs --max-args=1 --max-procs=100 chmod 777

It is still slow, but nowhere near as slow as it was.

Related questions

How can I mount an S3 bucket to an EC2 instance and write to it with PHP?

I'm working on a project that is being hosted on Amazon Web Services. The server setup consists of two EC2 instances, one Elastic Load Balancer and an extra Elastic Block Store on which the web application resides. The project is …

Read More
How to mount a Amazon S3 bucket by using FUSE - S3FS

This is driving me up the wall. I am still a little new to Linux but i do understand how to do most of day to day stuff I need to do. What I am trying to do is mount …

Read More
How to mount S3 bucket on Kubernetes container/pods?

I am trying to run my spark job on Amazon EKS cluster. My spark job required some static data (reference data) at each data nodes/worker/executor and this reference data is available at S3. Can somebody kindly help me …

Read More