Is it better to have multiple s3 buckets or one bucket with sub folders?

Question

Is it better to have multiple s3 buckets per category of uploads or one bucket with sub folders OR a linked s3 bucket? I know for sure there will be more user-images than there will be profille-pics and that there is a 5TB limit per bucket and 100 buckets per account. I'm doing this using aws boto library and https://github.com/amol-/depot

Which is the structure my folders in which of the following manner?

/app_bucket
    /profile-pic-folder
    /user-images-folder

OR

profile-pic-bucket
user-images-bucket


OR


/app_bucket_1
/app_bucket_2

The last one implies that its really a 10TB bucket where a new bucket is created when the files within bucket_1 exceeds 5TB. But all uploads will be read as if in one bucket. Or is there a better way of doing what I'm trying to do? Many thanks!

I'm not sure if this is correct... 100 buckets per account?

https://www.reddit.com/r/aws/comments/28vbjs/requesting_increase_in_number_of_s3_buckets/

Answer 1

Yes, there is actually a 100 bucket limit per account. I asked the reason for that to an architect in an AWS event. He said this is to avoid people hosting unlimited static websites on S3 as they think this may be abused. But you can apply for an increase.

By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your bucket limit by submitting a service limit increase.

Source: http://docs.aws.amazon.com/AmazonS3/latest/dev/BucketRestrictions.html

Also, please note that there are actually no folders in S3, just a flat file structure:

Amazon S3 has a flat structure with no hierarchy like you would see in a typical file system. However, for the sake of organizational simplicity, the Amazon S3 console supports the folder concept as a means of grouping objects. Amazon S3 does this by using key name prefixes for objects.

Source: http://docs.aws.amazon.com/AmazonS3/latest/UG/FolderOperations.html

Finally, the 5TB limit only applies to a single object. There is no limit on the number of objects or total size of the bucket.

Q: How much data can I store?

The total volume of data and number of objects you can store are unlimited.

Source: https://aws.amazon.com/s3/faqs/

Also the documentation states there is no performance difference between using a single bucket or multiple buckets so I guess both option 1 and 2 would be suitable for you.

Hope this helps.