Fastest way to get Google Storage bucket size?

google-cloud-storage gsutil
fredrik picture fredrik · Dec 9, 2014 · Viewed 17.6k times · Source

I'm currently doing this, but it's VERY slow since I have several terabytes of data in the bucket:

gsutil du -sh gs://my-bucket-1/

And the same for a sub-folder:

gsutil du -sh gs://my-bucket-1/folder

Is it possible to somehow obtain the total size of a complete bucket (or a sub-folder) elsewhere or in some other fashion which is much faster?

Answer

Brandon Yarbrough picture Brandon Yarbrough · Dec 9, 2014

Unfortunately, no. If you need to know what size the bucket is right now, there's no faster way than what you're doing.

If you need to check on this regularly, you can enable bucket logging. Google Cloud Storage will generate a daily storage log that you can use to check the size of the bucket. If that would be useful, you can read more about it here: https://cloud.google.com/storage/docs/accesslogs#delivery

Related questions

gsutil ServiceException: 401 Anonymous caller does not have storage.objects.list access to bucket even though I'm loggedin in gcloud

I am trying to create an internal app to upload files to google cloud. I don't want each individual user or this app to log in so I'm using a service account. I login into the service account and everything …

Read More
Gsutil - How can I check if a file exists in a GCS bucket (a sub-directory) using Gsutil

I have a GCS bucket containing some files in the path gs://main-bucket/sub-directory-bucket/object1.gz I would like to programmatically check if the sub-directory bucket contains one specific file. I would like to do this using gsutil. How could …

Read More
ResumableUploadAbortException: 403 Insufficient Permission

I am using Google Cloud Storage and Google Compute Engine and am trying to upload to my bucket from my VM. My bucket has read/write permissions and so does anyone that has access to it. They are labeled as …

Read More