How do I get the size of a boto3 Collection?

python collections boto3
Rahul Gupta-Iwasaki picture Rahul Gupta-Iwasaki · Sep 5, 2015 · Viewed 16.8k times · Source

The way I have been using is to transform the Collection into a List and query the length:

s3 = boto3.resource('s3')
bucket = s3.Bucket('my_bucket')
size = len(list(bucket.objects.all()))

However, this forces resolution of the whole collection and obviates the benefits of using a Collection in the first place. Is there a better way to do this?

Answer

AChampion picture AChampion · Sep 5, 2015

There is no way to get the count of keys in a bucket without listing all the objects this is a limitation of AWS S3 (see https://forums.aws.amazon.com/thread.jspa?messageID=164220).

Getting the Object Summaries (HEAD) doesn't get the actual data so should be a relatively inexpensive operation and if you are just discarding the list then you could do:

size = sum(1 for _ in bucket.objects.all())

Which will give you the number of objects without constructing a list.

Related questions

Is there a short contains function for lists?

I see people are using any to gather another list to see if an item exists in a list, but is there a quick way to just do?: if list.contains(myItem): # do something

Read More
Accessing items in an collections.OrderedDict by index

Lets say I have the following code: import collections d = collections.OrderedDict() d['foo'] = 'python' d['bar'] = 'spam' Is there a way I can access the items in a numbered manner, like: d(0) #foo's Output d(1) #bar's Output

Read More
How to sort Counter by value? - python

Other than doing list comprehensions of reversed list comprehension, is there a pythonic way to sort Counter by value? If so, it is faster than this: >>> from collections import Counter >>> x = Counter({'a':5, 'b':3, …

Read More