Can S3 Select search multiple objects?

amazon-web-services amazon-s3 amazon-s3-select
Nic Cottrell picture Nic Cottrell · Jun 6, 2018 · Viewed 11.3k times · Source

I'm testing out S3 Select and as far as I understand from the examples, you can treat a single object (CSV or JSON) as a data store.

I wanted to have a single JSON document per S3 object and search the entire bucket as a 'database'. I'm saving each 'file' as <ID>.json and each file has JSON documents with the same schema.

Is it possible to search multiple objects in a single call? i.e. Find all JSON documents where customerId = 123 ?

Answer

John Rotenstein picture John Rotenstein · Jun 7, 2018

It appears that Amazon S3 Select operates on only one object.

You can use Amazon Athena to run queries across paths, which will include all files within that path. It also supports partitioning.

Related questions

What is difference between s3 select and athena

I am trying to understand what is difference between aws athena service and newly released s3 select (still in preview). And how usecases different for both of those? It seems both helps in selecting partial data from s3.

Read More
s3-select querying data on field name

I am trying to query data from json data in s3-select. { person = [ { "Id": 1, "Name": "Anshu", "Address": "Templestowe", "Car": "Jeep" } { "Id": 2, "Name": "Ben Mostafa", "Address": "Las Vegas", "Car": "Mustang" } { "Id": 3, "Name": "Rohan Wood", "Address": "Wooddon", "Car": "VW" } ] } QUERY = "select * from …

Read More
Downloading an entire S3 bucket?

I noticed that there doesn't seem to be an option to download an entire S3 bucket from the AWS Management Console. Is there an easy way to grab everything in one of my buckets? I was thinking about making the …

Read More