How can directly access AWS S3 buckets as a dataset in RStudioServer?

r amazon-s3 amazon-ec2 rstudio-server
Josh Beauregard picture Josh Beauregard · Mar 19, 2016 · Viewed 8.7k times · Source

I have multiple s3 buckets on an aws account and I also have a EC2 machine running Rstudio Pro. I would like to access my S3 buckets (that are several Terabytes of data each).

I would like to be able to set up rstudio to mount the buckets as Data sets with out having to copy the whole thing into an EBS before reading it every time.

Any help would be great.

Answer

spsaaibi picture spsaaibi · Mar 19, 2016

It seems that you could try the aws.s3 package from the cloudyr project, https://github.com/cloudyr/aws.s3.

With this, assuming you have your data on a private bucket, you could access it as follows:

aws.s3::getbucket(
bucket = 'hpk',
key = YOUR_AWS_ACCESS_KEY,
secret = YOUR_AWS_SECRET_ACCESS_KEY
)

Hopefully this will help you accessing data from your buckets. You can then also try aws.ec2 to communicate with your ec2 machine.

Related questions

Remove rows with all or some NAs (missing values) in data.frame

I'd like to remove the lines in this data frame that: a) contain NAs across all columns. Below is my example data frame. gene hsap mmul mmus rnor cfam 1 ENSG00000208234 0 NA NA NA NA 2 ENSG00000199674 0 2 2 2 2 3 ENSG00000221622 0 NA NA NA NA 4 …

Read More
Drop data frame columns by name

I have a number of columns that I would like to remove from a data frame. I know that we can delete them individually using something like: df$x <- NULL But I was hoping to do this with …

Read More
Changing column names of a data frame

I have a data frame called "newprice" (see below) and I want to change the column names in my program in R. > newprice Chang. Chang. Chang. 1 100 36 136 2 120 -33 87 3 150 14 164 In fact this is what am doing: names(newprice)[1]<-paste("premium") …

Read More