Improving Data Transfer Rate on Amazon EC2
I've got a 1-Gig EBS Volume mounted to an EC2 instance, I am copying 600MB of binary data from a Local Hard Drive (via RDS Connection) and the copying process windows is showing 10 Hours remaining. Though I have a High Speed connection (100+Mbps)
Whatever the data volume, data transfer rate is 1 Min / MB (i.e 16Kbs / Sec)
I am hesitating between reading Moby Dick in front of my workstation or just taking a day-off.
Are there any reasonable options to speed up this transfer rate ?
(Ideally 512 Kbps/Sec at the minimum)
I am very open to ANY solution to shorten up the uploading/downloading time to/from and EC2 instance.
Thanks in advance.
EDIT :
I just stumbled upon [Amazon Export/Import Service][1]
"AWS Import/Export accelerates transferring large amounts of data between the AWS cloud and portable storage devices that you mail to us"
By "mail to us", they literally mean you "materially" shipping your storage device to Amazon.
Don't say this is Stoneage, this is BRAND new TECHNOLOGY, Dude ! :-)
EDIT2 :
This sounded great: [Aspera for AWs][2] But unfortunately way too expensive;
Tailored for Fortune 500 with big needs and big cash.
Answer
Sometimes, if you want something done it's better to do it yourself :-)
I did not find anything satisfying over the net so I spent the evening doing a rather Complete Bench Test of my own.
I have tested a few alternatives /scenarios and here are the results :
FTP server installed on EC2 was Filezilla Server (correct configuration is touchy)
FTP Client used for this Bench test was WinSPC (Filezilla Client didn't work. see this other post
Legend : [HC] stands for Home Connection (100MBps)
Upload Bandwidth
RDS Upload : 15 Kb/S => Worse Ever
FTP Upload [FTP Server Installed on EC2]: 100 Kb/S
Upload to S3 from AWS Management Console from HC: 60 Kb/S
Upload to S3 using AWS Console interface from EC2 : 145 Kb/S
Upload to S3 using S3 Browser from HC: 120 Kb/S
Upload to S3 using S3 Browser from EC2 : 2000 Kb/S
Download Bandwidth
RDS Download and Upload: 15 Kb/S => Worse Ever
FTP Download [FTP Server Installed on EC2]: 360 Kb/S
Download from S3 AWS Console interface from EC2 : 350 Kb/S
Download from S3 using S3 Browser: from HC: 380 to 620 Kb/S
Download from S3 using S3 Browser: from EC2 : 3000 Kb/S
Conclusions :
So, as of now, Amazon S3 combined with S3 Browser give the best results.
(S3 Browser is just a layer over S3 I don't get it why the upload rate is better)
However, one should keep in mind that an FTP Server on an EC2 instance has the great advantage of Mapping directly a local EC2 Directory into EC2. Unlike S3, there is only one transfer involved. Indeed, S3 requires 2 Transfers : Form Local Resource to S3/and from S3 to EC2 and the other way round, while FTP Access grants immediacy by shortening transfer cycles. Besides, it spares the cost of S3 Buckets.
It is also interesting to mention that EC2 Instance's bandwidth is really strong. It is hence far more interesting -needless to say, to use protocols that really take advantage of it such as S3 or FTP, rather than RDS.
I hope this will be useful to other people facing the same issue and spare them precious time.