Can we consider AWS Glue as a replacement for EMR?

amazon-web-services etl amazon-emr aws-glue
Yuva picture Yuva · Jan 12, 2018 · Viewed 17.5k times · Source

Just a quick question to clarify from Masters, since AWS Glue as an ETL tool, can provide companies with benefits such as, minimal or no server maintenance, cost savings by avoiding over-provisioning or under-provisioning resources, besides running on spark, I am looking for some clarifications, if AWS Glue can replace EMR?

If both can co-exist, how EMR can play a role along with AWS Glue?

Thanks & regards

Yuva

Answer

ctrl-c picture ctrl-c · Jan 13, 2018

As per my understanding, glue cannot be a replacement for EMR. It actually depends on your usecase. There are some limitations with glue ETL;

  • It does not support --packages.
  • You do not have an internal storage for storing temp data.

With glue catalog you can view data in Athena, but it also has few limitations like cannot create table as select, cannot create view etc. You can use Glue data catalog in EMR to overcome limitations of Athena.

So, currently glue can be a replacement for persistent metadata store.

Related questions

Load CSV into Redshift, with header?

Is there an option to load a CSV into Redshift with a header? I see the documentation for CSV but it says nothing about a header. Ideally it could use the header to determine the columns to load.

Read More
Permission denied (publickey) when SSH Access to Amazon EC2 instance

I want to use my Amazon ec2 instance but faced the following error: Permission denied (publickey). I have created my key pair and downloaded .pem file. Given: chmod 600 pem file. Then, this command ssh -i /home/kashif/serverkey.pem ubuntu@…

Read More
Trying to SSH into an Amazon Ec2 instance - permission error

This is probably a stupidly simple question to some :) I've created a new linux instance on Amazon EC2, and as part of that downloaded the .pem file to allow me to SSH in. When I tried to ssh with: ssh …

Read More