Is there any way to trigger a AWS Lambda function at the end of an AWS Glue job?

Question

Currently I'm using an AWS Glue job to load data into RedShift, but after that load I need to run some data cleansing tasks probably using an AWS Lambda function. Is there any way to trigger a Lambda function at the end of a Glue job? Lambda functions can be triggered using SNS messages, but I couldn't find a way to send an SNS at the end of the Glue job.

Answer 1

@oreoluwa is right, this can be done using Cloudwatch Events.

From the Cloudwatch dashboard:

• Click on 'Rules' from the left menu
• For 'Event Source', choose 'Event Pattern' and in 'Service Name' choose 'Glue'
• For 'Event Type' choose 'Glue Job State Change'
• On the right side of the page, in the 'Targets' section, click 'Add Target' -> 'Lambda Function' and then choose your function.

The event you'll get in Lambda will be of the format:

{
    'version': '0',
    'id': 'a9bc90be-xx00-03e0-9bc5-a0a0a0a0a0a0',
    'detail-type': 'GlueJobStateChange',
    'source': 'aws.glue',
    'account': 'xxxxxxxxxx',
    'time': '2018-05-10T16: 17: 03Z',
    'region': 'us-east-2',
    'resources': [],
    'detail': {
        'jobName': 'xxxx_myjobname_yyyy',
        'severity': 'INFO',
        'state': 'SUCCEEDED',
        'jobRunId': 'jr_565465465446788dfdsdf546545454654546546465454654',
        'message': 'Jobrunsucceeded'
    }
}