Cloudwatch failedinvocation error no logs available

Question

I have set up a Cloudwatch rule event where an ECS task definition is started when a previous task definition is completed.

I can see the event triggers the task definition however it fails.

The only visibility of this failure is in the rule metrics, where I see the metric failedinnvocations.

Question, are there any logs to see why the trigger failed?

I can manually set up the rule via the management console and everything works fine.

The error occurs when I set up the rule via a cloudformation template.

I have compared the two rules and both are identical, except the role. However, both roles have the same permissions.

Answer 1

This stumped us for ages, the main issue is the role problem Nathan B mentions but something else that tripped us up is that Scheduled Containers won't work in awsvpc mode (and by extension Fargate). Here's a sample CloudFormation template:

---
AWSTemplateFormatVersion: 2010-09-09
Description: Fee Recon infrastructure

Parameters:

  ClusterArn:
    Type: String
    Description: The Arn of the ECS Cluster to run the scheduled container on

Resources:

  TaskRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      AssumeRolePolicyDocument:
        Statement:
          - Action:
              - sts:AssumeRole
            Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
        Version: 2012-10-17
      Policies:
       - PolicyName: TaskPolicy
         PolicyDocument:
           Version: 2012-10-17
           Statement:
             - Effect: Allow
               Action:
                 - 'ses:SendEmail'
                 - 'ses:SendRawEmail'
               Resource: '*'

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      TaskRoleArn: !Ref TaskRole
      ContainerDefinitions:
        - Name: !Sub my-container
          Essential: true
          Image: !Sub <aws-account-no>.dkr.ecr.eu-west-1.amazonaws.com/mycontainer
          Memory: 2048
          Cpu: 1024

  CloudWatchEventECSRole:
   Type: AWS::IAM::Role
   Properties:
     AssumeRolePolicyDocument:
       Version: 2012-10-17
       Statement:
         - Effect: Allow
           Principal:
             Service:
               - events.amazonaws.com
           Action:
             - sts:AssumeRole
     Path: /
     Policies:
       - PolicyName: CloudwatchEventsInvokeECSRunTask
         PolicyDocument:
           Version: 2012-10-17
           Statement:
             - Effect: Allow
               Action: 'ecs:RunTask'
               Resource: !Ref TaskDefinition

  TaskSchedule:
    Type: AWS::Events::Rule
    Properties:
      Description: Runs every 10 minutes
      Name: ScheduledTask
      ScheduleExpression: cron(0/10 * * * ? *)
      State: ENABLED
      Targets:
        - Id: ScheduledEcsTask
          RoleArn: !GetAtt CloudWatchEventECSRole.Arn
          EcsParameters:
            TaskDefinitionArn: !Ref TaskDefinition
            TaskCount: 1
          Arn: !Ref ClusterArn

Note: I've added the ClusterArn as a parameter to the script but of course it's better to do this with a CloudFormation ImportValue statement.

There are two roles you need to care about, the first is the role (TaskRole) for the task itself: in this example the container just sends an email using SES so it has the necessary permissions. The second role (CloudWatchEventECSRole) is the one that makes it all work, note that in its Policies array the principle is events.amazonaws.com and the resource is the ECS task defined in the template.