Cloudwatch Log Alert - How to include error / exception / stack trace data in email notification
I just configured Cloudwatch logs on my ec2 instances and am loving it so far. I also set up alerts for certain keywords, like "ERROR". While the email alert seems to be working fine, I was wondering if there's a way to fine-tune the alert email to make it a little concise & informative. Specifically, I'm looking to
Get rid of all the boilerplate text in the alert email.
Include some information about the Error/Exception that triggered the alert. This could be something as simple as including the log statement that generated the alert.
Right now, the alert email looks like
You are receiving this email because your Amazon CloudWatch Alarm "App-Error-Alarm" in the US East - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0)." at "Tuesday 07 February, 2017 16:39:43 UTC".
View this alarm in the AWS Management Console: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Alarms&alarm=App-Error-Alarm
Alarm Details: - Name: App-Error-Alarm - Description: Errors in app.log - State Change: INSUFFICIENT_DATA -> ALARM - Reason for State Change: Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0). - Timestamp: Tuesday 07 February, 2017 16:39:43 UTC - AWS Account: <>
Threshold: - The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0 for 300 seconds.
Monitored Metric: - MetricNamespace: LogMetrics - MetricName: ERROR - Dimensions: - Period: 300 seconds - Statistic: Sum - Unit: not specified
State Change Actions: - OK: - ALARM: [arn:aws:sns:us-east-1:<>:support] - INSUFFICIENT_DATA:
I'd like it to something like
Alarm: App-Error-Alarm
Keyword: "ERROR"
Reason: ERROR 2017-02-07 07:31:47,375 [SimpleAsyncTaskExecutor-5] com.app.server.rest.Watcher: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
Its short, sweet and instantly tells me whether its something that needs my immediate attention. Can this be done without writing code as suggested here?
Answer
You have this problem because you configured an alarm and is meant for aggregated data, not for specific log record. You configure it for some metric (number of log records with ERROR keyword).
You can use log subscription instead and stream all log records matching a filter to a custom Lambda function. You can use it to send notifications to email or Slack.
To configure log streaming, go to Lambda in AWS console and create a new function from a blueprint named "cloudwatch-logs-process-data". It has a basic structure and is easy to customize to your needs.
