PagerDuty Integration

Workflows provides direct integration with PagerDuty for all CloudWatch alarms for the deployed infrastructure.

During initial setup, Aspect will provide a PagerDuty integration key, which will allow Aspect to be oncall for the Workflows install and respond to issues.

Initial Setup

Using the provided PagerDuty integration key, set the support.pagerduty_integration_key to the value provided.

main.tf

module "aspect_workflows" {
 support = {
    pagerduty_integration_key = "123abc"
 }
}

Excluding Runner Groups

It's possible to exclude certain alarms for the various configured runner groups from being monitored by PagerDuty, and therefore Aspect's on-call Workflows engineers. This can be useful if a particular runner group is used for canary runners, or running experiments.

note

Excluded alarms still appear as "in alarm" in the CloudWatch dashboard, but they do not notify PagerDuty of an issue.

To exclude an alarm, set the exclude_oncall_alerts attribute on the runner group:

main.tf

default = {
   exclude_oncall_alerts = ["Runner Alarms"]
}

Possible values for the exclusion list:

• Runner Alarms: Excludes alarms generated by runners, such as from bootstrap.

Subscribing to PagerDuty Alerts

It may be desired to subscribe aditional PagerDuty integrations to the same alerts that Aspect recieve, or to send the alarm data to another service. To send alarm data to another service, setup additional subscriptions on the exposed SNS topic from the Workflows module, via the alarms_sns_topic_arn output, replacing the PAGERDUTY_INTEGRATION_KEY value with the appropiate integration key for your service.

main.tf

resource "aws_sns_topic_subscription" "workflows_pagerduty" {
  topic_arn = module.aspect_workflows_aws.alarms_sns_topic_arn
  protocol  = "https"
  endpoint  = "https://events.pagerduty.com/integration/PAGERDUTY_INTEGRATION_KEY/enqueue"
  raw_message_delivery = false
}

For more information on the required PagerDuty setup, see Amazon CloudWatch Integration Guide from PagerDuty.