Skip to main content
Version: 5.11.x

workflows

Requirements

NameVersion
terraform>= 1.9.0
aws>= 4.67.0, < 6.0.0

Providers

NameVersion
aws>= 4.67.0, < 6.0.0

Modules

NameSourceVersion
alerting./alertingn/a
api./apin/a
bessie./services/bessien/a
bk./bkn/a
cci./ccin/a
configuration./utils/configuration_propertiesn/a
core./coren/a
dashboards./alerting/alarms/cloudwatch_dashboardsn/a
delivery./deliveryn/a
external_remote./remoten/a
gh_api_token_secret./utils/aws_secretn/a
gha./ghan/a
gl./gln/a
kilgore./services/kilgoren/a
logging./loggingn/a
remote./remoten/a
telemetry./telemetryn/a
warming./warmingn/a
webapp./webappn/a

Resources

NameType
aws_s3_object.runner_otel_configresource
aws_caller_identity.defaultdata source
aws_ecr_authorization_token.tokendata source
aws_region.defaultdata source

Inputs

NameDescriptionTypeDefaultRequired
account_idAccount ID of the AWS Account where CloudWatch alarms residestringnullno
aspect_artifacts_bucketS3 bucket where Aspect delivers workflows assetsstring"aspect-artifacts"no
bk_runner_groupsMapping of Buildkite runner group name to settings for that runner group
map(object({
# Common settings for all CI hosts
agent_idle_timeout_min = number
max_runners = number
min_runners = optional(number, 0)
min_free_runners = optional(number, 0)
policy_documents = optional(map(object({ json : string })), {})
policies = optional(map(string), {})
queue = string
resource_type = string
scale_out_factor = optional(number, 1)
scaling_polling_frequency = optional(number, 1)
reaper_sleep_minutes = optional(number, 10)
security_groups = optional(map(string), {})
warming = optional(bool, false)
warming_set = optional(string, "default")
exclude_oncall_alerts = optional(list(string), [])
tags = optional(map(string), {})
build_logs_bucket = optional(string, "BUCKET_PLACEHOLDER")
}))
{}no
cci_runner_groupsMapping of CircleCI runner group name to settings for that runner group
map(object({
# Common settings for all CI hosts
agent_idle_timeout_min = number
max_runners = number
min_runners = optional(number, 0)
min_free_runners = optional(number, 0)
policy_documents = optional(map(object({ json : string })), {})
policies = optional(map(string), {})
resource_type = string
scale_out_factor = optional(number, 1)
scaling_polling_frequency = optional(number, 1)
reaper_sleep_minutes = optional(number, 10)
security_groups = optional(map(string), {})
warming = optional(bool, false)
warming_set = optional(string, "default")
exclude_oncall_alerts = optional(list(string), [])
tags = optional(map(string), {})

# Settings specific to CircleCI
circleci_api_url = optional(string, "https://circleci.com")
circleci_runner_api_url = optional(string, "https://runner.circleci.com")
job_max_run_time_min = optional(number, 360)
}))
{}no
cost_allocation_tag(deprecated) The tag name used for cost taggingstring"CreatedBy"no
cost_allocation_tag_value(deprecated) The value of the cost tagstringnullno
create_security_groupsWhether to create security groups automatically for all resources.booltrueno
create_vpc_endpointsWhether to create VPC endpoints automatically.booltrueno
customer_idUnique, human-readable customer identifier provided by Aspectstringn/ayes
delivery_enabledIf delivery infrastructure is enabled for Aspect Workflowsbooltrueno
deployment_idUnique, human-readable deployment identifier provided by Aspectstring"default"no
experimental_webappn/a
object({
dns = object({
hosted_zone_id = optional(string, null)
hosted_zone_name = optional(string, null)
})
oidc = optional(object({
issuer = string
auth_endpoint = string
token_endpoint = string
user_info_endpoint = string
client_id = string
client_secret = string
session_timeout_seconds = optional(number, null)
}), null)
})
nullno
experimentsA map of experiment name (as given by Aspect) to its enabled statusmap(any){}no
external_remoteConfiguration for the externalized Bazel remote endpoint (cache and execution), specifically the ALB.
object({
debug_tools = optional(bool, false)
dns = object({
hosted_zone_id = optional(string, null)
hosted_zone_name = optional(string, null)
})
storage = object({
# Number of shards for the remote cache storage service
num_shards = optional(number, 3)
instance_type = optional(string)
instance_image_id = optional(string)
mirror = optional(bool)
})
frontend = optional(object({
cpu = optional(number, 1024)
memory = optional(number, 2048)
max_scaling = optional(number, 20)
min_scaling = optional(number)
}), {
cpu = 1024
memory = 2048
max_scaling = 20
})
downloader = optional(object({
frontend = optional(object({
cpu = optional(number, 1024)
memory = optional(number, 2048)
max_scaling = optional(number, 5)
min_scaling = optional(number, 1)
}), {
cpu = 1024
memory = 2048
max_scaling = 10
min_scaling = 1
})
}))
oidc = optional(object({
issuer = string
auth_endpoint = string
token_endpoint = string
user_info_endpoint = string
client_id = string
client_secret = string
session_timeout_seconds = optional(number, null)
}), null)
remote_execution = optional(object({
executors = map(object({
platform = optional(string)
image = string
additional_platform_properties = optional(map(string), {})
workers = optional(list(object({
scaling = optional(object({
minimum = optional(number)
maximum = optional(number)
fast = optional(object({
target = optional(number)
size = optional(number)
cooldown = optional(number)
}))
policy = optional(object({
target = optional(number, 100)
scale_in_cooldown = optional(number, 300)
scale_out_cooldown = optional(number, 60)
}), {
target = 100
scale_in_cooldown = 300
scale_out_cooldown = 60
})
rules = optional(map(object({
schedule = string
timezone = optional(string)
minimum = optional(number)
maximum = optional(number)
})))
}), {})
isolated_actions = optional(object({
cpu = optional(number, 1024)
memory = optional(number)
}))
network = optional(bool, true)
max_concurrency = optional(number)
max_download_concurrency = optional(number, 1000000)
max_upload_concurrency = optional(number, 1000000)
ec2 = optional(object({
instance_type = optional(string)
instance_image = optional(string)
docker_group = optional(string)
}))
ecs = optional(object({
architecture = optional(string)
cpu = optional(number, 1024)
memory = optional(number, 2048)
}))
runner = optional(object({
kill_processes = optional(bool)
set_tmp_dir_env_variable = optional(bool)
clean_tmp_directories = optional(list(string))
}))
docker_user = optional(string, null)
})), [{}])
}))
worker_cloudwatch_agent = optional(bool, true)
storage = optional(object({
num_shards = optional(number)
instance_type = optional(string)
instance_image_id = optional(string)
upstream_fallback = optional(bool, true)
mirror = optional(bool)
}))
}))
})
nullno
gha_runner_groupsMapping of GitHub Actions runner group name to settings for that runner group
map(object({
# Common settings for all CI hosts
agent_idle_timeout_min = number
max_runners = number
min_runners = optional(number, 0)
min_free_runners = optional(number, 0)
policy_documents = optional(map(object({ json : string })), {})
policies = optional(map(string), {})
queue = string
resource_type = string
scale_out_factor = optional(number, 1)
scaling_polling_frequency = optional(number, 1)
reaper_sleep_minutes = optional(number, 10)
security_groups = optional(map(string), {})
warming = optional(bool, false)
warming_set = optional(string, "default")
exclude_oncall_alerts = optional(list(string), [])
tags = optional(map(string), {})

# Settings specific to GitHub Actions
gh_repo = string
gha_workflow_ids = optional(list(string), [])
}))
{}no
gl_runner_groupsMapping of GitLab runner group name to settings for that runner group
map(object({
# Common settings for all CI hosts
agent_idle_timeout_min = number
max_runners = number
min_runners = optional(number, 0)
min_free_runners = optional(number, 0)
policy_documents = optional(map(object({ json : string })), {})
policies = optional(map(string), {})
queue = string
resource_type = string
scale_out_factor = optional(number, 1)
scaling_polling_frequency = optional(number, 1)
reaper_sleep_minutes = optional(number, 10)
security_groups = optional(map(string), {})
warming = optional(bool, false)
warming_set = optional(string, "default")
exclude_oncall_alerts = optional(list(string), [])
tags = optional(map(string), {})

# Settings specific to GitLab
gitlab_url = optional(string, "https://gitlab.com")
project_id = string
}))
{}no
hosts####################################### CI host configuration options #list(string)n/ayes
internal_use_onlyFor Aspect internal development use only
object({
allow_bucket_force_destroy = optional(bool, false) # Whether to set force_destroy on buckets
disable_db_deletion_protection = optional(bool, false) # Whether to disable deletion protection on databases
})
{}no
partitionThe partition to configure services in, if not commercialstringnullno
regionThe default region to setup services instringnullno
remoteConfiguration for the Bazel remote endpoint (cache and execution), specifically the ALB.
object({
debug_tools = optional(bool, false)
storage = object({
# Number of shards for the remote cache storage service
num_shards = optional(number, 3)
instance_type = optional(string)
instance_image_id = optional(string)
mirror = optional(bool)
})
frontend = optional(object({
cpu = optional(number, 1024)
memory = optional(number, 2048)
max_scaling = optional(number, 20)
min_scaling = optional(number)
}), {
cpu = 1024
memory = 2048
max_scaling = 20
})
downloader = optional(object({
frontend = optional(object({
cpu = optional(number, 1024)
memory = optional(number, 2048)
max_scaling = optional(number, 5)
min_scaling = optional(number, 1)
}), {
cpu = 1024
memory = 2048
max_scaling = 10
min_scaling = 1
})
}))
remote_execution = optional(object({
executors = map(object({
platform = optional(string)
image = string
additional_platform_properties = optional(map(string), {})
workers = optional(list(object({
scaling = optional(object({
minimum = optional(number)
maximum = optional(number)
fast = optional(object({
target = optional(number)
size = optional(number)
cooldown = optional(number)
}))
policy = optional(object({
target = optional(number, 100)
scale_in_cooldown = optional(number, 300)
scale_out_cooldown = optional(number, 60)
}), {
target = 100
scale_in_cooldown = 300
scale_out_cooldown = 60
})
rules = optional(map(object({
schedule = string
timezone = optional(string)
minimum = optional(number)
maximum = optional(number)
})))
}), {})
isolated_actions = optional(object({
cpu = optional(number, 1024)
memory = optional(number)
}))
network = optional(bool, true)
max_concurrency = optional(number)
max_download_concurrency = optional(number, 1000000)
max_upload_concurrency = optional(number, 1000000)
ec2 = optional(object({
instance_type = optional(string)
instance_image = optional(string)
docker_group = optional(string)
}))
ecs = optional(object({
architecture = optional(string)
cpu = optional(number, 1024)
memory = optional(number, 2048)
}))
runner = optional(object({
kill_processes = optional(bool)
set_tmp_dir_env_variable = optional(bool)
clean_tmp_directories = optional(list(string))
}))
docker_user = optional(string, null)
})), [{}])
}))
worker_cloudwatch_agent = optional(bool, true)
storage = optional(object({
num_shards = optional(number)
instance_type = optional(string)
instance_image_id = optional(string)
upstream_fallback = optional(bool, true)
mirror = optional(bool)
}))
}))
})
n/ayes
repository_urlsThe repository URLs for the Docker images used by this module. Meant to be used in concert with the ecr_images submodule.map(string)
{
"adot_exporter": "public.ecr.aws/aws-observability/aws-otel-collector",
"alert_manager": "quay.io/prometheus/alertmanager:v0.27.0",
"aws_cli": "public.ecr.aws/aws-cli/aws-cli",
"bash": "public.ecr.aws/docker/library/bash",
"bb_browser": "ghcr.io/buildbarn/bb-browser:20240613T055327Z-f0fbe96",
"bb_remote_asset": "ghcr.io/buildbarn/bb-remote-asset:20241014t200011z-5a41232",
"bb_replicator": "ghcr.io/buildbarn/bb-replicator:20241120T153311Z-7c63b92",
"bb_runner_installer": "ghcr.io/buildbarn/bb-runner-installer:20240716t044555z-9850e82",
"bb_scheduler": "ghcr.io/buildbarn/bb-scheduler:20240716t044555z-9850e82",
"bb_storage": "ghcr.io/buildbarn/bb-storage:20241120T153311Z-7c63b92",
"bb_worker": "ghcr.io/buildbarn/bb-worker:20240716t044555z-9850e82",
"busybox": "public.ecr.aws/docker/library/busybox",
"curl_jq": "registry.gitlab.com/gitlab-ci-utils/curl-jq:3.0.0",
"otel_collector_contrib": "otel/opentelemetry-collector-contrib:0.111.0",
"prometheus": "quay.io/prometheus/prometheus:v2.52.0"
}
no
resource_typesMapping of resource types name to settings for that type
map(object({
# The ID of the AMI to use for this resource
image_id = string

# A list of instance types that are acceptable in the ASG
instance_types = list(string)

# The size of the root EBS volume in GB
root_volume_size_gb = optional(number, 64)

# Tags to apply to this resource
tags = optional(map(string), {})

# Defines if spot instances should be used for this resource
use_spot = optional(bool, false)

# When using spot instances, allows further customization over the spot vs on-demand allocation
instance_policy = optional(object({
on_demand_base_capacity = optional(number, 0)
on_demand_percentage_above_base_capacity = optional(number, 0)
spot_allocation_strategy = optional(string, "price-capacity-optimized")
spot_max_price = optional(string, "")
spot_instance_pools = optional(number, 2)
}), {})
}))
{}no
scaling_function_memory_mbThe amount of memory to assign to scaling functions in MBnumber512no
security_group_idsOptional security group ID substitutions for Workflows resources.map(any){}no
supportSet of properties that allow Aspect to provide oncall support for Workflows
object({
# If true, alerts generated by Workflows will be reported back to Aspect.
# Depending on the severity of the alert, this may result in an oncall
# engineer being paged depending on the level of support included with this
# Workflows install.
alert_aspect = optional(bool, true)

# A set of secret IDs that can be overriden if required.
secrets = optional(object({
# Override the secret ID used for fetching the PagerDuty routing key from Aspects AWS account.
aspect_pagerduty_routing_key_id = optional(string)

# Override the secret ID used for fetching the Slack token from Aspects AWS account.
aspect_slack_token_secret_id = optional(string)

# Override the KMS ID used to decrypt the support secrets from Aspect.
aspect_support_secret_kms_id = optional(string)
}), {})

# Role ARN that allows support level access for Aspect.
support_role_name = optional(string, null)

# Role ARN that allows extended support access for Aspect.
# This role will have write access to various areas Workflows infrastructure,
# however it can only be assumed by a subset of Aspect oncall engineers.
operator_role_name = optional(string, null)

# Add policies that allow access to CI infrastructure instances via SSM
enable_ssm_access = optional(bool, false)
})
n/ayes
tagsTags to add to every resource Aspect Workflows createsmap(string){}no
telemetryConfiguration options for Workflows telemetry
object({
# Configuration for where Workflows telemetry data gets exported
destinations = optional(object({
# Which exporters to set up.
honeycomb = optional(object({
# Honeycome dataset to set for exports to this destination.
dataset = optional(string)
# Honeycomb Team secret reference, used for authentication.
team_secret = object({
id = string
arn = string
})
}))
datadog = optional(object({
# Datadog agent ingest site.
site = string
# Datadog API key secret reference, used for authentication.
key_secret = object({
id = string
arn = string
})
}))
generic_otlp = optional(object({
# Endpoint where to export telemetry to.
endpoint = string
}))
}), {})
})
{}no
vpc_idID of the VPC in which to deploystringn/ayes
vpc_subnetsList of subnet IDs to use for VM infrastructurelist(string)n/ayes
vpc_subnets_publicList of subnet IDs to use for public facing VM infrastructurelist(string)[]no
warming_setsMapping of warming set to settings for that setmap(object({})){}no

Outputs

NameDescription
alarms_sns_topic_arnSNS topic ARN that provides notifications of all Workflows alarms
api_secret_idSecret for the Aspect API
bk_agent_token_secret_idsMapping of Buildkite runner name to Buildkite agent token secret ID
bk_api_token_secret_idsMapping of Buildkite runner name to Buildkite API token secret ID
bk_git_ssh_key_secret_idsMapping of Buildkite runner name to ssh key secret ID
buildkite_agent_hooks_bucketsName of the bucket for storing custom Buildkite Agent hooks
cost_allocation_tag(deprecated) Name of the cost allocation tag to use
cost_allocation_tag_value(deprecated) The value of the cost allocation tag
external_remote_cache_endpointThe endpoint of the Internet-facing remote cache, if enabled.
external_remote_certificateThe ACM certificate in use by the external remote cluster, if present.
external_remote_load_balancerThe load balancer configuration for the external remote cluster, if present.
gha_lambda_webhook_secret_idsMapping of GitHub Actions runner name and repo key to the ID's of the secrets containing the webhook token that the scaling lambda will use to verify the event came from GitHub
gha_secret_idsMapping of GitHub Actions runner name and repo key to secret ID
github_token_secret_idSecret ID for a GitHub token used for making readonly calls to GitHub during a build
gl_secret_idsMapping of Gitlab runner name and repo key to secret ID
internal_remote_cache_certificateThe CA certificate for the VPC-facing remote cache.
internal_remote_cache_endpointThe endpoint of the VPC-facing remote cache.
managed_prometheus_endpointThe endpoint of the Amazon Managed Prometheus (AMP) endpoint
runner_secret_idsMapping of CircleCI runner name to secret ID
security_group_rulesSecurity group rules for the Workflows module
tagsTags that will be added on all resources
warming_management_policiesn/a