Installing on Buildkite

Configure a Pipeline

Create a new Pipeline for your Workflows build or choose an existing one and configure it to your preference. If you use GitHub, follow these steps to integrate your repository with Buildkite.

Click on Edit Steps and paste the following:

steps:
    - key: aspect-workflows-setup
      label: ':aspect-build: Setup Aspect Workflows'
      commands:
          - 'rosetta steps | buildkite-agent pipeline upload'
      agents:
          queue: <QUEUE>

Replace <QUEUE> with the name of the runner group's queue in your terraform module declaration. We recommend using aspect-default which is the default queue Aspect Workflows queue name. If using a non-default queue name, you'll also need to specify the queue name in your .aspect/workflows/config.yaml file with queue: <QUEUE>.

tip

You can use the Buildkite provider for Terraform to set up your pipeline programmatically.

Populate secrets

The Workflows terraform module creates two secrets to populate per runner group.

1. Buildkite agent token. The agent token allows our runners to connect to Buildkite. Under your organization, click on the Agents Tab then click Reveal Agent Token. Copy and paste this token into a secret named like aw_bk_agent_token__<RunnerGroup>_XXXXXXXXXXXXXXXX.

Alternatively, the value can be supplied using Terraform. We expose the AWS Secrets Manager Secret Id via an output from the Workflows terraform module. This ID is named bk_agent_token_secret_ids["runner group name"] where the "runner group name" matches the bk_runner_groups input parameter.

For example, if main.tf contains

main.tf

  bk_runner_groups = {
    default = {
      ...
    }
  }

Then the secret can be configured with:

resource "aws_secretsmanager_secret_version" "this" {
  secret_id     = module.aspect-workflows.bk_agent_token_secret_ids["default"]
  secret_string = "my-value"
}

The secret_string value should be supplied using whatever mechanism you already use for managing secrets.

2. Buildkite API access token. The API token allows us to monitor the agents that we have created for anomalies. Under Personal Settings go to the API Access Tokens tab and click the New API Access Token button. Give it a name such as "Aspect Workflows API Token" and select your organization from the dropdown. Under REST API Scopes we need the Read Agents permission. Once created, copy and paste this token into a secret named like aw_bk_api_token__<RunnerGroup>_XXXXXXXXXXXXXXXX.

Alternatively, the value can be supplied using Terraform. We expose the AWS Secrets Manager Secret Id via an output from the Workflows terraform module. This ID is named bk_api_token_secret_ids["runner group name"] where the "runner group name" matches the bk_runner_groups input parameter.

For example, if main.tf contains

main.tf

  bk_runner_groups = {
    default = {
      ...
    }
  }

Then the secret can be configured with:

resource "aws_secretsmanager_secret_version" "this" {
  secret_id     = module.aspect-workflows.bk_api_token_secret_ids["default"]
  secret_string = "my-value"
}

The secret_string value should be supplied using whatever mechanism you already use for managing secrets.

3. Git SSH key. If you use a private repository, set up an SSH key that Workflows runners can use to check out your repository. Under your pipeline's GitHub settings, select the SSH radio button. Then, upload the private key to a secret with a name like aw_bk_git_ssh_key__<RunnerGroup>-XXXXXXXX. If you use GitHub, we recommend using a Deploy Key with read-only access.

Alternatively, the value can be supplied using Terraform. We expose the AWS Secrets Manager Secret Id via an output from the Workflows terraform module. This ID is named bk_git_ssh_key_secret_ids["runner group name"] where the "runner group name" matches the bk_runner_groups input parameter.

For example, if main.tf contains

main.tf

  bk_runner_groups = {
    default = {
      ...
    }
  }

Then the secret can be configured with:

resource "aws_secretsmanager_secret_version" "this" {
  secret_id     = module.aspect-workflows.bk_git_ssh_key_secret_ids["default"]
  secret_string = "my-value"
}

The secret_string value should be supplied using whatever mechanism you already use for managing secrets.

tip

The Workflows module exposes the names of the secrets it creates as an outputs. If you use a tool like Vault, you can wire in secret values automatically via Terraform.

4. GitHub API Token.

A number of Workflows features require read-only access to the GitHub API.

For example, if the "Branch Freshness" strategy is set to rebase, it requires a GitHub token to make a GET request to https://api.github.com/repos/{owner}/{repo}/pulls/{pull_number}. The token must therefore be granted read permission to Pull Requests, scoped to any repositories that are tested by Workflows.

This secret can be supplied in Terraform, similarly to the tokens described above.

resource "aws_secretsmanager_secret_version" "gh_api_token" {
  secret_id     = module.aspect-workflows.github_token_secret_id
  secret_string = "my-github-token"
}

Configure workflows

Add a Workflows configuration file to your repository.

(Optional) Custom Buildkite agent hooks

The Buildkite agent allows you to declare custom hooks which it runs at various points throughout the lifecycle of a build.

AWS

To supply your own hook, add the hook to the S3 bucket named aw_buildkite_agent-hooks_XXXXXXXXX under the global/ folder. Runners will find the hooks and install them for the buildkite agent to use.

GCP

To supply your own hook, add the hook to the GCS bucket named aw-buildkite-agent-hooks-XXXXXXXXX under the global/ folder. Runners will find the hooks and install them for the Buildkite agent to use.

note

Hooks under the global/ folder are shared across runner groups. In the future, we will support per-runner-group hooks.

Not recommended: You can alternatively add hooks to your runner image under the directory /opt/buildkite-agent-hooks. This approach requires rebuilding your runner image each time you change the hooks. Hooks found in the bucket will override hooks found in this directory.

info

At the moment, we do not support pre-checkout and post-checkout hooks. These hooks will be ignored if added.

tip

You may wish to alternatively try Aspect Workflow's native hooks solution, which allows you to execute code before and after tasks.