Continuous Delivery

Background

The Workflows configuration for CD is based on Aspect's approach to delivering artifacts from Bazel CI builds.

Before configuring this subsystem, read our Continuous Delivery Guide to understand how this approach models the steps.

Configuration

Which targets to deliver

By default, Workflows delivers all targets tagged with tags = ["deliverable"]. Add this tag to each deliverable target, or a macro might add this tag to all targets of that rule kind.

You can customize this behavior with a Bazel query expression that identifies deliverable targets. For example, to deliver all container_push targets:

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - deliverable: 'kind("container_push_ rule", //...)'

Deliverable targets must be executable, as explained in the Continuous Delivery Guide.

Which changes to deliver

Any deliverable target that differs from a previous delivery step is released, as described in the Continuous Delivery Guide.

To make it easy to diagnose issues, Workflows uploads the list of targets as a "delivery manifest" found in delivery.mf in the artifacts uploaded by the CI pipeline.

First-time delivery

When migrating to Aspect Workflows Delivery from some other pipeline, your artifact storage is already populated with results from the legacy pipeline.

If this is not the case, after setting up Continuous Delivery it may be useful to perform a one-time "deliver everything" to populate the artifact storage.

To do this, run the query from the Which targets to deliver step above to list all the deliverable targets. Then copy-paste that list into the delivery_targets in a "break the glass" manual delivery step.

This should cause all targets to be delivered without regard for whether they are "changed".

Which branch(es) to deliver

Workflows delivers by default when running on a "release branch", which it considers to be either "^main$" and "^master$".

You can configure the condition property within a rule by setting branches. This property also supports a tags attribute that applies the same delivery behaviour, but based on the git tag that triggered the build.

note

Aspect always treats the expression as a regular expression, and is automatically wrapped with ^ and $ if not included.

For example:

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - condition:
                branches:
                    - '^main$'
                    - '^hotfix-.*$'

You can also supply multiple rules with different deliverable queries for more complex delivery conditions. Each rule can also be set to only deliver if the target has changed (as determined by the manifest), or to always deliver.

The example below shows two delivery rules.

• The first rule defines a deliverable of all container_push rules, delivered if they have changed on main and hotfix branches. This uses the default delivery condition of only_on_change: true which only delivers targets that have changed within the rule.
• The second rule defines a deliverable of a single target //services/bazel that is always delivered regardless of whether the inputs to the target have changed. Control this by setting the only_on_change property to false. These targets will then only be delivered on a branch that starts with release/.

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - deliverable: 'kind("container_push_ rule", //...)'
              condition:
                branches:
                  - '^main$'
                  - '^hotfix-.*$'
            - deliverable:
                - //services/bazel
              condition:
                only_on_change: false
                branches:
                  - 'release/.*'

When to deliver

By default, delivery is manual. A Release Engineer manually creates the delivery workflow step by logging into the CI system and triggering the workflow.

Set auto_deliver in the configuration to automatically run the delivery based on the delivery manifest:

.aspect/workflows/config.yaml

tasks:
  - delivery:
      auto_deliver: true

In this case, any green build on a release branch triggers a delivery workflow step.

Stamping

Workflows relies on the Bazel stamping setup in the workspace. When an artifact is built with --stamp (or some other Bazel flags that include it, such as --config=release), this should create release artifacts that satisfy the deployment system.

The version used is user-controlled. Read our blog article https://blog.aspect.build/versioning-releases-from-a-monorepo for more about choices on how to version artifacts.

By default, Workflows runs the delivery with bazel run --stamp. To use different stamping flags, set the stamp_flags property in the configuration. For example:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          stamp_flags:
              - --stamp
              - --workspace_status_command="${PWD}/workspace_status.sh"

Or by using a .bazelrc config flag such as:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          stamp_flags:
              - --config=release

Where the .bazelrc contains the following:

build:release --stamp
build:release --workspace_status_command="${PWD}/workspace_status.sh"

Salting

Delivery hashes are generated based on the contents of delivery artifacts. When using the only_on_change option this means you only get a delivery run for a given artifact, if it has changed. In certain situations however, it is desirable to view the artifact as "changed" given context external to the artifact itself. For example, it might be helpful for an artifact to be flagged as "changed" the first time it is seen on both the main and release branches. To do this, you can salt the hash with the branch name. Salting is stable, so if the salting input and the artifact itself have not changed, the resulting hash is the same.

Salting can be enabled with the following syntax:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          salt_envs:
              - SOME_ENVIRONMENT_VARIABLE

So for example, if running on Buildkite, to take the branch name into account when creating delivery hashes, the following could be used:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          salt_envs:
              - BUILDKITE_BRANCH

Manual Delivery

Examples of when to use manual delivery:

• The main branch is red and a product team believes that the breakage is unrelated to their application and feels strong pressure to ship.
• Shipping an application using a manual cadence.

To facilitate these cases and others, the release engineer can navigate to the CI webpage and trigger the delivery pipeline manually, providing special parameters:

• delivery_commit: What commit to check out and deliver.
• delivery_targets: Override the affected targets, and deliver this space-separated list of targets instead.
• workspace: The workspace that the delivery_targets live within.

Each CI system has a slightly different process to access manual delivery:

Buildkite

From a branch that has an associated delivery rule, create a "New Build" and put "Deliver" or "Delivery" in the message field.

CircleCI

From your branch, select "Trigger Pipeline" and manually "Add Parameters" for each parameter listed above (as `string` parameter). Additionally, add a `boolean` parameter of `perform_delivery: true`.

GitHubActions

As part of Workflows, there will be a delivery workflow named "delivery", "aspect-workflows-delivery" or similar which can be triggered manually with the above parameters provided.

info

Aspects plans a more auditable option in the future, where the release engineer can trigger the delivery with a GitHub comment on a commit.

Deployment

Deploying the artifacts is out-of-scope from Workflows, which assumes the existence of another system that promotes releases from one environment to another. For example, some clients use https://harness.io/.

API Doc

You can find the exhaustive list of attributes for the delivery task in the delivery documentation.