Continuous Delivery

Background

The Workflows configuration for Continuous Delivery (CD) is based on Aspect's approach to delivering artifacts from Bazel Continuous Integration (CI) builds.

Before configuring this feature, read the Continuous Delivery Guide to understand how this approach models the steps.

Configuration

Which targets to deliver

By default, Workflows delivers all targets tagged with tags = ["deliverable"]. Add this tag to each deliverable target, or a Bazel macro might add this tag to all targets of that rule kind.

You can customize this behavior with a Bazel query expression that identifies deliverable targets. For example, to deliver all container_push targets:

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - deliverable: 'kind("container_push_ rule", //...)'

tip

Deliverable targets must be executable, as explained in the Continuous Delivery Guide.

Which changes to deliver

Workflows releases any deliverable target that differs from a previous delivery, as described in the Continuous Delivery Guide.

To make it easy to diagnose issues, Workflows uploads the list of targets as a "delivery manifest" found in delivery.mf in the artifacts uploaded by the CI pipeline.

First-time delivery

When migrating to Aspect Workflows Delivery from some other pipeline, your artifact storage is already populated with results from the legacy pipeline.

If this is not the case, after setting up Continuous Delivery it may be useful to perform a one-time "deliver everything" to populate the artifact storage.

To do this, run the query from the Which targets to deliver step to list all the deliverable targets. Then copy-paste that list into the delivery_targets in a "break the glass" manual delivery step.

This should cause Workflows to deliver all targets without regard for whether they are "changed".

Which branch(es) to deliver

Workflows delivers by default when running on a "release branch", which it considers to be either "^main$" and "^master$".

You can configure the condition property within a rule by setting branches. This property also supports a tags attribute that applies the same delivery behavior, but based on the git tag that triggered the build.

note

Aspect always treats the expression as a regular expression, and is automatically wrapped with ^ and $ if not included.

For example:

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - condition:
                branches:
                    - '^main$'
                    - '^hotfix-.*$'

You can supply multiple rules with different deliverable queries for more complex delivery conditions. You can set each rule to only deliver if the target has changed (as determined by the manifest), or to always deliver.

The following example shows two delivery rules.

The first rule defines a deliverable of all container_push rules, delivered if they have changed on main and hotfix branches. This uses the default delivery condition of only_on_change: true which only delivers targets that have changed within the rule.

The second rule defines a deliverable of a single target //services/bazel that is always delivered regardless of whether the inputs to the target have changed.

Control the behavior by setting the only_on_change property to false. Workflows targets will then only deliver targets on a branch that starts with release/.

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            # First rule
            - deliverable: 'kind("container_push_ rule", //...)'
              condition:
                branches:
                  - '^main$'
                  - '^hotfix-.*$'
            # Second rule
            - deliverable:
                - //services/bazel
              condition:
                only_on_change: false
                branches:
                  - 'release/.*'

When to deliver

By default, delivery is manual. A Release Engineer manually creates the delivery workflow step by logging into the CI system and triggering the workflow.

Set auto_deliver in the configuration to automatically run the delivery based on the delivery manifest:

tasks:
  - delivery:
      auto_deliver: true

In this case, any green build on a release branch triggers a delivery workflow step.

Stamping

Workflows relies on the Bazel stamping setup in the workspace. When an artifact is built with --stamp (or some other Bazel flags that include it, such as --config=release), this should create release artifacts that satisfy the deployment system.

The version used is user-controlled. Versioning Releases From a Monorepo describes more about choices in how to version artifacts.

By default, Workflows runs the delivery with bazel run --stamp. To use different stamping flags, set the stamp_flags property in the configuration. For example:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          stamp_flags:
              - --stamp
              - --workspace_status_command="${PWD}/workspace_status.sh"

Or by using a .bazelrc config flag such as:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          stamp_flags:
              - --config=release

Where the .bazelrc contains the following:

build:release --stamp
build:release --workspace_status_command="${PWD}/workspace_status.sh"

Salting

Workflows generates based on the contents of delivery artifacts. When using the only_on_change option this means you only get a delivery run for a given artifact, if it has changed. In certain situations however, it is desirable to view the artifact as "changed" given context external to the artifact itself. For example, it might be helpful for an artifact to be flagged as "changed" the first time Workflows sees it on both the main and release branches. To do this, you can salt the hash with the branch name. Salting is stable, so if the salting input and the artifact itself have not changed, the resulting hash is the same.

Enable salting with the following syntax:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          salt_envs:
              - SOME_ENVIRONMENT_VARIABLE

For example, if running on Buildkite, to take the branch name into account when creating delivery hashes, use the following:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          salt_envs:
              - BUILDKITE_BRANCH

Manual Delivery

Some examples of when to use manual delivery are the following:

• The main branch is red and a product team believes that the breakage is unrelated to their application and feels a strong pressure to ship.
• Shipping an application using a manual cadence.

To facilitate these cases and others, the release engineer can navigate to the CI webpage and trigger the delivery pipeline manually, providing special parameters:

• delivery_commit: What commit to check out and deliver.
• delivery_targets: Override the affected targets, and deliver this space-separated list of targets instead.
• workspace: The workspace that the delivery_targets live within.

Each CI system has a different process to access manual delivery:

Buildkite

From a branch that has an associated delivery rule, create a "New Build" and put "Deliver" or "Delivery" in the message field.

CircleCI

From your branch, select "Trigger Pipeline" and manually "Add Parameters" for each parameter listed above (as string parameter). Additionally, add a boolean parameter of perform_delivery: true.

GitHubActions

As part of Workflows, there is a delivery workflow named "delivery", "aspect-workflows-delivery" or similar which you can trigger manually with the above parameters provided.

info

Aspects plans a more auditable option in the future, where the release engineer can trigger the delivery with a GitHub comment on a commit.

Deployment

Deploying the artifacts is out-of-scope from Workflows, which assumes the existence of another system that promotes releases from one environment to another. For example, some clients use Harness.

API Doc

You can find the exhaustive list of attributes for the delivery task in the delivery documentation.