Continuous Delivery

Background

The term "CD" is ambiguous. Some engineers use it to mean "Continuous Deployment", in which changes are automatically released, e.g. into a "dev" environment.

Aspect recommends that Continuous Delivery is modelled as the step of the pipeline where built artifacts are uploaded from the build machine to a well-known repository location. This could be a container image registry like Docker Hub, a blob store like AWS S3, or even a database.

This makes a clear separation of responsibilities between CI/CD and Deployment:

• The CI/CD pipeline should upload only the artifacts that are:
  • configured with BUILD.bazel files. Product engineers don't need to worry about setting up CD
  • green: it can prove that all relevant tests are passing
  • changed from a previous build
• The deployment system
  • locates and "promotes" artifacts to the next environment, such as "dev", "staging", or "prod".

What is "deliverable"

A deliverable artifact is one that contains both the binary or files to push, as well as the "pushing" logic that knows how to perform the upload. It might also send a message to the deployment system to trigger an auto-deployment of the new artifact.

In Bazel terms, this means a deliverable should be an executable program that can be bazel run.

Some examples of deliverable targets:

• docker push: The rules_oci oci_push or rules_docker container_push rules can both be executed with bazel run to push a Docker image to a registry like Docker Hub.

• git push: Sometimes artifacts belong in a separate code repository. For example, an SDK built from the API definitions in a monorepo needs to be published. See an example git_push executable in this repository.

• s3 sync: See s3_sync.

Which targets to deliver

A bazel query expression is the most convenient way to locate deliverable targets. Users may choose a tagging scheme for their workspace (i.e. "all targets with tags = ['artifact']"), or deliver well-known rule kinds (i.e. oci_push), or both.

Which changes to deliver

To optimize time and money, it's best to deliver only "changed" targets. This avoids wasted time and resources uploading the same artifact repeatedly. It also means that release engineers won't have to sort through a massive list of duplicates when choosing a release.

There are two approaches for choosing "changed" targets:

1. Predict the changes based on a version control delta. For example you could git diff between the hash being delivered and the "prior successful" delivery hash, then use a tool like bazel-diff or target-determinator to produce a list of targets that might be affected by those changes.
2. Determine empirically based on what is actually different. This requires determinism, so it must only use unstamped build results (--nostamp). In most cases a green CI run just completed, so these unstamped outputs are easily available.

Aspect recommends following the second approach because the first has some downsides:

• bazel-diff is incorrect and will sometimes miss affected targets, so they aren't delivered.
• target-determinator is slow and may hurt the "service level indicator" of time between pushing a hotfix and being able to release that fix.
• It will over-deliver, because sometimes a source change doesn't actually factor into whether the release binary changes, such as for a comment-only change.

The rest of this section provides more details about the second approach (determine empirically).

To determine whether Workflows should deliver that executable target on a particular commit, it is first hashed using the aspect outputs command with a special pseudo-mnemonic "ExecutableHash", for example:

$ aspect outputs 'attr("tags", "\bdeliverable\b", //...)' ExecutableHash
//cli:release h1:cj8OUC3l3fIr3Zxnffk6y7gukLOJmiWRCAQoqadg66Y=
//workflows/rosetta:release h1:kjHVajw+Nta2kh3Epcd32DkZxTE1NHA8b5N7hCNFNSM=

You need a lookup database to store previously delivered hashes. If the hash value matches one previously seen, then skip delivery of that target.

Debugging changes to deliver

You can run the aspect outputs command locally to understand whether a given change to a source file results in a new executable. Sometimes the result may be surprising. For example, if a comment in a .go source file is changed, the compiler produces the same .a file as a result, so the hash seen on the uploader executable is unchanged.

Another scenario that won't change the executable is when some production configuration is changed. For example, you may use Helm charts to deploy to Kubernetes. If these aren't included inside the image, then changes to these files won't cause a new delivery.

Perform the delivery

Run each deliverable target with stamping enabled. You can do this in a script which reads the targets from a manifest file such as cat $delivery_manifest | xargs -N1 bazel run --stamp.

The Workflows configuration for Continuous Delivery (CD) is based on Aspect's approach to delivering artifacts from Bazel Continuous Integration (CI) builds.

Before configuring this feature, read the [Continuous Delivery Guide] to understand how this approach models the steps.

Configuration

Having described the basic approach to continuous delivery, the rest of this document describes the configuration in Aspect Workflows.

Which targets to deliver

By default, Workflows delivers all targets tagged with tags = ["deliverable"]. Add this tag to each deliverable target, or a Bazel macro might add this tag to all targets of that rule kind.

You can customize this behavior with a Bazel query expression that identifies deliverable targets. For example, to deliver all container_push targets:

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - deliverable: 'kind("container_push_ rule", //...)'

tip

Deliverable targets must be executable, as explained in the [Continuous Delivery Guide].

Which changes to deliver

Workflows releases any deliverable target that differs from a previous delivery, as described in the [Continuous Delivery Guide].

To make it easy to diagnose issues, Workflows uploads the list of targets as a "delivery manifest" found in delivery.mf in the artifacts uploaded by the CI pipeline.

First-time delivery

When migrating to Aspect Workflows Delivery from some other pipeline, your artifact storage is already populated with results from the legacy pipeline.

If this is not the case, after setting up Continuous Delivery it may be useful to perform a one-time "deliver everything" to populate the artifact storage.

To do this, run the query from the Which targets to deliver step to list all the deliverable targets. Then copy-paste that list into the delivery_targets in a "break the glass" manual delivery step.

This should cause Workflows to deliver all targets without regard for whether they are "changed".

Which branch(es) to deliver

Workflows delivers by default when running on a "release branch", which it considers to be either "^main$" and "^master$".

You can configure the condition property within a rule by setting branches. This property also supports a tags attribute that applies the same delivery behavior, but based on the git tag that triggered the build.

note

Aspect always treats the expression as a regular expression, and is automatically wrapped with ^ and $ if not included.

For example:

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            - condition:
                branches:
                    - '^main$'
                    - '^hotfix-.*$'

You can supply multiple rules with different deliverable queries for more complex delivery conditions. You can set each rule to only deliver if the target has changed (as determined by the manifest), or to always deliver.

The following example shows two delivery rules.

The first rule defines a deliverable of all container_push rules, delivered if they have changed on main and hotfix branches. This uses the default delivery condition of only_on_change: true which only delivers targets that have changed within the rule.

The second rule defines a deliverable of a single target //services/bazel that is always delivered regardless of whether the inputs to the target have changed.

Control the behavior by setting the only_on_change property to false. Workflows targets will then only deliver targets on a branch that starts with release/.

.aspect/workflows/config.yaml

tasks:
    - delivery:
        rules:
            # First rule
            - deliverable: 'kind("container_push_ rule", //...)'
              condition:
                branches:
                  - '^main$'
                  - '^hotfix-.*$'
            # Second rule
            - deliverable:
                - //services/bazel
              condition:
                only_on_change: false
                branches:
                  - 'release/.*'

When to deliver

By default, delivery is manual. A Release Engineer manually creates the delivery workflow step by logging into the CI system and triggering the workflow.

Set auto_deliver in the configuration to automatically run the delivery based on the delivery manifest:

tasks:
  - delivery:
      auto_deliver: true

In this case, any green build on a release branch triggers a delivery workflow step.

Stamping

Workflows relies on the Bazel stamping setup in the workspace. When an artifact is built with --stamp (or some other Bazel flags that include it, such as --config=release), this should create release artifacts that satisfy the deployment system.

The version used is user-controlled. Versioning Releases From a Monorepo describes more about choices in how to version artifacts.

By default, Workflows runs the delivery with bazel run --stamp. To use different stamping flags, set the stamp_flags property in the configuration. For example:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          stamp_flags:
              - --stamp
              - --workspace_status_command="${PWD}/workspace_status.sh"

Or by using a .bazelrc config flag such as:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          stamp_flags:
              - --config=release

Where the .bazelrc contains the following:

build:release --stamp
build:release --workspace_status_command="${PWD}/workspace_status.sh"

Salting

Workflows generates based on the contents of delivery artifacts. When using the only_on_change option this means you only get a delivery run for a given artifact, if it has changed. In certain situations however, it is desirable to view the artifact as "changed" given context external to the artifact itself. For example, it might be helpful for an artifact to be flagged as "changed" the first time Workflows sees it on both the main and release branches. To do this, you can salt the hash with the branch name. Salting is stable, so if the salting input and the artifact itself have not changed, the resulting hash is the same.

Enable salting with the following syntax:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          salt_envs:
              - SOME_ENVIRONMENT_VARIABLE

So for example, if running on Buildkite, to take the branch name into account when creating delivery hashes, the following could be used:

.aspect/workflows/config.yaml

tasks:
    - delivery:
          salt_envs:
              - BUILDKITE_BRANCH

Manual Delivery

Some examples of when to use manual delivery:

• The main branch is red and a product team believes that the breakage is unrelated to their application and feels strong pressure to ship.
• Shipping an application using a manual cadence.

To facilitate these cases and others, the release engineer can navigate to the CI webpage and trigger the delivery pipeline manually, providing special parameters:

• delivery_commit: What commit to check out and deliver.
• delivery_targets: Override the affected targets, and deliver this space-separated list of targets instead.
• workspace: The workspace that the delivery_targets live within.

Each CI system has a different process to create manual delivery:

Buildkite

From a branch that has an associated delivery rule, create a New Build and put "Deliver" or "Delivery" in the message field.

CircleCI

From your branch, select Trigger Pipeline and manually "Add Parameters" for each parameter listed above (as string parameter). Additionally, add a boolean parameter of perform_delivery: true.

GitHubActions

As part of Workflows, there is a delivery workflow named "delivery", "aspect-workflows-delivery" or similar which you can trigger manually with the parameters above provided.

info

Aspects plans a more auditable option in the future, where the release engineer can trigger the delivery with a GitHub comment on a commit.

Deployment

Deploying the artifacts is out-of-scope from Workflows, which assumes the existence of another system that promotes releases from one environment to another. For example, some clients use https://harness.io/.

API Doc

You can find the exhaustive list of attributes for the delivery task in the delivery documentation.