Modeling Continuous Delivery under Bazel

The term "CD" is ambiguous. Some engineers use it to mean "Continuous Deployment", in which changes are automatically released, e.g. into a "dev" environment.

Aspect recommends that Continuous Delivery is modelled as the step of the pipeline where built artifacts are uploaded from the build machine to a well-known repository location. This could be a container image registry like Docker Hub, a blob store like AWS S3, or even a database.

This makes a clear separation of responsibilities between CI/CD and Deployment:

• The CI/CD pipeline should upload only the artifacts that are:
  • configured with BUILD.bazel files. Product engineers don't need to worry about setting up CD
  • green: it can prove that all relevant tests are passing
  • changed from a previous build
• The deployment system
  • locates and "promotes" artifacts to the next environment, such as "dev", "staging", or "prod".

Build vs. Buy

The recommendations in this guide can be applied in two ways:

• DevInfra teams may wish to implement and operate a custom system for their organization, or
• Use Aspect Workflows, which provides this feature out-of-the-box.

What is "deliverable"

A deliverable artifact is one that contains both the binary or files to push, as well as the "pushing" logic that knows how to perform the upload. It might also send a message to the deployment system to trigger an auto-deployment of the new artifact.

In Bazel terms, this means a deliverable should be an executable program that can be bazel run.

Container Images

The rules_oci oci_push or rules_docker container_push rules can both be executed with bazel run to push a Docker image to a registry like Docker Hub.

Therefore these rules are considered "deliverable".

Git Push

Sometimes artifacts belong in a separate code repository. For example, an SDK built from the API definitions in a monorepo needs to be published.

See an example git_push executable in this repository.

S3 upload

See s3_sync.

Which targets to deliver

A bazel query expression is the most convenient way to locate deliverable targets. Users may choose a tagging scheme for their workspace (i.e. "all targets with tags = ['artifact']"), or deliver well-known rule kinds (i.e. oci_push), or both.

Which changes to deliver

To optimize time and money, it's best to deliver only "changed" targets. This avoids wasted time and resources uploading the same artifact repeatedly. It also means that release engineers won't have to sort through a massive list of duplicates when choosing a release.

There are two approaches for choosing "changed" targets:

1. Predict the changes based on a version control delta. For example you could git diff between the hash being delivered and the "prior successful" delivery hash, then use a tool like bazel-diff or target-determinator to produce a list of targets that might be affected by those changes.
2. Determine empirically based on what is actually different. This requires determinism, so it must only use unstamped build results (--nostamp). In most cases a green CI run just completed, so these unstamped outputs are easily available.

Aspect recommends following the second approach because the first has some downsides:

• bazel-diff is incorrect and will sometimes miss affected targets, so they aren't delivered.
• target-determinator is slow and may hurt the "service level indicator" of time between pushing a hotfix and being able to release that fix.
• It will over-deliver, because sometimes a source change doesn't actually factor into whether the release binary changes, such as for a comment-only change.

The rest of this section provides more details about the second approach (determine empirically).

To determine whether Workflows should deliver that executable target on a particular commit, it is first hashed using the aspect outputs command with a special pseudo-mnemonic "ExecutableHash", for example:

$ aspect outputs 'attr("tags", "\bdeliverable\b", //...)' ExecutableHash
//cli:release h1:cj8OUC3l3fIr3Zxnffk6y7gukLOJmiWRCAQoqadg66Y=
//workflows/rosetta:release h1:kjHVajw+Nta2kh3Epcd32DkZxTE1NHA8b5N7hCNFNSM=

You need a lookup database to store previously delivered hashes. If the hash value matches one previously seen, then skip delivery of that target.

Debugging changes to deliver

You can run the aspect outputs command locally to understand whether a given change to a source file results in a new executable. Sometimes the result may be surprising. For example, if a comment in a .go source file is changed, the compiler produces the same .a file as a result, so the hash seen on the uploader executable is unchanged.

Another scenario that won't change the executable is when some production configuration is changed. For example, you may use Helm charts to deploy to Kubernetes. If these aren't included inside the image, then changes to these files won't cause a new delivery.

Perform the delivery

Run each deliverable target with stamping enabled. You can do this in a script which reads the targets from a manifest file such as cat $delivery_manifest | xargs -N1 bazel run --stamp.