Remote Build Execution

Remote execution

Every Workflows deployment includes a remote cluster compliant with the Bazel Remote Execution Protocol v2. Remote build execution (RBE) is available with minimal additional configuration needed. This allows for the creation of specially tailored runners for individual jobs, configured throughout the build tree. This also means that jobs can be parallelized effectively, where pending actions can unblock many parallel runners at once without repeated work. As a result, cost can be effectively managed by only provisioning large workers for large actions, which can then be spun down more readily while the smaller actions continue on smaller, less powerful workers. Remote execution supports both arm64 and amd64 architectures simultaneously, meaning that a remote execution cluster can be provisioned that supports cross-compilation/publishing or any other cross-platform use case, with minimal additional configuration.

For configuration of the remote cache, see remote cache. For configuration of an external remote cluster, see external remote.

Enable remote execution

By default, remote execution is not enabled in Workflows remote resource clusters. This is because remote execution requires a special focus on sandboxing and reproducibility beyond what may be required on an ephemeral runner or a user's machine. That said, remote execution can be enabled as simply as adding the following to the Workflows remote cluster configuration.

From within your Aspect Workflows module definition, add the following code:

AWS

remote = {
  remote_execution = {
    executors = {
      default = {
        image = "ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448"
      }
    }
  }
}

GCP

Set up a node pool for remote executors under k8s_cluster and configure an executor type under remote.remote_execution. Below, the default executors use the default node type, although you may have several executors with different docker images using the same node pool.

remote = {
  remote_execution = {
    executors = {
      default = {
        image       = "ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448"
        node_type   = "default"
        min_scaling = 0
        max_scaling = 5
        concurrency = 4 # The # vCPUs on the node type (c2d-standard-4 below)
      }
    }
  }
}

k8s_cluster = {
  remote_exec_nodes = {
    default = {
      min_count    = 0
      max_count    = 5
      machine_type = "c2d-standard-4"
      num_ssds     = 1
    }
  }
}

This configuration spins up a new set of runners that pick up work using the Docker image specified. All the scaling and provisioning is otherwise handled automatically, and the endpoint is the same as the one for the remote cache. To use this in Bazel, a new platform needs to be added that looks something like the following.

In a BUILD file, e.g. <WORKSPACE root>/platforms/BUILD.bazel

platform(
  name = "my_remote_platform",
  exec_properties = {
    "OSFamily": "linux",
    "container-image": "docker://<some Docker image>",
  },
)

Then, the .bazelrc file needs to be updated to point to the new platform for a given configuration, e.g. --config rbe:

build:rbe --extra_execution_platforms=@<WORKSPACE name>//<path to platform>:my_remote_platform
# local fallback allows genrule to be executed locally if requested explicitly
build:rbe --genrule_strategy=remote,local
build:rbe --host_platform=@<WORKSPACE name>//<path to platform>:my_remote_platform
build:rbe --jobs=32
build:rbe --remote_timeout=3600

Note: the above is simply an example. Please adjust as needed to a given use case.

Once completed, jobs should be able to run on the remote executors seamlessly, provided the configuration points to the right place. Additional build failures may be encountered initially that did not occur before; as stated previously, remote execution requires greater attention to detail in the structure of the build tree.

Further configuration of the remote executors, including but not limited to direct provisioning of the underlying compute, can be found in the Workflows configuration documentation.