Remote Cache

Remote resources

Every Workflows deployment includes a remote cache compliant with the Bazel Remote Execution Protocol v2. This means that the CI runners instantiated by Workflows are able to take advantage of previous work done, leading to significantly faster warm builds. All the traffic between the CI runners and the remote cache takes place solely within the VPC of the cloud provider, meaning there is no data leakage or egress, and only the builds the CI runners provision can read from/write to the cache.

Configuration

AWS

Replace N with the desired number of remote cache shards in your Workflows module configuration. Multiple cache shards improves the maximum network throughput.

The cache capacity is determined by the number of shards multiplied by the number and size of NVMe volumes attached, which is dependent on the instance type.

To use recommended defaults, set remote = {}.

remote = {
  storage = {
    num_shards = N
    instance_type = "im4gn.large" # 1 NVMe SSD volume with 937GiB
  }
}

info

We recommend the im4gn.large instance type because of its large SSD capacity and high network throughput at its on-demand price point.

GCP

Replace N with the desired number of remote cache shards in your Workflows module configuration. Multiple cache shards improves the maximum network throughput.

The cache capacity is determined by multiplying the number of shards by the number of 375GiB local SSDs attached to each node. For machine types that have a fixed number of SSDs (such as C3), specifying the machine type is sufficient. For types where you can configure the number of attached SSDs, set k8s_custer.remote_cache_nodes.num_ssds to the desired count.

remote = {
  cache = {
    shards = N
  }
}

k8s_cluster = {
  remote_cache_nodes = {
    count        = N
    machine_type = "c3-standard-4-lssd" # 1 local SSD with 375GiB
  }
}

info

We recommend the C3 machine type. It is performant for caching and has higher data egress.

For configuration of the remote build execution (RBE), see remote build execution. For configuration of an external remote cluster, see external remote.

Optional (AWS): accessing the CI runner cache from a non-Aspect runner instance

You may need to access the CI runner remote cache from a runner that is not managed by Aspect, and thus outside the VPC in which the remote cache is contained. For instance, there may be pre-existing hardware that needs to share the cache with the CI runners.

To accomplish this, peer your VPC to the Aspect VPC. Then, allow ingress from the VPC to the CI runner remote cache by adding a new rule to its security group. For example:

data "aws_vpc" "vpc" {
  id = "<Aspect VPC ID>"
}

data "aws_security_group" "aw_cache_alb_sg" {
  vpc_id = data.aws_vpc.vpc.id

  filter {
    name   = "tag:Name"
    values = ["aw-cache-service"]
  }
}

resource "aws_security_group_rule" "legacy_vpc_ingress" {
  type = "ingress"
  description = "gRPC from legacy VPC"
  security_group_id = data.aws_security_group.aw_cache_alb_sg.id
  cidr_blocks = ["<Your VPC CIDR blocks>"]
  protocol = "TCP"
  from_port = 8980
  to_port = 8980
}

Once the security group is updated, download the self-signed certificate tied to the load balancer for the CI runner cache and attach it to Bazel calls. First, retrieve the cache endpoint and its certificate, either from AWS SSM, or from two outputs from the Aspect Workflows module:

• external_remote_cache_endpoint: the CI runner remote cache endpoint
• internal_remote_cache_certificate: the certificate tied to the CI runner remote cache load balancer

Extra BuildBarn configuration

When using BuildBarn's bb-clientd, add the following to the configuration file:

grpc: {
  tls: {
    server_certificate_authorities: "<internal_remote_cache_certificate contents>"
  }
}

You need to add this block wherever gRPC configuration is present in the configuration file.

Extra Bazel command-line tool configuration

When using the Bazel command-line tool directly, add the following flags:

--remote_cache=grpcs://<external_remote_cache_endpoint>:8980 --tls_certificate=<file containing internal_remote_cache_certificate>

Probing the remote cache

To probe and check the status of the remote cache from a legacy CI pipeline before starting a build, use grpcurl.

For example, using Docker, and assuming that you have downloaded the self-signed certificate and it's available locally to where you run the following command:

docker run -v $(pwd):/config fullstorydev/grpcurl -cacert /config/<cert filename> <ECS remote cache endpoint>:8980 list

You can find other ways to install and run grpcurl in its GitHub repository.