Token-based authN/authZ for remote clusters

Token-based authN and authZ

Following on the external remote cluster documentation, there is an additional auth schema available for AWS customers: token-based auth. In this schema, users rely on long-lived personal access tokens (PATs), similarly to those available via GitHub or GitLab. This affords some benefits for developer experience (DX) and ergonomics, as users no longer have to rely on a CLI tool to frequently refresh credentials. Instead, token registration occurs behind an OIDC layer, validation occurs at the cluster level, and token management is possible through an administrator UI.

info

It is typically recommended to start with the OIDC-based solution described in the linked documentation before moving to token-based auth. OIDC/OAuth is an inherently more secure and integrated solution for security management.

Enable token-based auth

Enabling token-based auth is straightforward, especially for customers that have previously adopted an OIDC-based auth schema for the external cluster. This guide presupposes that you have already completed those steps in practice.

info

This solution works with internal-facing caches, but still requires OIDC and a public-facing ALB in order to host the token management interface.

To begin, setup a dedicated "token auth" module. This module will be the source of truth for all token configuration, regardless of how many clusters are configured. Most customers will only have one cluster per install, but this allows flexibility to reuse the configuration if/when necessary.

module "aspect_workflows_token_auth" {
  source = "<WORKFLOWS INSTALL URL>//remote/token"

  admin_users = [<ADMIN USER EMAIL>, ...]
  clusters = {
    "aw-remote-ext" = {
      permissions = {
        read    = true
        write   = false
        execute = false
      }
    }
  }
}

In the example above, the cluster name is aw-remote-ext. This is the default cluster name for external remote clusters in Aspect Workflows. If you have customized this name or have a different deployment separate from Workflows, use that instead.

Once this module is added, run terraform apply. This is to ensure that the base resources are available in Terraform state, and can be referenced in subsequent applies.

Before proceeding to the next step, ensure that the web UI is enabled. This is the token management portal, and allows for leveraging OIDC out of the box for token management (both for users and administrators).

Once the token module is applied, and the web UI is enabled, add the following line to the external_remote block in the Workflows module declaration.

module "aspect_workflows" {
  ...
  external_remote = {
    token = module.aspect_workflows_token_auth
  }
}

Once again run terraform apply to affect the changes needed to switch the cache from an ALB to an NLB and to wire up the necessary services and permissions to make the feature available.

Generate a token

From the web UI enabled in the previous step, navigate to <WEB UI URL>/token. This is the user token management interface. Users can generate one token to be used across different remote clusters, centrally managed in one place.

The tokens generated in the web UI have an automated expiry, by default set to one year. The tokens are encrypted using a customer-managed key (CMK) unique to each AWS account. The information contained in each token is leveraged to validate user information at request time, and is cached at the service level for at most one hour.

Each token is by default granted permissions specified at the cluster level in the Terraform configuration. Additional permissions can be granted by administrators in a separate console, specified below. When a request is made, the user info is analyzed at cluster ingress to determine whether an action can proceed. There are three permission types: Read, Write, and Execute. The latter only takes effect if remote execution is enabled for the cluster, and is otherwise a no-op.

The UI displays the generated token once for a user, and then is irretrievable. If needed, a user can rotate their token to both invalidate the previous token and generate a new one. By default, all previous permissions are kept when rotating a token (in the event a user has more than the default permissions). There is also the option to delete a token if the user no longer needs access to the remote cluster.

Using tokens

Once a user generates a token, the web UI will generate a prompt for how to leverage the token in Bazel configuration in source control. It is highly recommended that each customer develop their own installation instructions, as each repository setup is different (especially considering .gitignore settings).

info

The default header (X-Aspect) and token prefix (aw_) can both be changed in the Terraform configuration. The header can obfuscate the underlying provider and make the header more obvious. The prefix can be changed to allow for custom code scanning tools to pick up any accidentally committed tokens (similarly to how GitHub scans the web for their tokens by prefix).

Managing tokens

For administrators, there is an additional management console at <WEB UI URL>/token/admin. This console is only available to users specified in the admin_users block in Terraform, and is enforced via OIDC.

With the token management UI, administrators can see all user tokens, as well as any and all expanded permission sets. Administrators have the ability to modify permissions for all users, and to rotate/delete tokens on their behalf. To offboard a user, an administrator need only remove the user from OIDC (to prevent regenerating a deleted token), and then delete their token in this interface. The cache expiry of one hour will eventually invalidate their credential; otherwise, a user with AWS permissions can bounce the AWS ECS service responsible for cluster auth to have changes take effect immediately.

Machine tokens

It is desirable in certain circumstances to have machine users authenticate with the remote cluster in order to "prime" the cache with regularly-generated artifacts. This allows users to have relatively fresh artifacts when they're ready to build locally.

To create a machine token, click the Create token button and input a dedicated email address for the machine token. It is recommended to follow an identifiable convention like bot-<some identifier>@<customer email>. This token acts just as any other token, and can be given elevated permissions, rotated, deleted, etc. as needed. The token will also likewise expire with the configured expiry, like all other tokens, so be sure to set a reminder to rotate the token as needed. Usage instructions for this token are also identical to user tokens and can be referenced above.