Migrating to rules_oci

This document contains some of the lessons we've learned at https://aspect.dev from doing consulting work, migrating some large client repos from rules_docker to rules_oci.

This guide remains a work-in-progress as we find new patterns.

Update WORKSPACE

The WORKSPACE file contains Bazel module dependency fetching and installation.

Add install steps from a release of rules_oci, along with related rulesets you plan to use.

Note that the container_test rule from rules_docker is now in a separate repo. See container_structure_test

Replacements by rule

rules_docker	rules_oci	How to migrate
container_push	oci_push	See below
container_image	oci_image	See below
container_bundle	oci_tarball	See below
container_pull	oci_pull	See below
container_test	N/A	Use container_structure_test
container_import	N/A
container_load	N/A
container_run_and_*	N/A	See below
go_image	N/A	Wrap go_binary, see go docs
java_image	N/A	Wrap java_binary, see java docs
py3_image	N/A	Wrap py_binary, see python docs
nodejs_image	N/A	Wrap js_binary, see javascript docs
rust_image	N/A	Wrap rust_binary, see rust docs

container_pull

puller_darwin, puller_linux_* are unsupported. Unlike container_pull, oci_pull uses Bazel's downloader instead of a custom puller binary.

import_tags is unsupported. no plans to add support for it.

cred_helpers is unsupported. credential helpers should be installed on the host where bazel runs.

docker_client_config is unsupported. instead use DOCKER_CONFIG environment variable to override the config.

os_version is unsupported.

os_features is unsupported.

platform_features is unsupported.

os, architecture, cpu_variant is supported, however, are combined into single string that can be passed to platforms attributes.

-load("@io_bazel_rules_docker//container:container.bzl", "container_pull")
+load("@rules_oci//oci:pull.bzl", "oci_pull")
-container_pull(
+oci_pull(
- os = "linux",
- architecture = "arm64"
- cpu_variant = "v8"
+ platforms = [
+     "linux/arm64/v8"
+ ]
)

timeout is unsupported. Bazel's --http_timeout_scaling can be used to increase/decrease the default timeout.

Environment variables

DOCKER_REPO_CACHE is not supported. Since oci_pull uses Bazel's downloader, remote manifests/blobs are cached via Bazel's repository cache as long as a digest is provided.

PULLER_TIMEOUT is not supported.

container_push

Update BUILD files with: buildozer 'set kind oci_push' //...:%container_push

registry and repository are merged into one attribute; repository.

-registry = "registry.io",
-repository = "org/image"
+repository = "registry.io/org/image"

tag attribute is now remote_tags and allows multiple tags.

-tag = "latest",
+remote_tags = ["latest"]

insecure_repository is not needed anymore as it's automatically detected for most cases. However, in case it can't be detected --insecure can be added to the args attribute.

-insecure_repository = true,
+args = ["--insecure"]

tag_file is now remote_tags and expects one tag per line.

-tag_file = ":tag_file",
+remote_tags = ":tag_file",

repository_file is now repository and accepts a file

-repository_file = ":repository_file",
+repository = ":repository_file",

stamp is not supported directly. We encourage users to stamp in a separate target and pass the stamped file to repository or remote_tags attribute respectively. Note that container_push has different behavior where changes to volatile-status.txt are always seen by the program under bazel run, but oci_push does not. See https://github.com/bazel-contrib/rules_oci/issues/269#issuecomment-1611953197.

# an example from rules_oci stamping tags
stamp_tags(
    name = "stamped",
    remote_tags = [
        # With --stamp, use the --embed_label value, otherwise use 0.0.0
        """($stamp.BUILD_EMBED_LABEL // "0.0.0")""",
        "nightly",
    ],
)

oci_push(
    name = "push",
    image = ":oci_image_target",
    repository = "index.docker.io/<ORG>/image",
    remote_tags = ":stamped",
)

format is unsupported. the default format is oci and legacy docker format isn't supported.

skip_unchanged_digest is unsupported. oci_push skips existing blobs by default.

extension is unsupported. the default extension is sh. no plans to support it.

tag_tpl and windows_paths attributes are unsupported. no plans to support it.

container_image

Update BUILD files with: buildozer 'set kind oci_image' //...:%container_image

oci_image does not support following attributes directly, but they can be passed to pkg_tar for layer creation.

• files -> pkg_tar#srcs
• compression -> pkg_tar#compressor
• compression_options -> pkg_tar#compressor_args
• data_path -> pkg_tar#strip_prefix
• directory -> pkg_tar#package_dir
• empty_dirs -> pkg_tar#empty_dirs
• empty_files -> pkg_tar#empty_files
• tars -> pkg_tar#deps
• mode -> pkg_tar#mode
• mtime -> pkg_tar#mtime
• symlinks -> pkg_tar#symlinks

An example demonstrating migrating tars, directory, symlinks from container_image to oci_image

@@ -1,6 +1,13 @@
-container_image(
+oci_image(
     name = "image",
+    tars = [
+        ":new_layer"
+    ]
+)
+
+pkg_tar(
+    name = "new_layer",
     symlinks = { "/usr/bin/app": "/app"},
     tars = [":app"],
-    directory = "/org/image"
+    package_dir = "/org/image"
 )

debs is not supported directly as an attribute. However, since .deb files consist of two tar files data.tar.xz and control.tar.xz, these can be passed to oci_image#tars attribute after extracted from a deb file.

If the debian files are downloaded via http_archive, bazel recently landed support for extracting these files on the fly.

An example genrule extracting .deb files before passing down to oci_image

genrule(
    name = "new_layer",
    srcs = [":deb"],
    outs = ["data.tar.xz"],
    cmd = "tar -xvf -c $@/ $(location :deb)",
)

oci_image(
    tars = [":new_layer"]
)

enable_mtime_preservation is unsupported. pkg_tar doesn't support this feature either. Generally, this is a bad idea since it leads to non-reproducible builds.

layers is now tars. it accepts list of arbitrary .tar or .tar.gz files.

Note that oci_image#tars attribute behaves differently than container_image#tars. While container_image#tars squashes multiple tars into single tar, oci_image#tars preserves tars and creates a layer per tar respecting the order of the tars given at oci_image#tars attribute.

-container_image(
+oci_image(
     name = "image",
-    layers = [
+    tars = [
         ":layer"
     ]
 )

launcher and launcher_args are unsupported. As a replacement cmd and entrypoint can be used instead. However, since entrypoint attribute overrides the entrypoint, it wouldn't be possible to inherit the entrypoint from the base.

-container_image(
-container_image(
+pkg_tar(
+    name = "launcher",
+    srcs = [":launcher"]
+)
+oci_image(
     name = "image",
-    launcher = ":launcher",
-    launcher_args = ["--arg1", "--arg2"],
-    entrypoint = ["/app"],
-    cmd = ["--apparg1"]
+    entrypoint = ["/path/to/launcher", "--arg1", "--arg2"]
+    cmd = ["/app", "--apparg1"],
+    tars = [
+        ":launcher"
+    ]
 )

legacy_run_behavior, legacy_repository_naming, and docker_run_flags attributes are unsupported. Instead oci_tarball rule should be used for loading the oci_image into a docker daemon.

the oci_image target can be loaded into the daemon by running bazel build :tarball, docker load -i bazel-bin/tarball/tarball.tar respectively.

docker_run_flags can be passed to docker directly when running docker run gcr.io/test:latest

-container_image(
+oci_image(
     name = "image",
+)
+
+oci_tarball(
+    name = "tarball",
+    image = ":image",
+    repo_tags = ["gcr.io/test:latest"]
 )

repository and tag_name is unsupported. use oci_tarball#repo_tags as a replacement.

-container_image(
+oci_image(
     name = "image",
     tag_name = "latest",
-    repository = "gcr.io"
+    repository = "gcr.io/image"
+)
+
+oci_tarball(
+    name = "tarball",
+    repo_tags = ["gcr.io/image:latest"]
 )

compression and compression_options are unsupported. use pkg_tar#compressor and pkg_tar#compressor_args when creating layers instead.

experimental_tarball_format is unsupported. oci_image does not produce tarballs. oci_tarball, which produces tarballs out of oci_image, should be used instead.

stamp is not supported directly. oci_image allows attributes such as labels, annotations to be stamped. If the intent is to stamp layers, pkg_tar#stamp is the preferred way to stamp layers.

creation_time is unsupported at the moment. by default creation time for the image is static. follow rules_oci#49 for updates.

os_version is unsupported at the moment. follow rules_oci#48 for updates.

ports is unsupported. most runtimes support --port|-p flag which can be passed at container startup. follow rules_oci#220 for updates.

volumes is unsupported. most runtimes support --volume|-v|--mount flag which can be passed at container startup.

Docker run rules

rules_docker has three rules that rely on running a command in a Docker container during a bazel build action:

• container_run_and_commit
• container_run_and_commit_layer
• container_run_and_extract

As well as the dockerfile_build rule

However these are not hermetic, so they interact incorrectly with Bazel caching. rules_oci does not contain an equivalent rule, see discussion on https://github.com/bazel-contrib/rules_oci/issues/132

These are the approaches typically used instead of these rules:

1. Build a base layer, push/pull it from a registry
2. Fetch packages "the Bazel way"
3. Bazel's docker spawn strategy

1. Build base layer

This is a two-phase strategy, and the most commonly used.

First, build a base layer containing all the system packages you need to be in the image. It's not necessary to use Bazel for this step - many users continue using a non-hermetic Dockerfile in some external image-builder pipeline. When needed, re-build that base layer and push it to some Docker registry.

Second, use oci_pull to fetch that base layer, then add layers on top with your application dependencies and application code.

2. Fetch packages

Bazel can construct a hermetic, reproducible package given explicit instructions on fetching and installing packages, doing so outside of any containerization. This is possible because Docker container layers are simply tar files, and most packages are trivial to install.

For this method, just use oci_pull to fetch a base image, then add layers containing the system packages you need, using one of these:

• For Alpine packages, see rules_apko
• For Debian packages, see rules_debian_packages
• Some useful Linux-specific recipes may be found in https://github.com/GoogleContainerTools/rules_distroless

Finally, add your application's dependencies and application code as layers to the image.

3. Docker spawn strategy

Bazel has long had a built-in way to run bazel build actions inside a container. Some documentation for this may be found on https://bazel.build/remote/sandbox#troubleshooting-natively

In practice, Aspect doesn't know of anyone using this approach to replace rules_docker, so it may be something of a science project. Please report back to us if you've gotten this working, so we can share your recipe with others!