BUILD file generators in Starlark
Aspect CLI Pro includes the ability to write BUILD file generation extensions in Starlark.
These extensions are executed when a user runs aspect configure
.
This feature is experimental, and the Starlark API is subject to change without notice.
Why we made it
Compared with writing Gazelle extensions in Go, there are numerous advantages:
It's pre-compiled and statically linked
- Starlark is an interpreted language, so there's no need to recompile a binary when code changes. Product engineers are never slowed down waiting for compilation, and aren't affected by problems with building Go code.
- Some extensions require CGo, like rules_python. This requires a functional C++ toolchain on every users machine, making it even less portable or forcing you to setup a hermetic C++ toolchain, including a giant sysroot download, even in repositories that have no C++ code. See https://github.com/bazelbuild/rules_python/issues/1913
Starlark is the language of Bazel extensibility
- Logic can be shared between a rule implementation and the corresponding BUILD generator.
Also, logic implemented in a macro that provides a user experience like
my_abstraction
can be ported to a generator which writes the equivalent targets into theBUILD
file (imagine this as "inline macro" refactoring) - and vice versa. - All developers interacting with Bazel have basic Starlark familiarity and can read the code. Not everyone knows Go.
- It's much easier to customize the logic in a user's repository, obviating the need for more expressive "directives" which are load-bearing comments that are easy to miss and don't get syntax highlighting.
It's approachable
- Our API is designed to be easy for novices to use. In contrast, the effort to implement and ship a Gazelle extension is high because the API abstractions are low-level.
- Writing and sharing a general-purpose Gazelle extension is difficult because it's expected to handle every possible scenario. In your repo you can make a tradeoff to take shortcuts based on your needs.
Design
Aspect CLI embeds a starlark interpreter as a Gazelle "extension".
Inside this interpreter a new top-level symbol aspect
is exposed which gives access to the API.
This allows existing Gazelle extensions written in Go to interoperate with Starlark extensions.
Currently those other Go extensions must be statically compiled into the aspect
binary, however
we anticipate that https://github.com/bazelbuild/bazel-gazelle/issues/938 will allow pre-compiled
custom Gazelle extensions to participate under aspect configure
.
Writing plugins
Create a starlark source file.
We recommend using a .star
extension, so that GitHub and other tools will provide syntax highlighting, formatting, etc.
Typical locations include
/tools/configure/my_extension.star
: next to other tool setup/bazel/rules_mylang.star
: next to Bazel-specific support code/.aspect/cli/my_ruletype.star
: alongside configuration of Aspect CLI
The plugin will use the aspect
top-level symbol we provide in the Starlark interpreter context.
You'll call aspect.register_configure_extension
at minimum.
Here's a very simple example that generates sh_library
targets for all Shell scripts:
"Create sh_library targets for .bash and .sh files"
aspect.register_configure_extension(
id = "rules_sh",
prepare = lambda cfg: aspect.PrepareResult(
sources = [
aspect.SourceExtensions(".bash", ".sh"),
],
),
declare = lambda ctx: ctx.targets.add(
kind = "sh_library",
name = "shell",
attrs = {
"srcs": [s.path for s in ctx.sources],
},
),
)
See a basic rules_cc example
currently using basic regular expressions to detect #include
statements and main()
methods to generate cc_library
and cc_binary
targets.
We plan to provide more examples in the future. For now, consult the API docs below.
Loading plugins
The starlark interpreter runtime is shipped in Aspect CLI. Check that page for install instructions first.
Next, add a section in the .aspect/cli/config.yaml
:
configure:
plugins:
WORKSPACE_relative/path/to/my_plugin.star
Enabling plugins
Individual plugins can be enabled/disabled via BUILD directives:
# aspect:{plugin_id} enabled|disabled
Extension registration API
aspect.register_rule_kind
Register a new rule kind that may be generated by a configure
extension.
Args:
name
: the name of the rule kindFrom
: the target .bzl file that defines the ruleNonEmptyAttrs
: a set of attributes that, if present, disqualify a rule from being deleted after merge.MergeableAttrs
: a set of attributes that should be merged before dependency resolutionResolveAttrs
: a set of attributes that should be merged after dependency resolution
aspect.register_configure_extension
Register a configure
extension for generating targets in BUILD
files.
Args:
name
: a unique identifier for the extension, may be referenced in Starlark API or used in# aspect:{name} enabled|disabled
directives etcproperties
: a list of property definitions (optional)prepare
: the prepare stage callback (optional)analyze
: the analyze stage callback (optional)declare
: the declare stage callback (optional)
Stages
Starzelle has multiple stages for generating BUILD
files which extensions can hook into:
- Prepare
- Analyze
- Declare
All stages are optional for extensions.
Stages are executed per BUILD
file. BUILD
files may or may not be pre-existing depending on the # aspect:generation_mode update|create
.
Stages are executed in sequence, however within a stage extensions may be executed in parallel.
Prepare
Prepare(ctx PrepareContext) PrepareResult
Declares which files the extension will process and any queries to run on those files.
PrepareContext:
The context for a Prepare
invocation.
Properties:
.repo_name
: the name of the Bazel repository.rel
: the directory being prepared relative to the repository root.properties
: a name:value map of extension property values configured inBUILD
files via# aspect:{name} {value}
aspect.PrepareResult(sources, queries):
The factory method for a Prepare
result.
Args:
sources
: a list of source file matchersqueries
: aname:aspect.*Query
map of queries to run on matching files, see Query Types
Source Matchers
aspect.SourceFiles(files...):
Match specific file paths.
aspect.SourceExtensions(exts...):
Match files with the trailing extensions. Extensions should include the leading .
.
aspect.SourceGlobs(patterns...):
Match files matching glob patterns. Note that globs are significantly slower than exact paths or extension based matchers.
Analyze
Analyze(ctx AnalyzeContext) error
Analyze source code query results and potentially declare symbols importable by rules.
AnalyzeContext:
Properties:
.source
: aaspect.TargetSource
of the source file being analyzed
Methods:
.add_symbol(id, provider_type, label)
: add a symbol to the symbol database.
Args:
id
: the symbol identifierprovider_type
: the type of the provider such as "java_info" for java packages etclabel
: the Bazel label producing the symbol
Types
aspect.TargetSource:
Metadata about a source file being analyzed.
Properties:
.path
: the path to the source file relative to theBUILD
.query_results
: aname:result
map for each query run on this source file
See Query Types for more information on query result types.
aspect.Label(repo, pkg, name)
Construct a Bazel label.
Args:
repo
: the repository name (optional)pkg
: the label package (optional)name
: the label name
DeclareTargets
DeclareTargets(ctx DeclareTargetsContext) DeclareTargetsResult
Declare targets to be generated in the BUILD
file given the declaration context
DeclareTargetsContext:
The context for a DeclareTargets
invocation.
Properties:
.repo_name
: the name of the Bazel repository.rel
: the directory being prepared relative to the repository root.properties
: a name:value map of extension property values configured inBUILD
files via# aspect:{name} {value}
.sources
: a list ofaspect.TargetSource
s to process based on theprepare
stage results.targets
: actions to modify targets in theBUILD
file, seeaspect.DeclareTargetActions
DeclareTargetActions:
Actions to add/remove targets for a BUILD
file.
Methods:
.add(name, kind[, attrs][, symbols])
: add a rule of the specified kind to theBUILD
file with a set of attributes and exported symbols Params:name
: the name of the rulekind
: the rule kind, a native/builtin rule or one registered withaspect.register_rule_kind
attrs
: a name:value map of attributes for the rule, values of typeaspect.Import
will be resolved to Bazel labelssymbols
: a list of symbols exported by the rule
.remove(name)
: remove a rule from the BUILD file
Query Types
Source files can be queried using various methods to extract information for analysis. Some query types return data
directly from the source code, such JSON and other structured data, while others return QueryMatch
objects describing
the matched content.
aspect.AstQuery(grammar, filter, query):
The factory method for an AstQuery
.
Args:
filter
: a glob pattern to match file names to querygrammar
: the tree-sitter grammar to parse source code as (optional, default based on file extension)query
: a tree-sitter query to run on the source code AST
A tree-sitter query to run on the parsed AST of the file.
See tree-sitter pattern matching with queries
including details such as query syntax,
predicates for filtering,
capturing nodes for extracting QueryMatch.captures
.
The query result is a list of QueryMatch
objects for each matching AST node. Tree-sitter capture nodes
are returned in the QueryMatch.captures
, the QueryMatch.result
is undefined.
aspect.RegexQuery(filter, expression):
The factory method for a RegexQuery
.
Args:
filter
: a glob pattern to match file names to queryexpression
: a regular expression to run on the file
The query result is a list of QueryMatch
objects for each match in the file.
Regex capture groups are returned in the QueryMatch.captures
, keyed by the capture group name.
For example, import (?P<name>.*)
will populate QueryMatch.captures["name"]
with the captured value.
The full match is returned in the QueryMatch.result
.
See the golang regex documentation for more information.
aspect.RawQuery(filter):
The factory method for a RawQuery
.
Args:
filter
: a glob pattern to match file names to return
The query result is the file content as-is with no parsing or filtering.
aspect.JsonQuery(filter, query):
The factory method for a JsonQuery
.
Args:
filter
: a glob pattern to match file names to queryquery
: a JQ filter expression to run on the JSON document
The query result is a list of each matching JSON node in the document.
For queries designed to return a single result the result will be an array of one object, or empty array if no result is found.
JSON data types are represented as golang primitives and basic arrays and maps, see json.Unmarshal.
See the jq manual for query expressions. See golang jq for information on the golang jq implementation used by starzelle.
aspect.QueryMatch:
The result of a query on a source file.
Properties:
.result
: the matched content from the source file such as raw text.captures
: aname:value
map of captures from the query
Utils
path.join(parts...)
Joins one or more path components intelligently.
path.dirname(p)
Returns the dirname of a path.
path.base(p)
Returns the basename (i.e., the file portion) of a path, including the extension.
path.ext(p)
Returns the extension of the file portion of the path.