Using Ninja¶

As we saw the previous chapter, RCL can abstract away repetition in files like GitHub Actions workflows and Kubernetes manifests, and enable sharing data between tools that do not natively share data. To do that, we still need to run rcl to generate the .yml, .tf.json, .json, and .toml files that can be consumed by existing tools.

Updating those generated files is the job of a build system. In the previous chapter we saw a lightweight solution that is built into RCL: rcl build. Using rcl build is easy and avoids the need to bring in external tools, but it has two main limitations:

• It can only evaluate expressions, it does not call external programs. Suppose one of the RCL files imports a JSON file that is generated by an external program. Then now we are back to having to run multiple commands to update all generated files.
• It rewrites files even when inputs did not change. Unless your configuration is truly massive, RCL is probably fast enough that regenerating all files is not a problem, but it can still have downsides. For example, if you use the generated files in the next stage of a build system, the updated mtimes may cause unnecessary rebuilds, and rewrites break reflink sharing on copy-on-write file systems.

To get around those limitations, we can to switch to a proper build system, such as Make, Bazel, or Ninja.

Make¶

Updating generated files when inputs change is the role of a build tool. We could use Make and write a makefile:

policies.json: policies.rcl
    rcl evaluate --format=json --output=$@ $<

Aside from the somewhat arcane syntax, this makefile has one big problem. If policies.rcl imports an RCL file, say users.rcl, then Make will not rebuild policies.json when we change users.rcl, because we haven’t specified the dependency in the makefile. Manually listing all transitive dependencies is tedious and prone to go out of date, which is why Make can automatically remake makefiles. Unfortunately the syntax for achieving this is so vexing¹ that it’s hard to seriously consider Make when clearer alternatives exist.

Ninja is a different build tool that can solve this problem by reading transitive dependencies from a depfile, and RCL can write such a depfile. In the remainder of this chapter, we’ll explore using Ninja as the build tool.

Ninja¶

Ninja is a fast and flexible build tool, but its build files are low-level and intended to be generated, not written by hand. Let’s write one by hand anyway, to better understand what we are working with.

In a Ninja file, we first define a rule that specifies how to invoke a program. This is also where we can tell Ninja to use a depfile.

rule rcl
  description = Generating $out
  command = rcl eval --color=ansi --format=$format --output=$out --output-depfile=$out.d $in
  depfile = $out.d
  deps = gcc

Here $in, $out, and $format are variables. Ninja itself sets $in and $out, and $format is one that we define because it varies per target. The deps = gcc line is not required, but it makes Ninja store the depedency information in .ninja_deps and then delete the generated depfile, instead of reading it on demand. This is nice to keep the repository clean.

Next, we add a build statement that specifies how to build a file:

build policies.json: rcl policies.rcl
  format = json

This is enough for Ninja to work. Save the file to build.ninja and then build policies.json:

$ ninja
[1/1] Generating policies.json

$ ninja
ninja: no work to do.

$ touch users.rcl
$ ninja
[1/1] Generating policies.json

Generating Ninja files¶

Okay, so we can write a Ninja file by hand, it’s quite readable even. But at some point, we’re going to end up with lots of similar build statements, and wish we had a way to abstract that. If only we had a tool that could abstract away this repetition …

We can write a ninja.rcl that evaluates to a Ninja build file like so:

#!/usr/bin/env -S rcl evaluate --output=build.ninja --format=raw

let ninja_prelude =
  """
  rule rcl
    description = Generating $out
    command = rcl eval --color=ansi --format=$format --output=$out --output-depfile=$out.d $in
    depfile = $out.d
    deps = gcc
  """;

let build_json = basename =>
  f"""
  build {basename}.json: rcl {basename}.rcl
    format = json
  """;

// File basenames that we want to generate build rules for.
// This is the part we need to edit when we add more files.
let basenames_json = [
  "policies",
];

let sections = [
  ninja_prelude,
  for basename in basenames_json: build_json(basename),
];

sections.join("\n")

Now we can generate the same build file that we previously wrote by hand, and when we add more json target files, we only need to add one string to the list. By adding a #!-line and making the file executable, we can even record how the Ninja file is generated. Unfortunately, even with the #!-line we are back to multiple build steps: first ./ninja.rcl, and then ninja. Can we do better?

For bootstrapping build.ninja, that will always need a manual step. But after we run ./ninja.rcl once, Ninja can keep build.ninja up to date for us. We just need to list it as a build target:

let sections = [
  ninja_prelude,
  """
  build build.ninja: rcl ninja.rcl
    format = raw
  """,
  for basename in basenames_json: build_json(basename),
];

Dynamic targets¶

Now that we generate our build.nina from ninja.rcl, we can import RCL documents to dynamically create build tagets. For instance, we can leverage rcl query to build all the keys of a document manifests.rcl as separate files. We could do that as follows:

#!/usr/bin/env -S rcl evaluate --format=raw --output=build.ninja

let command = [
  "rcl",
  "query",
  "--color=ansi",
  "--format=$format",
  "--output=$out",
  "--output-depfile=$out.d",
  "$in",
  "$query",
];
let ninja_prelude =
  f"""
  rule rcl
    description = Generating $out
    command = {command.join(" ")}
    depfile = $out.d
    deps = gcc
  """;

let build_raw = (target, src) =>
  f"""
  build {target}: rcl {src}
    query = input
    format = raw
  """;

let build_json_query = (target, src, query) =>
  f"""
  build {target}: rcl {src}
    format = json
    query = {query}
  """;

let manifests = import "manifests.rcl";

let sections = [
  ninja_prelude,
  build_raw("build.ninja", "ninja.rcl"),
  for key, _ in manifests:
  // Warning, this assumes that the key is both a valid filename
  // and RCL expression. Currently no built-in functions exist for
  // validating this.
  build_json_query(f"{key}.yml", "manifests.rcl", f"input.{key}"),
];

sections.join("\n")

Warning: Generating targets dynamically is powerful, but also a sure way to make your build process intractable quickly! Use sparingly and with good judgement!

Conclusion¶

RCL enables sharing configuration between systems that do not natively share data. To do so, you will likely need to generate files. Keeping those files up to date is the job of a build tool. In this chapter we have seen how to use the Ninja build tool, and how to use RCL to write Ninja build files.