How does Bazel track local resource usage?

How does Bazel avoid melting your workstation with concurrent subprocesses? Or… tries to, because I know it still does that sometimes? There are two mechanisms as play: the jobs number and the local resources tracker. Let’s dive into them.

The jobs number, given by the --jobs flag, configures the number of concurrent Skyframe evaluators during the execution phase¹. What a mouthful. What this essentially means is that jobs indicates the number of threads used to walk the graph looking for actions to execute—and also executing them. So if we have N threads, each of which processes one node of the graph at a time, and we know that the most nodes trigger process executions, we have at most N concurrent spawns.

Which works well, but it’s not sufficient. Not all actions are equally costly: some may require more than one CPU and this model doesn’t account for RAM usage at all. Also, with remote execution, the number of jobs has to be increased to the hundreds to reap its benefits, so if you have set --jobs=300 and configured all actions to run remotely except for a few mnemonics, the latter could still run en masse.

To control this, Bazel has another mechanism known as the local resources tracker. To summarize what this does, know that Bazel has its own idea of the resources available on the machine and each spawn has an associated cost. Whenever Bazel has to run a spawn, Bazel subtracts the spawn’s cost from the tracker if resources are available or waits until they are, and then returns the resources once the spawn completes.

The key concepts and code references to navigate how this works are:

• The amount of resources, be them the total available resources of the machine or the cost attached to an action, are modeled by the ResourceSet class, which essentially is a value class that specifies a CPU and RAM quantity.

• The resource tracker is implemented by the ResourceManager class class, which provides convenient methods like acquireResources to either deduct or wait for resources. The tracker is initialized at the beginning of the build with the values of --local_cpu_resources and --local_ram_resources if provided. The default value of these flags is auto, which means to use a quantity derived from the machine running Bazel as exposed via the LocalHostCapacity class.

• Each action (or, really, each spawn) carries a resource cost with it, which is assigned at creation time by the rule creating the action. Such resources are available via the getResources() method.

• And finally, each spawn runner (see What are Bazel strategies? for details) that executes things locally ends up requesting the necessary resources from the ResourceManager right at exec() time to gate the execution of a spawn.

That’s it. Those are all the pieces you need to know to puzzle together how local resource tracking works.

So now let’s analyze why this local resource tracking mechanism is… very rudimentary:

• First of all, Bazel is not the only process running in the machine—yet the ResourceManager assumes it has all machine resources available to itself by default. This works as long as you don’t do anything else on the machine, like using cough Chrome cough, or if your build is primarily remote-only, but breaks badly on busy machines.

• Resource modeling for actions is… inaccurate. Most actions have a requirement of a single CPU and an almost-arbitrary RAM reservation. Some others, like C++ linking, attempt to model RAM consumption based on the number of inputs… which I guess “works” but is so tied to the behavior of a specific linker that encoding this in the Java code is plain wrong.

• And this is hard to reason about. Most people are used to the --jobs=N model as they have almost-certainly played with make -jN at some point, but understanding what’s happening in Bazel is hard.

That said, after a couple of refinements we made (like removing completely-broken I/O tracking and forcing all actions to request at least 1 CPU), this model works pretty well. In fact, because of the latter:

--local_cpu_resources essentially acts as --jobs to control the number of concurrent local processes, with only a few exceptions.

But of course there is room for improvement. Here are a couple of ideas that would make all of this better:

• bazelbuild/bazel #6477: Allow rules defined in Starlark to specify action costs. At the moment this is only possible from rules defined in Java or via undocumented tags in some cases, yet this would be very useful for rule maintainers.

• bazelbuild/bazel #10443: Allow Bazel to dynamically determine the resources given to a spawn, not the other way around. Suppose you have a multithreaded compiler like swiftc or are wrapping another build system like cargo. At the moment, you are forced to configure these tools with a fixed number of threads, and this number should match the resource cost attached to the action. This is fine when your dependency graph is broad enough to fill all cores with work. But what happens if you reach a choke point in the dependency graph and all you can do is issue that one swiftc or cargo invocation before anything else can run? In this case, you’d want Bazel to configure the invocation to use all available resources, and only Bazel knows what those are based on current machine usage and available Skyframe parallelism.

Do these ideas sound interesting? I’d really appreciate help in addressing these, so drop me a note!