“Strategies? Will you talk about Bazel’s strategy for world domination 🙀?” No… not exactly that.

Dynamic execution has been quite a hot topic in my work over the last few months and I am getting ready to publish a series of posts on it soon. But before I do that, I need to first review Bazel’s execution strategies because they play a big role in understanding what dynamic execution is and how it’s implemented.

Simply put, strategies are Bazel’s abstraction over the mechanism to run a subprocess as part of an action (think compiler invocation). In essence, they are a glorified exec() system call in Unix or CreateProcess in Windows and supercharge them to run subprocesses under very different environments.

The invariant, however, is that strategies do not affect the semantics of the execution: that is, running the same command line on strategy A and strategy B must yield the same output files. As a consequence, changing strategies does not invalidate the analysis graph.

Because of the previous invariant, we can mix and match strategies at run time without affecting the output behavior. This is exposed via the repeated --strategy flag, which takes arguments of the form [mnemonic]=strategy. With this flag, you can tell Bazel to run certain classes of actions under a specific strategy. For example, you might say --strategy=remote --strategy=Javac=worker to set the default strategy to remote and to override it with worker for Java compiles alone. There also is the --strategy_regexp flag to select strategies based on action messages, but I’ll leave the “magic” of that aside. And there is also some fancy support for automatic strategy selection.

Regarding implementation, all strategies are classes that implement the SpawnActionContext interface. Overly simplifying, they must provide an exec() method that takes a Spawn (essentially a command line description) and returns a collection of SpawnResults. Why a collection? Because a strategy may implement retries, and their failed results are exposed as part of the return value.

Looking further into their implementation, strategies wrap a SpawnRunner, which is the thing that actually knows how to run a spawn. You will notice that this interface’s exec() returns a single SpawnResult, which helps understand how the strategy and its spawn runner relate: the strategy contains higher-level logic around the spawn runner and the spawn runner is in charge of the execution details.

With that, let’s now review the primary strategies in Bazel.

The standalone strategy

The standalone (aka local) strategy is the simplest you can have, and is a good starting point to learn more about the internals of strategies and process execution in Bazel.

This strategy executes spawns directly on the output tree. That is: the commands are run with a current working directory that resides inside the output tree and all file references are within that directory. There are no restrictions on what the process can do, so the process can inadvertently have side-effects on unrelated files.

This strategy is at StandaloneSpawnStrategy and uses the LocalSpawnRunner.

The sandboxed strategy

The sandboxed strategy exists to support Bazel’s promise of correct builds and is the default strategy for any local action (which means all actions unless you are doing any customization).

This strategy runs each spawn under a controlled environment that is isolated from the output tree and that is prevented from interacting with the system in certain ways (e.g. no networking access).

During an exec(), the sandbox strategy performs these steps:

1. Create a directory outside of the output tree that exclusively contains the inputs required by the spawn as read-only files. This is currently done in the form of a symlink forest.
2. Execute the spawn under that separate directory, using system-specific technologies like namespaces on Linux and sandbox-exec on macOS to constrain what the process can do.
3. Move the outputs out of the sandbox and into the output tree.
4. Delete the sandbox.

This approach works… but can have a significant penalty on action execution performance. This has been a pet peeve of mine and I’ve been trying to improve it for a long time with sandboxfs, though I haven’t gotten to major breakthroughs just yet.

This strategy is implemented as one class per supported operating system and all of them live under the sandbox subdirectory.

The worker strategy

The worker strategy exists to speed up the edit/build/test cycle of languages whose compiler is costly to start or whose compiler keeps state around to optimize incremental builds. The strategy communicates with a long-lived persistent worker process and essentially sends the command lines to execute to that separate process.

For example: in the case of Java, the persistent worker avoids the penalty of JVM startup and JIT warmup times; and in the case of Dart, the persistent worker allows the compiler to keep file state in memory so that subsequent compilations are much faster.

The risk of allowing workers is that they can introduce correctness issues more easily than the typical startup/shutdown process: any bugs while processing an action, which may be harmless if the compiler is shut down immediately afterwards, can have cascading effects if the compiler is kept around for a long time and reused for other files. (Yes, we have hit these kind of issues in e.g. the Swift worker.)

Additionally, worker management in Bazel is currently very rudimentary so blindly enabling workers can make your machine melt due to memory pressure. We have plans to improve this significantly but haven’t gotten to them just yet.

This strategy is at WorkerSpawnStrategy and uses the WorkerSpawnRunner.

The remote strategy

The remote strategy is the crown jewel of Bazel. This strategy is what allows Bazel to execute processes on remote machines thus letting your project break loose of the constraints of a single machine’s build power.

This is the most complex strategy of all if only because of the need to coordinate with remote machines over gRPC, having to multiplex requests, and having to deal with networking hiccups. I will not dive into any of its details here.

This strategy is at RemoteSpawnStrategy and uses the RemoteSpawnRunner.

That’s all for today. In the coming posts, I’ll cover dynamic execution in detail and will likely refer to this post in multiple occasions.

Be aware, however, that a lot of the details on how strategies are defined and registered in Bazel are about to change as part of the ongoing Platforms & Toolchains work. The specifics behind strategy implementation should not change, though.