jmmv.dev

Artifact downloads and dynamic execution

In the previous post of this series, we looked at how the now-legacy implementation of the dynamic strategy uses a per-spawn lock to guard accesses to the output tree. This lock is problematic for a variety of reasons and we are going to peek into one of those here. To recap, the remote strategy does the following: Send spawn execution RPC to the remote service. Wait for successful execution (which can come quickly from a cache hit).

December 30, 2019 · Tags: bazel
Continue reading (about 5 minutes)

Output conflicts and dynamic execution

When the dynamic scheduler is active, Bazel runs the same spawn (aka command line) remotely and locally at the same time via two separate strategies. These two strategies want to write to the same output files (e.g. object files, archives, or final binaries) on the local disk. In computing, two things trying to affect the same thing require some kind of coördination. You might think, however, that because we assume that both strategies are equivalent and will write the same contents to disk1, this is not problematic.

December 27, 2019 · Tags: bazel
Continue reading (about 4 minutes)

Bazel's dynamic strategy

After introducing Bazel’s dynamic execution a couple of posts ago, it’s time to dive into its actual implementation details as promised. But pardon for the interruption in the last post, as I had to take a little detour to cover a necessary topic (local resources) for today’s article. Simply put, dynamic execution is implemented as “just” one more strategy called dynamic. The dynamic strategy, however, is different from all others because it does not have a corresponding spawn runner.

December 26, 2019 · Tags: bazel
Continue reading (about 3 minutes)

How does Bazel track local resource usage?

How does Bazel avoid melting your workstation with concurrent subprocesses? Or… tries to, because I know it still does that sometimes? There are two mechanisms as play: the jobs number and the local resources tracker. Let’s dive into them. The jobs number, given by the --jobs flag, configures the number of concurrent Skyframe evaluators during the execution phase1. What a mouthful. What this essentially means is that jobs indicates the number of threads used to walk the graph looking for actions to execute—and also executing them.

December 23, 2019 · Tags: bazel
Continue reading (about 5 minutes)

Introduction to Bazel's dynamic execution

Bazel’s dynamic execution is a feature that makes your builds faster by using remote and local resources, transparently and at the same time. We launched this feature in Bazel 0.21 back in February 2019 along an introductory blog post and have been hard at work since then to improve it. The reason dynamic execution makes builds faster is two-fold: first, because we can hide hiccups in the connectivity to the remote build service; and, second, because we can take advantage of things like persistent workers, which are designed to offer super-fast edit/build/test cycles.

December 20, 2019 · Tags: bazel
Continue reading (about 3 minutes)

What are Bazel's strategies?

“Strategies? Will you talk about Bazel’s strategy for world domination 🙀?” No… not exactly that. Dynamic execution has been quite a hot topic in my work over the last few months and I am getting ready to publish a series of posts on it soon. But before I do that, I need to first review Bazel’s execution strategies because they play a big role in understanding what dynamic execution is and how it’s implemented.

December 14, 2019 · Tags: bazel
Continue reading (about 6 minutes)

The /bin/bash baggage of macOS

As you may know, macOS ships with an ancient version of the Bash shell interpreter, 3.2.57. Let's see why that is and why this is a problem.

November 20, 2019 · Tags: macos, shell
Continue reading (about 4 minutes)