# BazelCon 2024 - Day 1 ## Keynote First BazelCon not run by Google. Sharing ownership with the ecosystem. Starlarkification. Google equal platinum sponsor with BuildBuddy and EngFlow. Why does Google do Bazel? Get contributions from the community. Easy externalization of projects from google3. New community model (where is it?) to ensure that even if Google pulls the plug, it's not game over for the community. First "user conference" was BazelCon 2017 in Sunnyvale. BazelCon 2023 (250 attendees, 200 waitlisted). Last year we had non-Googlers in the program selection. This year talked to new "homes". The Linux Foundation provides a dedicated planning team for events; they have tech projects; and many other companies are already involved. Created the bazel-contrib organization in GitHub. Google has "donated" many repos to this new org. These were never maintained by Google before anyway, so this doesn't mean Google isn't working on them anymore. There is also a "directed fund" so that companies can join as members and can help staff these projects. Need about 5 companies to start this new foundation; still looking for them. We don't know yet what it'll fund, but it'll depend on the companies that join. ## SOTU * Dependency management: vendor mode for bzlmod to download all deps for a target or all targets; very useful for CI/CD offline builds (disaster recovery?). `MODULE.bazel` now has include support and `use_repo_rule` for easier migration. Better lock file format to minimize merge conflicts. The `WORKSPACE` is disabled by default in Bazel 8 and won't be available in Bazel 9. Planning to share the repo cache (after evaluation, not just use it as a download cache). * RE: Remote output service is done; bb-clientd offers the implementation. Compact execution log for debugging; 100x smaller than before and 3% runtime overhead; useful to debug cache hits. Upcoming: concurrent uploads to the cache in the background, without blocking action completion. Local caches GC (disk cache, install cache, etc.). BwoB improvements to decrease incremental build times when Skymeld is in use. * Rules: Python, Android, Java, Shell... have been moved to Starlark. Protobuf and Android too, which comes with new things like "mobile install v3". Goal for next year is to complete Starlarkification (with possible new extension points to call into Bazel from the Build API). * Starlark and the Build API: no more struct providers, aspects can propagate to target toolchains, dormant deps for tests. Upcoming: symbolic macros in Bazel 8. Rule finalizers (?). Types for Starlark (how does this relate to Buck2's work?). * Bazel "kernel": 100k+ LOC deleted, JDk 21 upgrade (virtual threads and GC improvements), better worker API. Upcoming: project specs to group things edited together. Tame flags via flag sets. Analysis phase caching. Peak and retained heap reduction. Async action execution. * Bazel 8. ## The classics never go out of style (Bazel downgrades) Leading the "Software rebels" group at U. of Waterloo. RE - Repository excavation: analyzing git repos. BE - Build engineering: insights in the build. Even if Bazel exists and promises correctness and performance, some orgs don't use it. Why? "World of Code" dataset. Sampled projects and focused on 35k. Of those, 542 are Bazel adopters (1.5%). Not too bad given the recency of Bazel, but are these adopters sticking? 481 are still on Bazel, so 11% adopters abandoned Bazel. 33% of leavers are hosted by top organizations. Looking at abandoners by language, Go is the most prevalent. Followed by C++. Abandoners usually take 1 year to make the decision to drop it. "11% of large and active projects abandon Bazel after a median of 638 days" Why? Focused on the Bazel leavers (61 projects). Mined repos for docs on the topic and found 212 docs (PRs that removed the files, community discussions, bug reports). Then did thematic analysis. First find large themes together. Then re-label separately. Then produced set of 7 major themes. * Maintenance and troubleshooting. 20%. * Support and interoperability. (Apple, Windows, etc.) 26% * Parity with native toolchains. (Go. Correlated with a go build release that added caching.) 22% * Experimental adoption. 4% * Barrier to contribution. 14% * Influence of other projects. (Kubernetes dropping Bazel.) ?? "Technical challenges, team coordination & onboarding, and community trends." What did they move to? Language-specific tools (Go, Swift, 34 cases). Classics (cmake, gmake, 21 cases). "Less feature rich, language-specific, conventional, and platform-specific technologies." Cases for adoption Bazel: 8.7%. Dep management, faster builds, and influence of other projects. ## RE postmortems (Ulf) Types of root causes: technical failures. Process failures. Human factors (guardrails for customers, etc.) Call graph between services must not have cycles. Also must have timeouts on all RPCs but the RE protocol has long-running RPCs unfortunately. Flow control in a stream must be connected across all nodes in the communication. Otherwise, think a slow Bazel client downloading a large file from a server through an intermediate server. The intermediate server may want to buffer everything. Slow goma builds: very aggressive autoscaler, which was even downscaling when there was queuing. ## Large Java repos (Janusz from Airbnb) 30M+ LOC (Scala, Java, Kotlin, and more). 1000+ deployable artifacts. 1000+ contributors. 3000+ 3P libraries. 8000+ versions of 3P libraries, 4 versions of Scala, 3 versions of Java. * Java is cross-platform, but Bazel doesn't like mixing host systems. New features like scrubbing don't work with RE. We've implemented "universal binaries" (wrappers). Biggest drawback is the size because we support 4 platforms, but in practice it's not a problem. There is an open doc from Thiago on this (universal platforms). Output directories are still a problem because they have the CPU name; new flag `--experimental_platform_in_output_dir` to resolve. * No C++. proto shows up pretty much everywhere (if you use a local worker, you need proto) which means you need C++. * RBE with persistent workers. 2x slower builds without remote persistent workers. `--experimental_remote_mark_tool_inputs`. We use dedicated pools for Java and Kotlin and route all actions into these pools. Careful with tagging targets, because targets can have multiple actions and just one of them needs the worker. We also have "high priority" pools for interactive builds (want to ensure there is no queuing). * Determinism checks. Set up builds for comparison. Timestamps in jars (`rules_antrl` is still impacted), absolute paths (`rules_kotlin` is impacted), Scala (`sdeps` file for diagnostics is problematic), etc. * Avoid Bazel in inner-loop. (Personal opinion: Bazel is too heavy. It should be usable, but it isn't. Too hard to scale it down -- perpetual problem.) Overhead and lack of incrementality in actions. We do the equivalent of "fast builds" in the plugin, and have a custom fork. Users still have to sync to handle new deps or codegen, but it's minimized. 30x faster incremental builds with this work (30 secs to 1 sec is a big deal for users). * Generate BUILD files. Auto-generating them is the right choice, and Gazelle does this, but we wrote our own version. Infer deps from imports, and we have an IDE plugin that updates build files and more. * Java. Using multiple source and runtime versions. Using `common --javacopt="--release 8"`. * Version resolution. One version policy. * Tests. Custom test runner because of junit4 and junit5. Collecting profiling data on tests is nice because it tells devs where the time is going and they can fix it. * Configuration. ## Action deduplication (merging, coalescing) Save money by using less compute. Save time by finishing earlier for the second developer. But where does this shine? In CI. We see 500k dedups a day on about 3 webpack builds. Flaky test passes/failures are amplified by this. Failures (and passes) may show up clustered instead of over time. ## Building 1500 container images in three minutes A container image is a JSON payload that describes the layer combination. Custom BES backend that looks at TargetCompleted entries, looking for image layers, and the backend downloads the information to generate the container image, which it then uploads to the container registry. Only downloads and reuploads layers that are missing, so things like the base Debian image is almost never transferred. ## Performant Bazel builds for web monorepos at scale Eng runs builds on macOS and CI is all Linux. * Stabilized external dependencies. * Universal binaries (per the previous). * Continuous non-determinism checks. * Optimize size of targets. Avoid depending on tons of individual small files (because of RE overheads, especially in tree computations): instead, tar them up. Opposite of tree artifacts? When do they really help? ## Perfect sandboxing in Bazel Lots of custom/vendor tooling -- cannot modify it. Common compute environment with shared NFS. Absolute paths everywhere. Hermetic Linux sandbox restricts access to the host file system. The sandbox starts empty, and we need to map in some directories. Use a toolchain to define the base libraries and tools (bash, etc.) ## Running a start-up on Bazel Good things: * Keep it simple: GCS cache is not perfect, but it works. * `bazel test ...` is super easy in CI. * Wrapping Gradle inside Bazel. Similar for Unity. * bazel-diff. * Slack has been very useful. * Consistent coverage metrics across the codebase is very powerful. Things we don't like: * Docs. Gap between trivial tutorials and reference docs. * IDE integration. Most experienced team members don't feel it, but the less experienced individuals rely on the missing features because they are used to them. Forked Android IDE plugin to build missing features that were started at Uber. How to make it work in a small org: * Have a champion. It doesn't have to be the CTO... but it helps. * Focus on "good enough". * Write documentation x3. * Join the community. Prefer doing small changes incrementally over a big one-off change. ## Spotify 2021: 100 devs, 2.6M LOC, 1M LOC C++. 3 years for the whole migration. `--experimental_prune_transitive_deps`. Combining classes into single test targets was good for performance (producing fewer outputs) and good for usability given that devs usually run certain tests in unison anyway. Results: CI dropped from 45 minutes to 20 minutes. We build release apps in merge checks, which use R8 which is slow and not parallelizable. CI agent time per PR dropped from 15 hours to 5 hours. Codebase is growing but this number isn't. ## IntelliJ plugin New plugin: * Released today. * Based on BSP. * Module representation inside IntelliJ and navigation. * Syntax highlighting for bazelrc and flag docs on hover. * Starlark debugging with breakpoints. * Run configurations. * "Fast build" supported natively. * Separate sync only vs. build+sync. Goal is to make this transparent for users. * Support adding new files automatically. * * * * Major gaps from the old plugin: language support for Python, Go, C++. (Python and Go are available behind a feature flag.) Query sync. Starlark related features. * Need to use a bazelproject file to scope the project down for very large repos. * Open to adding any hooks we may need. We have sync hooks (pre-sync for build file generation). * Completions are only available for the "current scope" (transitive closure of dependencies), but there is an option in IntelliJ to query the whole project. It should just be one extra key press away. * The logic is mostly aspect-based, which means it's very easy to release fixes compared to the previous plugin. Getting custom rules to work with the BSP is very complicated. Have had to modify the aspect, but it's not easy. The BSP is still a very young protocol. There is a thought to add support for custom rules: two different approaches. One is to define a set of known providers, but they need to be quite language specific. The other one is to add an extra option to plug into the existing mechanism by providing additional aspects or similar. Old plugin: * Fully supported until mid-2025; reevaluation after that. CLion support may follow a different timeline. * Materialize code generation so that you can investigate e.g. protobuf generated code when you navigate to an import from the code. There is a plugin for protobuf already. * Google's involvement in the old plugin is mostly around Android Studio integration. * Query sync came from Android Studio, but it should now work for IntelliJ and CLion. Needs to be manually enabled in the settings. Important to also enable "Automatically sync project on file changes" in the settings. Once you navigate to a file, some features may not work: for those, you must manually "Enable analysis" for them. Adding a file is now super-fast (after adding it to the build files), but if build files use globs, files are immediately added (the sync runs in the background automatically). Usability concerns: will users understand this? "Compiling protobuf" goes away by switching to bzlmod because with bzlmod we end up using the prebuilt binaries. ## Fast, correct, seamless - Choose 3 Allow users to use native tooling for Go in the shell. Created a rule to fetch necessary stuff and generate a `.env` file to make the Go tools work (same for the VSCode settings). Using `direnv` to make this transparent. ## Optimizing Gazelle Backend team at Uber. Go and Java. Adopting Gazelle for everything, so it must be fast. 10k commits/week, 1M LOC files. 90% of the changes in 10 files. Needed to optimize commits from O(size of repo) to O(size of change). Gazelle was eating 40% of the CI time. Implement conventions (paths, etc.) and heuristics to avoid having to do the "real work". Eliminated the Gazelle index with this. Try profiling Gazelle with your codebase (there is a flag upstreamed for it). You might find different bottlenecks than we did. Prebuild gazelle and other tools so that fresh hosts don't need to rebuild it. (Why doesn't remote caching help though?)