Blog

Code reviews: A success story

Code reviews have a bad rep: they are antagonistic in nature and, sometimes, pure red tape. Some argue that they are bad practice; others say that peer programming is better. And while these may be true, I want to tell you a story about a case where code reviews worked well! Meet X: a junior engineer in the Bazel team circa 2018, tasked to implement two features: A and B. As you may know, Google is big into code reviews—and their tooling for this is awesome; believe me—so this was the standard process for X to get his code checked in.

November 21, 2023 · Tags: programming, twitter-thread
Continue reading (about 4 minutes)

SSH agent forwarding and tmux done right

The SSH agent is a little daemon that holds your private keys in memory. This is particularly handy when your keys are protected by a passphrase: you can unlock and add your keys to the agent once and, from then on, any SSH client such as ssh(1) can interact with the keys without asking you for the passphrase again. The SSH agent becomes even handier when you primarily work on a remote workstation over SSH.

November 17, 2023 · Tags: blogsystem5, unix
Continue reading (about 9 minutes)

Why do I know shell, and how can you?

“Why do you know so much shell?” is a question I’m getting a lot at work lately. So yeah, why? And how can you learn it too? There is no secret here: I know the shell well because I was “forced” to write tools in it for a while and, because of that, I made a conscious effort to learn the language and get better at it.

You see, most people that write shell don’t want to deal with it. They stitch together whatever works into a script and call it a day, making a bunch of spaghetti even if it goes against the coding best practices they already know. And when they encounter some odd syntax they don’t recognize, their reaction is to say “this has to be rewritten in Python!” instead of taking a breath and trying to really understand what’s going on. It doesn’t help that plenty of senior engineers scoff at shell scripts.

And it is true: the shell is arcane and has many flaws as a programming language. I don’t want to convince you to start writing new tools in it. But the shell is also an incredible rapid prototyping language, and you can use it to solve business problems really quickly and with surprisingly little code. If you pause for a second to learn it, you’ll realize that you can bend tradition and write maintainable shell code too. Hear out how I got into writing so much shell and how you can get better at it too.

November 10, 2023 · Tags: blogsystem5, personal-story, shell
Continue reading (about 7 minutes)

End-to-end tool testing with Bazel and shtk

If you use Bazel, your project is of moderate size. And because your project is of moderate size, it almost-certainly builds one or more binaries, at least one of which is a CLI tool. But let’s face it: you don’t have end-to-end testing for those tools, do you?

I’m sure you have split the binary’s main function into its own file so that the rest of the tool can be put in a library, and I’m extra-sure that you have unit tests for such library. But… those tests do little to verify the functionality and quality of the tool as experienced by the end user. Consider: What exactly does the tool print to the console on success? Does it show errors nicely when they happen, or does it dump internal stack traces? How does it handle unknown flags or bad arguments? Is the built-in help message nicely rendered when your terminal is really wide? What if the terminal is narrow?

You must write end-to-end tests for your tools but, usually, that isn’t easy to do. Until today. Combining shtk with Bazel via the new rules_shtk ruleset makes it trivial to write tests that verify the behavior of your CLI tools—no matter what language they are written in—and in this article I’m going to show you how.

November 4, 2023 · Tags: bazel, blogsystem5, shell, shtk, testing
Continue reading (about 7 minutes)

Links: October 2023 edition

Inspired by the works of Phil Eaton, I’ve been highlighting articles and projects that I find interesting in Twitter X and Mastodon. Some of these posts were more “successful” than I had expected, which I take to mean that doing this is interesting to you all. So, it’s probably a good idea to periodically collect them all in a post with a very brief commentary on each.

Here is a recap of the interesting articles that came my way in October 2023. This does not mean that these articles were published during this period: some of them are older but I just (re)discovered them now. I’ll avoid referencing my own articles: you can find those by in the archive.

October 31, 2023 · Tags: blogsystem5, recap
Continue reading (about 5 minutes)

BazelCon 2023 et al. trip report

I’m exhausted. I just came back to Seattle from a 10-day trip in which I attended three different Bazel events: the Build Meetup in Reykjavik, the Bazel Community Day in Munich, and BazelCon 2023 in Munich too. Oh, and because I was on the other side of the world, I also paid a visit to my family in Spain.

Attending these events has been incredibly useful and productive: I got exposure to many ideas and discussions that would just not happen online, I got to build connections with very interesting people and, of course, it has also been super fun too to reconnect with old coworkers and friends.

This article contains the summary of the things I learned and the things I want to follow up on. These are just a bunch of cleaned-up notes which I took and are in the context of my work with Bazel at Snowflake and my interests on build tools, so this is not endorsed by Snowflake.

October 30, 2023 · Tags: bazel, blogsystem5, snowflake
Continue reading (about 15 minutes)

Hello, Blog System/5!

Blog System/5 is my new Substack publication in which I write about the variety of software and systems engineering topics that pique my interest. If that sounds too generic to you, it’s because it is: there are too many cool things to write about! And yes, this sounds exactly the same as this blog. Read on to understand the motivation behind the new publication and subscribe now to not miss a beat!

October 27, 2023 · Tags: blog, blogsystem5
Continue reading (about 3 minutes)

Build farm visualizations

If you have followed our recent infrastructure posts, you know by now that we are actively migrating Snowflake’s build to Bazel. What we haven’t shared yet is that we have deployed our own Build Barn cluster to support Bazel’s remote execution features. We have chosen to run our own build farm service for resource governance and security purposes, but also because the behavior of this system impacts the developer experience so directly that we want to have full in-house control and knowledge of it.

October 20, 2023 · Tags: bazel, snowflake
Continue reading (about 10 minutes)

Unit-testing shell scripts and tools with shtk

While working on this static blog a few days ago, I made a change to its templates that warranted an automated test. I could have written a trivial shell script to do it, but instead I reached out for shtk’s unit-testing module. I tweeted about it right away to just say that you can, in fact, write tests in shell because lots of developers are skeptical about any script longer than 10 lines of code.

Interestingly, this reply came through: a pointer to a contemporary, under-development library for writing tests in Bash. Which made me think: “Hey, I had already done that years ago… but nobody knows about it. Gotta fix that with a blog post!” But first, I had to bring shtk back from its ashes because I had not touched it for more than 6 years and it wasn’t read for show and tell. So I did something that I wanted to do back in the day but never did: I put together a website for shtk to host its reference manual and I fixed a few obvious rough edges.

With those tweaks out of the way, we come to this article. In here, I want to show you how writing decent tests in shell is entirely possible and how shtk’s testing platform provides unique features to do integration testing of CLI apps written in any language.

October 11, 2023 · Tags: shell, shtk, testing
Continue reading (about 12 minutes)

Analyzing OOMs in IntelliJ with Bazel

A few months ago, we described how we fixed three different OOM scenarios in our ongoing migration to the Bazel build system here at Snowflake. Things had been sailing along just fine since then… but a new issue showed up recently: our IntelliJ with Bazel (IjwB) Java project started showing OOMs during its sync phase.

The reason this issue surfaced now is because, as we continue our migration to Bazel, our IjwB project has grown in size. Months ago, our project only covered a Java binary, but now that we have migrated all of its unit and integration tests as well, the project covers them too. It is common for tests to be more expensive to build and run than the binary they validate—tests depend on the binary’s dependencies plus many other helper tools for testing—and these caused the project to grow too big to fit in our development environments. Or did they?

October 7, 2023 · Tags: bazel, snowflake
Continue reading (about 9 minutes)