How "new type" helps avoid production outages

My January links recap included the “Phantom Types” article by David Soria Parra. In it, the author briefly touches upon the “new type” idiom, its typical implementation in Rust, and then proceeds to propose a better alternative. But the question arises: why should you care? To answer why this idiom is useful, I want to present you with a real production problem we faced in the Storage Infrastructure team at Google circa 2010.

March 9, 2024 · Tags: blogsystem5, rust, sre, twitter-thread
Continue reading (about 4 minutes)

To C or not to C

Over the last few days, there has been this… debate over at Twitter sparked by a claim that you cannot be a good programmer without knowing C. You obviously can be one, but there is some nuance in what “knowing” C is truly about. Here is my take on the matter. Let me repeat this first: of course you can be a perfectly good programmer without knowing C. Knowing a language doesn’t make or break a programmer, and there are great programmers out there that don’t touch C.

February 21, 2024 · Tags: blogsystem5, opinion, programming, twitter-thread
Continue reading (about 4 minutes)

Hard disk LEDs and noisy machines

The computers of yesteryear had this little feature known as blinking LED lights ๐Ÿ”†. They also had this other feature called noisy disks ๐Ÿ’พ and loud fans ๐Ÿชญ. Uh wait. Features? Why “features” and not “annoyances”?! ๐Ÿงต๐Ÿ‘‡ Front panel of a common PC case in the late 1990s. My Pentium MMX 166 was hosted in one of these. You see, these bright lights and loud noises acted as canaries ๐Ÿฆ in a performance mine.

December 15, 2023 · Tags: blogsystem5, history, twitter-thread
Continue reading (about 5 minutes)

Code reviews: A success story

Code reviews have a bad rep: they are antagonistic in nature and, sometimes, pure red tape. Some argue that they are bad practice; others say that peer programming is better. And while these may be true, I want to tell you a story about a case where code reviews worked well! Meet X: a junior engineer in the Bazel team circa 2018, tasked to implement two features: A and B. As you may know, Google is big into code reviews—and their tooling for this is awesome; believe me—so this was the standard process for X to get his code checked in.

November 21, 2023 · Tags: programming, twitter-thread
Continue reading (about 4 minutes)

Good performance is not just big O

Having a fast and responsive app is orthogonal to “knowing your big Os”. Unfortunately, most tech companies over-emphasize algorithms in interviews and downplay systems knowledge, and I believe that’s one reason behind sluggish apps and bloated systems. I’ve seen this play out repeatedly. Interviewers ask a LeetCode-style coding question, which is then followed by the ritual of discussing time and memory complexity. Candidates ace the answers. But then… their “real” code suffers from subtle yet impactful performance problems. Focusing on big O complexity rarely matters in most apps. Sure, it’s important to think about your algorithmic choices, but there are so many more details to worry about that have a direct impact on app performance and responsiveness. Let’s look at a bunch of them!

September 8, 2023 · Tags: programming, sre, twitter-thread
Continue reading (about 6 minutes)

Fast machines, slow machines

Well, that was unexpected. I recorded a couple of crappy videos in 5 minutes, posted them on a Twitter thread, and went viral with 8.8K likes at this point. I really could not have predicted that, given that I’ve been posting what-I-believe-is interesting content for years and… nothing, almost-zero interest. Now that things have cooled down, it’s time to stir the pot and elaborate on those thoughts a bit more rationally. To summarize, the Twitter thread shows two videos: one of an old computer running Windows NT 3.51 and one of a new computer running Windows 11. In each video, I opened and closed a command prompt, File Explorer, Notepad, and Paint. You can clearly see how apps on the old computer open up instantly whereas apps on the new computer show significant lag as they load. I questioned how computers are actually getting better when trivial things like this have regressed. And boom, the likes and reshares started coming in. Obviously some people had issues with my claims, but there seems to be an overwhelming majority of people that agree we have a problem. To open up, I’ll stand my ground: latency in modern computer interfaces, with modern OSes and modern applications, is terrible and getting worse. This applies to smartphones as well. At the same time, while UIs were much more responsible on computers of the past, those computers were also awful in many ways: new systems have changed our lives substantially. So, what gives?

June 27, 2023 · Tags: featured, opinion, twitter-thread
Continue reading (about 16 minutes)

Principal engineers should be on-call

A recent tweet that caught my attention read: “principal engineers should be on-call”. Of course they should be! I’m “surprised” they aren’t everywhere, but I can imagine some reasons to justify their situation. Let’s change that in this thread. ๐Ÿงต ๐Ÿ‘‡

July 14, 2021 · Tags: opinion, sre, twitter-thread
Continue reading (about 4 minutes)

Running a healthy production service

In a previous thread, I covered some techniques to approach on-call shifts and maintain your own well-being. In this thread, I will touch upon the things you can do, as a team, to make your service more sustainable. ๐Ÿงต ๐Ÿ‘‡

June 18, 2021 · Tags: sre, twitter-thread
Continue reading (about 4 minutes)

On software deadlines

In my career so far (which isn’t that long… but also not that short ๐Ÿ˜‰), I’ve been lucky enough to not be subject to strict deadlines. That’s why I’m always in awe every time I talk to fellow long-time Microsoft employees. A thread! ๐Ÿงต

April 20, 2021 · Tags: twitter-thread
Continue reading (about 3 minutes)

Always be quitting

A good philosophy to live by at work is to “always be quitting”. No, don’t be constantly thinking of leaving your job ๐Ÿ˜ฑ. But act as if you might leave on short notice ๐Ÿ˜Ž. Counterintuitively, this will make you a better engineer and open up growth opportunities. A thread ๐Ÿ‘‡.

April 12, 2021 · Tags: featured, opinion, twitter-thread
Continue reading (about 4 minutes)

Tips on well-being while on-call

Last week, I was first-time on-call for a part of Azure Storage. My previous background as an SRE at Google helped me remain calm despite my inexperience. And as we have more first-time on-callers joining soon, I couldn’t resist writing some advice for them. Let’s start! ๐Ÿงต

March 10, 2021 · Tags: sre, twitter-thread
Continue reading (about 4 minutes)

Down the memory lane with OS/2

OS/2 Warp 3 was the operating system that led me into an adventure back in 1994. This OS made me escape MS-DOS and Windows, and then made me enter the Linux world when I couldn’t afford Warp 4. Let’s walk down the memory lane!

August 4, 2020 · Tags: os2, retrocomputing, twitter-thread
Continue reading (about 7 minutes)

My 18 steps to prepare a great presentation

In the previous post, I shared my tips to give great presentations. But how do I prepare to give one? That’s a different story, so let’s take a look at my complete process.

July 17, 2020 · Tags: twitter-thread
Continue reading (about 4 minutes)

My 18 tips to deliver great presentations

I used to dread public speaking in middle and high school. College was OKish. Now… even though Iโ€™m far from an expert, I really enjoy it. How? I forced myself to give talks and strive for improvement in each of them. Here are the things I learned and do.

July 13, 2020 · Tags: twitter-thread
Continue reading (about 4 minutes)