Showing 30 posts
My interest in storage is longstanding—I loved playing with different file systems in my early Unix days and then I worked on Google’s and Microsoft’s distributed storage solutions—and, about four years ago, I started running a home-grown NAS leveraging FreeBSD and its excellent ZFS support. I first hosted the server on a PowerMac G5 and then upgraded it to an overkill 72-core ThinkStation that I snapped second-hand for a great price. But as stable and low maintenance as FreeBSD is, running day-to-day services myself is not my idea of “fun”. This drove me to replace this machine’s routing functionality with a dedicated pfSense box a year ago and, for similar reasons, I have been curious about dedicated NAS solutions. I was pretty close to buying a second-hand NAS from the work classifieds channel when a Synology marketing person (hi Kyle!) contacted me to offer a partnership: they’d ship me one of their devices for free in exchange for me publishing a few articles about it. Given my interest to drive-test one of these appliances without committing to buying one (they ain’t cheap and I wasn’t convinced I wanted to get rid of my FreeBSD-based solution), I was game. And you guessed right: this article is one of those I promised to write but, before you stop reading, the answer is no. This post is not sponsored by Synology and has not been reviewed nor approved by them. The content here, including any opinions, are purely my own. And what I want do do here is compare how the Synology appliance stacks against my home-built FreeBSD server.
December 13, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/hardware">hardware</a>
Continue reading (about
17 minutes)
I recently got a Synology DS923+ for evaluation purposes which led me to setting up NFSv4 with Kerberos. I had done this about a year ago with FreeBSD as the host, and going through this process once again reminded me of how painful it is to secure an NFS connection. You see, Samba is much easier to set up, but because NFS is the native file sharing protocol of Unix systems, I felt compelled to use it instead. However, if you opt for NFSv3 (the “easy default”), you are left with a system that has zero security: traffic travels unencrypted and unsigned, and the server trusts the client when the client asserts who is who. Madness for today’s standards. Yet, when you look around, people say “oh, but NFSv3 is fine if you trust the network!” But seriously, who trusts the network in this day and age? You have to turn to NFSv4 and combine it with Kerberos for a secure file sharing option. And let me tell you: the experience of setting these up and getting things to work is horrible, and the documentation out there is terrible. Most documents are operating-system specific so they only tell you what works when a specific server and a specific client talk to each other. Other documents just assume, and thus omit, various important details of the configuration. So. This article is my recollection of “lab notes” on how to set this whole thing up along with the necessary background to understand NFSv4 and Kerberos. My specific setup involes the Synology DS923+ as the NFSv4 server; Fedora, Debian, and FreeBSD clients; and the supporting KDC on a pfSense (or FreeBSD) box.
November 3, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/unix">unix</a>
Continue reading (about
24 minutes)
Just like that, BazelCon 2024 came and went. So… it’s obviously time to summarize the two events of last week: BazelCon 2024 and the adjacent Build Meetup. There is A LOT to cover, but everything is here in just one article!
October 22, 2024
·
Tags:
<a href="/tags/bazel">bazel</a>, <a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/snowflake">snowflake</a>
Continue reading (about
42 minutes)
If you read my previous article on DOS memory models, you may have dismissed everything I wrote as “legacy cruft from the 1990s that nobody cares about any longer”. After all, computers have evolved from sporting 8-bit processors to 64-bit processors and, on the way, the amount of memory that these computers can leverage has grown orders of magnitude: the 8086, a 16-bit machine with a 20-bit address space, could only use 1MB of memory while today’s 64-bit machines can theoretically access 16EB. All of this growth has been in service of ever-growing programs. But… even if programs are now more sophisticated than they were before, do they all really require access to a 64-bit address space? Has the growth from 8 to 64 bits been a net positive in performance terms? Let’s try to answer those questions to find some very surprising answers. But first, some theory.
October 7, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/hardware">hardware</a>, <a href="/tags/unix">unix</a>
Continue reading (about
18 minutes)
At the beginning of the year, I wrote a bunch of articles on the various tricks DOS played to overcome the tight memory limits of x86's real mode. There was one question that came up and remained unanswered: what were the various models that the compilers of the day offered? Tiny, small, medium, compact, large, huge... What did these options mean? What were their effects? And, more importantly... is any of that legacy relevant today in the world of 64-bit machines and gigabytes of RAM? To answer those questions, we must start with a brief review of the 8086 architecture and the binary formats supported by DOS.
September 30, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/dos">dos</a>
Continue reading (about
10 minutes)
Over the years, I’ve repeatedly heard that Windows NT is a very advanced operating system and, being a Unix person myself, it has bothered me to not know why. I’ve been meaning to answer this question for years and I can do so now, which means I want to present you my findings. My desire to know about NT’s internals started in 2006 when I applied to the Google Summer of Code program to develop Boost.Process. I needed such a library for ATF, but I also saw the project as a chance to learn something about the Win32 API. This journey then continued in 2020 with me choosing to join Microsoft after a long stint at Google and me buying the Windows Internals 5th edition book in 2021 (which I never fully read due to its incredible detail and length). None of these made me learn what I wanted though: the ways in which NT fundamentally differs from Unix, if at all.
September 9, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/unix">unix</a>, <a href="/tags/windows">windows</a>
Continue reading (about
23 minutes)
In a recent work discussion, I came across an argument that didn’t sound quite right. The claim was that we needed to set up containers in our developer machines in order to run tests against a modern glibc. The justifications were that using LD_LIBRARY_PATH to load a different glibc didn’t work and statically linking glibc wasn’t possible either. But… running a program against a version of glibc that’s different from the one installed on the system seems like a pretty standard requirement, doesn’t it? Consider this: how do the developers of glibc test their changes? glibc has existed for much longer than containers have. And before containers existed, they surely weren’t testing glibc changes by installing modified versions of the library over the system-wide one and YOLOing it.
August 11, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/programming">programming</a>, <a href="/tags/unix">unix</a>
Continue reading (about
11 minutes)
After years of inactivity, the Kyua project has graduated as an open source citizen and has a new home under the FreeBSD umbrella! But uh… wait, what is Kyua and why is this exciting? To resolve confusion and celebrate this milestone, I’d like to revisit what Kyua is, how it came to be, why I stopped working on it for a while, why that was a problem for FreeBSD—and, indirectly, NetBSD—and how Kyua being free software has helped keep it alive.
August 2, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about
14 minutes)
Look, I like Rust. I really, really do, and I agree with the premise that memory-unsafe languages like C++ should not be used anymore. But claiming that Rust would have prevented the massive outage that the world went through last Friday is misleading and actively harmful to Rust’s evangelism. Having CrowdStrike written in Rust would have minimized the chances of the outage happening, but not resolved the root cause that allowed the outage to happen in the first place. Thus, it irks me to see various folks blanket-claiming that Rust is the answer. It’s not, and pushing this agenda hurts Rust’s adoption more than it helps: C++ experts can understand the root cause and see that this claim is misleading, causing further divide in the systems programming world. So, why won’t Rust help? Let me try to answer that question, but while we are at it, let’s also delve deeper into the causes of the outage. In a way, let me put my SRE hat on and write my own version of the postmortem.
July 23, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/opinion">opinion</a>, <a href="/tags/rust">rust</a>
Continue reading (about
12 minutes)
Blog System/5 hasn’t always been called this way and it hasn’t been my first experience with blogging either. In fact, today marks the 20th anniversary of this publication in its various incarnations so it’s time for a bit of reflection. Just to set context for when 20 years ago was: Windows XP was almost 3 years old, Ubuntu had just debuted, Apple computers were still PowerPC-based, Half Life 2 was about to launch, and Slashdot was the place to be instead of the yet-to-be-created Hacker News. As for myself, I was still in college, had copious amounts of free time, and was a really active contributor to NetBSD.
June 22, 2024
·
Tags:
<a href="/tags/blog">blog</a>, <a href="/tags/blogsystem5">blogsystem5</a>
Continue reading (about
14 minutes)
Hello again Blog System/5 and sorry for the radio silence for the last couple of months. I had been writing too much in here and neglecting my side projects so I needed to get back to them. And now that I’ve made significant progress on cool new features for EndBASIC, it’s time to write about them a little! One of the defining characteristics of EndBASIC is its hybrid console: what looks like a simple text terminal at first glance can actually render overlapping graphics and text at the same time. This is a feature that I believe is critical to simplify learning and it first appeared with the 0.8 release back in 2021.
April 26, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/endbasic">endbasic</a>, <a href="/tags/rust">rust</a>
Continue reading (about
16 minutes)
My January links recap included the “Phantom Types” article by David Soria Parra. In it, the author briefly touches upon the “new type” idiom, its typical implementation in Rust, and then proceeds to propose a better alternative. But the question arises: why should you care? To answer why this idiom is useful, I want to present you with a real production problem we faced in the Storage Infrastructure team at Google circa 2010. That issue made me a convert and I’ve kept it in mind when designing APIs since then.
March 9, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/rust">rust</a>, <a href="/tags/sre">sre</a>, <a href="/tags/twitter-thread">twitter-thread</a>
Continue reading (about
4 minutes)
Hi folks! Another month has passed so it’s time for a brief recap of the main news, articles, and projects that made the rounds during this period and are on topic for Blog System/5. As usual, this is not just a list: every entry is accompanied by a short blurb detailing why I found the content interesting, which is meant to nudge you into reading it! Also, the list is ordered by when the links made it my way, not chronologically, and some items are not from this time period.
February 29, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/recap">recap</a>
Continue reading (about
8 minutes)
Over the last few days, there has been this… debate over at Twitter sparked by a claim that you cannot be a good programmer without knowing C. You obviously can be one, but there is some nuance in what “knowing” C is truly about. Here is my take on the matter. Let me repeat this first: of course you can be a perfectly good programmer without knowing C. Knowing a language doesn’t make or break a programmer, and there are great programmers out there that don’t touch C. However, knowing C says something about your journey.
February 21, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/opinion">opinion</a>, <a href="/tags/programming">programming</a>, <a href="/tags/twitter-thread">twitter-thread</a>
Continue reading (about
4 minutes)
The recent deep dive into the IDEs of the DOS times 30 years ago made me reminisce of DJGPP, a distribution of the GNU development tools for DOS. I remember using DJGPP back in the 1990s before I had been exposed to Linux and feeling that it was a strange beast. Compared to the Microsoft C Compiler and Turbo C++, the tooling was bloated and alien to DOS, and the resulting binaries were huge. But DJGPP provided a complete development environment for free, which I got from a monthly magazine, and I could even look at its source code if I wished. You can’t imagine what a big deal that was at the time. But even if I could look under the cover, I never did. I never really understood why was DJGPP so strange, slow, and huge, or why it even existed. Until now. As I’m in the mood of looking back, I’ve spent the last couple of months figuring out what the foundations of this software were and how it actually worked. Part of this research has resulted in the previous two posts on DOS memory management. And part of this research is this article. Let’s take a look!
February 14, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/dos">dos</a>
Continue reading (about
20 minutes)
In “From 0 to 1 MB in DOS”, I presented an overview of all the ways in which DOS and its applications tried to maximize the use of the 1 MB address space inherited from the 8086—even after the 80286 introduced support for 16 MB of memory and the 80386 opened the gates to 4 GB. I know I promised that this follow-up article would be about DJGPP, but before getting into that review, I realized I had to take another detour to cover three more topics. Namely: unreal mode, which I intentionally ignored to not derail the post; LOADALL, which I didn’t know about until you readers mentioned it; and DOS extenders, which I was planning to describe in the DJGPP article but they are a better fit for this one. So… strap your seat belts on and dive right in for another tour through the ancient techniques that DOS had to pull off to peek into the memory address space above the first MB. And get your hands ready because we’ll go over assembly code for a step-by-step jump into unreal mode.
February 7, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/dos">dos</a>
Continue reading (about
24 minutes)
It is hard to believe but we are already one month into 2024. January has flown by for me and I haven’t done a good job at keeping up with news sites… but I have been reading them on and off and I have collected a small set of interesting articles. To everyone new around here, hello and thanks for subscribing! For some context, what follows is my manual selection of cool articles, videos, and projects I stumbled upon during this time period. However, this is not just a dump of links: each link is accompanied by a short commentary to justifies why I thought the material was interesting, why it is relevant to this publication and, more importantly, an attempt to nudge you into reading the source. Let’s get to it.
January 31, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/recap">recap</a>
Continue reading (about
7 minutes)
Since the last article on the text-based IDEs of old, I’ve been meaning to write about the GCC port to DOS, namely DJGPP. As I worked on the draft for that topic, I realized that there is a ton of ground to cover to set the stage so I took most of the content on memory management out and wrote this separate post. This article is a deep dive on how DOS had to pull out tricks to maximize the use of the very limited 1 MB address space of the 8086. Those tricks could exist because of the features later introduced by the 80286 and the 80386, but these were just clutches to paper over the fact that DOS could not leverage the real improvements provided by protected mode. This detour is long but I hope you’ll enjoy it as much as I enjoyed researching the topic. I’ll walk you through the changes in the x86 architecture over time, starting with the 8086 and ending in the 80386, and how DOS kept up along the way. I’ll conclude with a peek into DOS’ own MEM and MemMaker utilities. I must omit details to keep the text manageable in size though, so please excuse the lack of detail in some areas; just follow the links to external documentation to learn more.
January 17, 2024
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/dos">dos</a>
Continue reading (about
18 minutes)
December draws to a close as does 2023, which means it’s time for yet another monthly links recap. For context to everyone new around here, what follows is my manual curation of cool articles, videos, and projects I stumbled upon during this time period. But this is not just a dump of links: each link is accompanied by a short commentary that justifies why I thought the material was interesting, why it is relevant to this publication and, more importantly, an attempt to nudge you into reading it.
December 31, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/recap">recap</a>
Continue reading (about
7 minutes)
I grew up learning to program in the late 1980s / early 1990s. Back then, I did not fully comprehend what I was doing and why the tools I used were impressive given the constraints of the hardware we had. Having gained more knowledge throughout the years, it is now really fun to pick up DOSBox to re-experience those programs and compare them with our current state of affairs. This time around, I want to look at the pure text-based IDEs that we had in that era before Windows eclipsed the PC industry. I want to do this because those IDEs had little to envy from the IDEs of today—yet it feels as if we went through a dark era where we lost most of those features for years and they are only resurfacing now. If anything, stay for a nostalgic ride back in time and a little rant on “bloat”. But, more importantly, read on to gain perspective on what existed before so that you can evaluate future feature launches more critically.
December 25, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/history">history</a>
Continue reading (about
12 minutes)
Just a bit over 2 months ago, on October 5th, 2023, Jordi Mon Companys interviewed me about Bazel for an episode in the Software Engineering Daily podcast. The episode finally came out on December 18th, 2023, so here is your announcement to stop by and listen to it! Cover image (and link) to the Bazel interview in Software Engineering Daily. If you don’t have time to listen to the whole 45 minutes, or if you want to get a sense of what you will get out of it, here is a recap of everything we touched on. Every paragraph is annotated with the rough time where the discussion starts so that you can jump right in to whatever interests you the most.
December 21, 2023
·
Tags:
<a href="/tags/bazel">bazel</a>, <a href="/tags/blogsystem5">blogsystem5</a>
Continue reading (about
5 minutes)
The computers of yesteryear had this little feature known as blinking LED lights π. They also had this other feature called noisy disks πΎ and loud fans πͺ. Uh wait. Features? Why “features” and not “annoyances”?! π§΅π Front panel of a common PC case in the late 1990s. My Pentium MMX 166 was hosted in one of these. You see, these bright lights and loud noises acted as canaries π¦ in a performance mine. They gave developers a chance to notice when things were off performance-wise. If your code abused the CPU or the hard disk by mistake, you could tell right away.
December 15, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/history">history</a>, <a href="/tags/twitter-thread">twitter-thread</a>
Continue reading (about
5 minutes)
Just yesterday, Twitter user @vkrajacic wrote: Advice for new C programmers: “Avoid null-terminated strings; they’re outdated, inefficient and impractical.” Create your own type with basic functions. It’s not that hard, and it goes a long way. One of the benefits of this approach, among others, is slicing without copying. This suggestion has its merits and I understand where it is coming from: performance. You see: the traditional way to represent strings in C is to use NUL-terminated byte arrays. Yet… this has deemed to be the most expensive one-byte mistake because of the adverse performance implications that this carries. (NUL, not NULL, is the better name for the \0 byte by the way.)
December 3, 2023
·
Tags:
<a href="/tags/bazel">bazel</a>, <a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/java">java</a>
Continue reading (about
4 minutes)
Welcome to the second edition of my “interesting links” recap, this time covering the month of November 2023. For context, what follows is my manual curation of cool articles, videos, and projects I stumbled upon during this time period. But this is not just a dump of links: each link is accompanied by a 1-paragraph commentary that justifies why I thought the material was cool, why it is relevant to this publication and, more importantly, an attempt to nudge you into reading it.
November 30, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/recap">recap</a>
Continue reading (about
12 minutes)
The SSH agent is a little daemon that holds your private keys in memory. This is particularly handy when your keys are protected by a passphrase: you can unlock and add your keys to the agent once and, from then on, any SSH client such as ssh(1) can interact with the keys without asking you for the passphrase again. The SSH agent becomes even handier when you primarily work on a remote workstation over SSH. Under these circumstances, you will often need the remote workstation to establish SSH connections to other remote machines (e.g. to contact GitHub). In those situations, you can: copy your private keys to the remote workstation; generate different private keys on the remote workstation; or forward your SSH agent so that the remote workstation can leverage the keys from your client machine without them ever traveling over the network.
November 17, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/unix">unix</a>
Continue reading (about
9 minutes)
“Why do you know so much shell?” is a question I’m getting a lot at work lately. So yeah, why? And how can you learn it too? There is no secret here: I know the shell well because I was “forced” to write tools in it for a while and, because of that, I made a conscious effort to learn the language and get better at it. You see, most people that write shell don’t want to deal with it. They stitch together whatever works into a script and call it a day, making a bunch of spaghetti even if it goes against the coding best practices they already know. And when they encounter some odd syntax they don’t recognize, their reaction is to say “this has to be rewritten in Python!” instead of taking a breath and trying to really understand what’s going on. It doesn’t help that plenty of senior engineers scoff at shell scripts. And it is true: the shell is arcane and has many flaws as a programming language. I don’t want to convince you to start writing new tools in it. But the shell is also an incredible rapid prototyping language, and you can use it to solve business problems really quickly and with surprisingly little code. If you pause for a second to learn it, you’ll realize that you can bend tradition and write maintainable shell code too. Hear out how I got into writing so much shell and how you can get better at it too.
November 10, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/personal-story">personal-story</a>, <a href="/tags/shell">shell</a>
Continue reading (about
7 minutes)
If you use Bazel, your project is of moderate size. And because your project is of moderate size, it almost-certainly builds one or more binaries, at least one of which is a CLI tool. But let’s face it: you don’t have end-to-end testing for those tools, do you? I’m sure you have split the binary’s main function into its own file so that the rest of the tool can be put in a library, and I’m extra-sure that you have unit tests for such library. But… those tests do little to verify the functionality and quality of the tool as experienced by the end user. Consider: What exactly does the tool print to the console on success? Does it show errors nicely when they happen, or does it dump internal stack traces? How does it handle unknown flags or bad arguments? Is the built-in help message nicely rendered when your terminal is really wide? What if the terminal is narrow? You must write end-to-end tests for your tools but, usually, that isnβt easy to do. Until today. Combining shtk with Bazel via the new rules_shtk ruleset makes it trivial to write tests that verify the behavior of your CLI tools—no matter what language they are written in—and in this article Iβm going to show you how.
November 4, 2023
·
Tags:
<a href="/tags/bazel">bazel</a>, <a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/shell">shell</a>, <a href="/tags/shtk">shtk</a>, <a href="/tags/testing">testing</a>
Continue reading (about
7 minutes)
Inspired by the works of Phil Eaton, I’ve been highlighting articles and projects that I find interesting in Twitter X and Mastodon. Some of these posts were more “successful” than I had expected, which I take to mean that doing this is interesting to you all. So, it’s probably a good idea to periodically collect them all in a post with a very brief commentary on each. Here is a recap of the interesting articles that came my way in October 2023. This does not mean that these articles were published during this period: some of them are older but I just (re)discovered them now. I’ll avoid referencing my own articles: you can find those by in the archive.
October 31, 2023
·
Tags:
<a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/recap">recap</a>
Continue reading (about
5 minutes)
I’m exhausted. I just came back to Seattle from a 10-day trip in which I attended three different Bazel events: the Build Meetup in Reykjavik, the Bazel Community Day in Munich, and BazelCon 2023 in Munich too. Oh, and because I was on the other side of the world, I also paid a visit to my family in Spain. Attending these events has been incredibly useful and productive: I got exposure to many ideas and discussions that would just not happen online, I got to build connections with very interesting people and, of course, it has also been super fun too to reconnect with old coworkers and friends. This article contains the summary of the things I learned and the things I want to follow up on. These are just a bunch of cleaned-up notes which I took and are in the context of my work with Bazel at Snowflake and my interests on build tools, so this is not endorsed by Snowflake.
October 30, 2023
·
Tags:
<a href="/tags/bazel">bazel</a>, <a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/snowflake">snowflake</a>
Continue reading (about
15 minutes)
Blog System/5 is my new Substack publication in which I write about the variety of software and systems engineering topics that pique my interest. If that sounds too generic to you, it’s because it is: there are too many cool things to write about! And yes, this sounds exactly the same as this blog. Read on to understand the motivation behind the new publication and subscribe now to not miss a beat!
October 27, 2023
·
Tags:
<a href="/tags/blog">blog</a>, <a href="/tags/blogsystem5">blogsystem5</a>
Continue reading (about
3 minutes)