BSD vs. Linux

Here we are, getting ready to discuss BSD vs. Linux. Why would I do this though? This topic been beaten to death time and time again. But I routinely witness articles and arguments on the topic and the vast majority completely miss the point on what truly makes a difference between the two systems.

In this article, I would like to present my personal take on how BSD systems compare to Linux. I will do so by discussing first the various areas in which everyone seems to focus, and I will finish by discussiong where I think the real difference lies.

Disclaimers first: in case you don't know, I'm a BSD advocate. Everything but the last section will hopefully read neutral, but I am going to conclude (spoilers!) vowing in favor of BSD.

Be aware that pretty much everywhere I say Linux I really mean GNU/Linux. I am not comparing kernels: I am comparing operating systems: that is, the "thing" that you can install on your machine to actually do stuff with it.

And also please note that I am vastly generalizing below. If you are one of the few people that truly cares about some specific detail, that's just fine: opinions are like colors (TODO: verify quote). I'd like to ask you to continue reading anyway with an open mind.

Grandiose features

A common argument in favor of the BSDs is that they provide very nice pieces of technology and don't get me wrong: they really, really do. For example: OpenBSD has the pf packet filter, which various other systems have adopted; FreeBSD has ZFS and DTrace, both of which originated from Sun but are killer features; NetBSD has a very nice cross-build system and rump; and on and on.

But "so what?" Linux has many nice features that the BSDs don't have as well. If you want ZFS, you have btrfs. If you want DTrace, you have systemtap. If you want pf, well... maybe you have nothing comparable, but that doesn't matter a little bit to the vast majority of users..

Technologies can be replicated pretty easily. With enough manpower (and that's something that Linux doesn't lack), any sufficiently-good and widely-appreciated technology will be implemented in Linux relatively quickly. It's a simple matter of programming (SMOP) really, and that can never be your differentiator in the market.

So features, as a difference, is something that matters little. BSD advocates will continue to see their own technologies as the best while Linux advocates will do the same on their part. I do not have to dig examples in recent history detailing how vastly-superior technological products failed in favor of demonstrably-worse products; it's obvious. When the time comes to implement a production service, most system administrators will favor the most widespread technology.

Code cleanliness, portability, etc.

Another common argument in favor of the BSDs is how clean and portable their code is. Uh, huh, sure, let me disagree on that.

Both Linux and BSD are huge systems composed of millions of lines of code built by hundreds or thousands of different people with different skillsets. There will be components that are awesomely architected and there will also be other components that you wish you could unsee. This is true for any large-enough code base you look at and there is no way around it.

Oftentimes, BSD advocates will claim that because there is a single tree, it is easier to enforce code cleanliness. That is true, to some extent. It is hard to keep up with all changes going into a massive development tree, and a lot of the code is really old in computer years: the majority of the old code is where you will find the worst abominations.

But "so what?" The thing works. Both BSD and Linux work just fine in spite of how ugly some parts of their code might be. Depending on what you are working on, you may hate more or less the existing code, but that will not make you change your choice of operating system: by the time you are already touching code, you are pretty much sold on one of the two alrady.

License

Here we get into a more interesting territory: the license of each operating system cannot be changed. BSDs have traditionally focused around MIT-like licenses and tried to avoid GPL code in as much as possible, while GNU/Linux systems are made, mostly, of GPL code.

But "so what?" The vast majority of users would not care less about the license the software they run is covered by as long as it is some free software license.

In the few cases where this detail matters, the decision will really be made based on the license terms (not the technology) and, while the BSD license will "win" in most corporate cases, this will not always be the case. Some vendors may prefer the GPL or just may not care at all; just look at how many routers ship with Linux, for example.

Development model

The next difference, not always mentioned, is the development model behind the BSDs and the Linux distributions.

In the GNU/Linux world, each component of the operating system is most likely built by an independent set of developers. Each team works solely for its component's interest and interactions between them are rare. There is a relative large group of people working on the development of the Linux kernel. There is another large group of people working on X.org. There is a smaller group of people working on GTK+. There is an even smaller group of people working on libjpeg.

All these components must later be assembled by third-parties in what is known as Linux distributions to offer something useful to end users. The team behind the distribution has to merge software from dozens or hundreds of vendors into a single package that people can use. (As an ex-software packager, let me tell you that this is hell.) The important thing to notice is that the only entity overseeing the resulting product is the distributor: the developers of every individual component do not really care about the distributions — and even if they did care (as a few do), paying attention to all the distributions out there is impossible.

An interesting example here is systemd. Some notable Linux distributions have had to switch to systemd even when they were originally against it or would have preferred a different alternative. And there really is nothing they can do: they are subject to the will of the upstream developers, and those in turn are subject to the will of the handful of major distributions that dominate the market. It's a vicious circle that cannot be broken as long as your distribution is considered to be Linux and you care about remaining relevant.

In the BSD world, things are a little bit different: there is a single team of developers building the whole operating system and working together on a single source tree. Certainly, this source tree includes third-party components like clang, but changes to these third-party components have to be reviewed and considered just like any other change to the system.

Having a single source tree permits the developers to do things like developing a kernel change and its userland support in unison; fixing the handling of a widespread command-line option about a multitude of tools; defining what the default configuration of the system is with eas; or even sweeping through all manual pages to give them a consistent visual style.

It is easier to think about this in terms of other operating systems: if I asked you to describe what is in OS X 10.9, you would be able to give me a list of all included libraries and their functionality; you would be able to tell me what the standard applications that ship with the system (TextEdit, Terminal, Mail) are and their features. (The same goes for Windows if you are not familiar with Macs: the set of base libraries, applications and UI features that ship with Windows 8 is well-determined.)

Everything that is not part of the base system is shipped separately as packages, and it is understood that these are not as tightly nor nicely integrated with the system because they are out of control of the system developers. I.e. the same as what happens when you download and install a third-party application for Mac or Windows.

But the development model, per se, does not matter that much. Linux distributors are the gatekeepers as to what ships in their releases, so they could be more picky about what releases and features they pick.

Building a product

All that said, the development model described above is the key difference between the two systems. However, you need to look at it with the right lens to understand why this is the key competitive advantage of the BSDs:

Each BSD project works towards building a product and they fully control the development and integration of all of the product's components.

This, my friend, is the true difference. This is what made it possible for individual contributors to come up with the amazing cross-building infrastructure for NetBSD or the implementation of RUMP kernels. This is what makes it possible for me to build a whole-OS test suite for NetBSD and FreeBSD. This is what makes it possible for companies to base their production environments or appliances on specific BSD releases, knowing that the migration path forward is well thought of.

But I'm getting too excited. Let me take a step back to clarify the difference between building a product and developing software. These two concepts are by no means the same: the latter supplements the former but is in no way a replacement. Building a product involves writing code, yes, but it also involves gathering requirements from customers (users) and figuring out how to solve their problems; it involves writing cohesive documentation for the shipped product; it involves performing release engineering; it involves having a Q.A. process; it involves, well, having a package that you can ship to the world and that people can use and find useful right out of the box.

GNU/Linux doesn't offer you that: the distributions do, although the majority of people (including developers) do not understand the difference. Developers develop "for Linux" without realizing that this is pretty much meaningless unless you put your efforts in the context of a distribution. And users want to just run Linux and cannot wrap their heads around the multitude of distributions out there.

And this is also why calling FreeBSD, NetBSD, OpenBSD, etc. all "distributions of BSD" is wrong: they are not. Each and every one of these projects ships their own operating system. They take product decisions separately and they develop the features that go into their products on their own. Yes, there is code sharing and a lot of similarities between the systems, but they really are different products.

What are the practical implications?

For starters, releases that make sense. A BSD release is not just a bunch of newer software packages stitched together: BSD releases often show a "story" behind the work that has gone into the product. There will probably be major themes of development: major cross-functional features that you can only build by controlling the whole development environment.

Then, there will also be a clear migration path. For example, NetBSD takes extremely good care of not breaking binary compatibility between major versions. FreeBSD breaks compatibility, but when it does, it is very well documented where that is and what to do about it.

And lastly, the single tree where development happens. Maintaining an in-house fork of a BSD system is a relatively-easy process. (This is only true if you stay minimally up-to-date and avoid massive local patches, but the same can be said about Linux.) I cannot fathom maintaining a "brach" of a Linux distribution. Well, actually, I have seen that happen and let me spoil you the end of the story: it was not pretty.

A hard read maybe. But I think that's all I wanted to say.

In the GNU/Linux world, each component of the operating system is built by a different set of developers; pretty much all the developer teams, if you wish, are disjoint, and each team is working towards building a mini-product for their little component. You have a relative large group of people working on the development of the Linux kernel. You have another large group of people working on X.org. You have a smaller group of people working on GTK+. You have an even smaller group of people working on libjpeg.

But there is no product in the above and this is why Linux distributions exist, where an often-disjoint group of people work towards packaging all the various components from different "vendors" into a single product that they can ship. As an ex-software packager, this is hell. There is no entity overseeing the resulting product: not even the distribution owners, because they do not control what the upstream developers do for every individual component.

An interesting example here is systemd. Many Linux distributions have had to switch to systemd even if they were originally against it or would prefer a different alternative. And there really is nothing they can do: they are subject to the will of the upstream developers, and those in turn are subject to the will of the handful of major distributions that dominate the market.

In the BSD world, there is a single team of developers building the whole operating system. Or, rather, there is a single team working towards a product. Let me stress that out again because it's very important: building a product is a completely different to developing software. Shipping a BSD system as a product is a much easier endeavor.

They have access to the whole set of components that form the final operating system and can modify them in unison to implement new features. System releases are "checkpoints" of this whole development environment that deliver well-known feature sets and clear transition paths from one older version to the newer. The developers can work to ensure that this transition is seamless and control all the paths through it.

While it is true that BSD systems bundle external software components (GCC being the biggest example), this is not relevant: the developers have to pay incredible attention to this integration process to ensure that the whole system continues to work as it did before, that the documentation for the whole system is accurate and that users transitioning from a previous release to the newest are not surprised by unexpected incompatibilities.

In the GNU/Linux world... well, each developer works towards shipping newer releases of his own component: be it the Linux kernel itself, or coreutils, or gcc, etc. There is absolutely no way that a developer of coreutils can coordinate his change with a dependency on the kernel because the two sets of developers are completely disjoint and they share no release process.

Linux distributions exist to form the "product" around the Linux kernel: their job is to thousands of different components from as many sources, compiling them together and polshing the roughest edges around them. The developers of the distributions have little control on what goes into each component and have zero control on their relase schedules. It is incredibly hard to ship a clean and functional product out of pieces you do not control at all.