Let’s open with the controversy: the scripts that live under /etc/rc.d/ in FreeBSD, NetBSD, and OpenBSD are in the wrong place. They all should live in /libexec/rc.d/ because they are code, not configuration.

This misplacement is something that has bugged me for ages but I never had the energy to open this can of worms back when I was very involved in NetBSD. I suspect it would have been a draining discussion and a very difficult thing to change.

But… what am I talking about anyway?

If you have administered a BSD system, you have certainly encountered the /etc/rc.d/ directory; and if you have administered pre-systemd Linux systems, you have dealt with /etc/init.d/. These directories contain startup scripts to configure the system at boot time and are immutable. Their code is parameterized to allow changing their behavior via configuration files, not via code edits. And that’s the base of my critique.

But before getting into why the current state is problematic and how things should look like, let’s first dig into how we got here. And, for that, we need to go back in history.

History: The BSD approach

4.4BSD’s (1993) boot process was rather simple: the kernel started init which in turn ran the /etc/rc script before starting getty on each console. The /etc/rc monolith was in charge of configuring the machine’s file systems and processes, and delegated to two other scripts: /etc/netstart for network configuration and /etc/rc.local for locally-added services. Current BSD systems are more advanced in this area as we shall see later, but the core boot process remains the same: /etc/rc is the primary entry point and bootstraps a collection of shell scripts.

In the early days, package management and file provenance tracking, like we are used to having in popular Linux distributions, was not a thing. You were expected to tune the systems’ behavior by editing files which might or might not have been designed to support edits. If you had to edit /etc/rc, which was a script shipped by the system, that was alright.

/etc/rc.local, on the other hard, was not shipped by the system, and it was up to you to create it if you wanted to add custom startup commands without modifying /etc/rc. And this is where things get interesting. /etc/rc.local didn’t need to be supported: if you were expected and able to edit /etc/rc anyway, why would you deal with a separate file? The reason is, most likely, to simplify system upgrades: during an upgrade, you want to benefit from any upstream changes made to /etc/rc (some of which might actually be necessary for proper system operation). Applying updates to a manually-modified file is tricky, so putting as many of your manual overrides into /etc/rc.local helped minimize this problem.

History: The System V approach

System V 4 (SVR4, 1988) also came with its own, and very different, boot process. The key difference was that System V had the concept of runlevels. As a result, configuring the boot process was a more convoluted endeavour because it was possible to select different services per runlevel.

To accomplish per-runlevel tuning, the system used configuration directories rather than files: there was a separate /etc/rcX.d/ directory for each runlevel (where X was the number of the runlevel), and these directories contained one file per action to take at startup time. To avoid duplicates, these files were just symlinks to common files under /etc/init.d/—and the symlinks, not their targets, were named so that their lexicographical order determined startup execution order.

Once again, we can already observe issues here: the symlinks under /etc/rcX.d/ are configuration because their presence indicates what to start and their names determine their startup order. But the files under /etc/init.d/ are not: they are shell scripts shipped with the system and should not be manually modified.

History: The NetBSD modernization

NetBSD modernized the boot process in its 1.5 release (2000), and it did so in two ways: first, it introduced /etc/rc.d/ as a directory to contain separate scripts per action and service; and, second, it introduced the rcorder(8) tool to determine the order in which these services run. rcorder(8) uses dependency information encoded in the scripts as comments—not lexicographical ordering as System V did. FreeBSD inherited this design in its 5.0 release (2003) and OpenBSD reimplemented something similar in its 4.9 release (2011).

With these two pieces in place, the /etc/rc script in NetBSD and FreeBSD changed to execute all files from /etc/rc.d/ based on the output of rcorder(8). Among these scripts is /etc/rc.d/local, whose purpose is to run /etc/rc.local if it exists. And that’s all, really. The /etc/rc script thus became trivial.

The key thing to notice here is that the scripts shipped in /etc/rc.d/ are highly configurable via the user-controlled /etc/rc.conf file. This essentially makes the scripts read-only, as it shifts local customizations to the configuration file. System administrators are not supposed to edit the scripts. Instead, they are supposed to: modify /etc/rc.conf to customize what gets run and how; add new scripts under /etc/rc.d/ if they so choose; and edit /etc/rc.local to easily run arbitrary commands.

Why does rc.d not belong in /etc?

My main gripe is that the files under /etc/rc.d/ are immutable scripts. They do not belong in /etc/ and their presence there makes system upgrades harder for no good reason.

You see: in NetBSD and FreeBSD, system upgrades happen by unpacking new distribution sets in the root directory and then running a script to incorporate configuration updates. This script is interactive and helps highlight how new system-provided updates to configuration files conflict with previous manual edits. (This process might seem rudimentary to you, but it’s actually pretty robust and easy to understand—and you can use tooling like sysupgrade to make it trivial.)

So why is that a problem? Because you will always face merges like this:

--- /etc/rc.d/npf               2019-08-09 19:09:42.800758233 -0400
+++ /tmp/temproot/etc/rc.d/npf  2019-11-16 10:39:27.000000000 -0500
@@ -1,6 +1,6 @@
 #!/bin/sh
 #
-# $NetBSD: npf,v 1.3 2012/11/01 06:06:14 mrg Exp $
+# $NetBSD: npf,v 1.4 2019/04/19 18:36:25 leot Exp $
 #
 # Public Domain.
 #
@@ -36,7 +36,11 @@
        echo "Enabling NPF."
        npf_cfg_check
        /sbin/npfctl reload
-       /sbin/npfctl start
+
+       # The npf_boot script has enabled npf already.
+       if [ "$autoboot" != "yes" ]; then
+               /sbin/npfctl start
+       fi
 }
 
 npf_stop()

File: /etc/rc.d/npf (modified)

Please select one of the following operations:

  d  Don't install the new file (keep your old file)
  i  Install the new file (overwrites your local modifications!)
  m  Merge the currently installed and new files
  s  Show the differences between the currently installed and new files
  su  Show differences in unified format ("diff -u")
  sc  Show differences in context format ("diff -c")
  ss  Show differences side by side ("sdiff -w187")
  scommand Show differences using the specified diff-like command
  v  Show the new file

What do you want to do? [Leave it for later]

And, really, who cares? Why are you being distracted to review a code change when what you are trying to do is assess configuration conflicts? How many times have you actually objected to these merges?

You might say: well, I want to know exactly how the boot process of my machine changes during an upgrade. Sure, that’s a fine goal, but then this procedure is flawed and completely insufficient to achieve such goal. In the example above, whatever /etc/rc.d/npf does can also be done from within the /sbin/npfctl binary it invokes… and you were never asked to review changes to the latter during an upgrade, were you? And of course you could review the binary’s code as part of your own system build, but if you did that, then you could have reviewed the startup script as well, right?

Where should rc.d live instead?

Startup scripts provided by the system need to live in a location that can contain executables—but we don’t want those executables to show up in the PATH. These requirements discard bin and sbin, and points us towards libexec on BSD systems and somewhere under lib on Linux.

Therefore, the read-only startup scripts should move from /etc/rc.d/ to /libexec/rc.d/ (which, by the way, also applies to /etc/rc, /etc/rc.subr, and /etc/rc.shutdown). And that’s it. /etc/rc /libexec/rc should continue to use rcorder(8) to check what’s needed to run, but it should read files from /libexec/rc.d/. You might even want to support a separate location for user-created services, which might still be /etc/rc.d/ or, better yet, a more fitting location like /usr/pkg/libexec/rc.d/ (though that quickly runs into problems if you have multiple file systems).

With this design, system upgrades would be much saner because the configuration merge process would focus, purely, on actual configuration changes and not on irrelevant code changes. All updates to /libexec/rc.d/ would be applied by unpacking the new distribution sets (base.txz in this case) without disturbing you about how exactly they changed.

Does this relate to Linux distributions at all? I briefly mentioned /etc/init.d/ at the beginning, and the problem there is similar. But it’s also an obsolete problem given that Linux distributions have moved onto systemd by now. That said, systemd still has to manage individual services and, like it or not, has gotten this right. If we look at the systemd.unit(5) manual page, which describes where units are loaded from, the system first looks into various /etc/ and /run/ directories, but then also looks at /usr/lib/systemd/system/ (a read-only location correctly controlled by the package manager)—and the vast majority of the scripts live inside the latter.

Want to help?

I currently don’t run BSD systems any more so my incentives to make this happen are low… but if this all makes sense and is something you’d like to pursue, by all means please do! I’m happy to help but I’m not looking forward to the discussions needed to push this through 🙄.