Have you ever wondered why an increasing number of programs are configured by placing small files in .d directories instead of by just editing a single file?

Have you ever wondered why these .d directories seem to proliferate in ~~Unix~~Linux installations? Because a pretty bare-bones Debian installation (WSL) shows… a ton of them:

debian:/etc> ls -ld *.d */*.d
drwxr-xr-x 1 root root 4096 Aug 14 22:23 apparmor.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 apt/apt.conf.d
drwxr-xr-x 1 root root 4096 May 12 05:57 apt/auth.conf.d
drwxr-xr-x 1 root root 4096 May 12 05:57 apt/preferences.d
drwxr-xr-x 1 root root 4096 May 12 05:57 apt/sources.list.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 apt/trusted.gpg.d
drwxr-xr-x 1 root root 4096 Aug 12 16:08 bash_completion.d
drwxr-xr-x 1 root root 4096 Apr 27 13:02 binfmt.d
drwxr-xr-x 1 root root 4096 Jun  3 14:09 ca-certificates/update.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 cron.d
drwxr-xr-x 1 root root 4096 Jul  5 12:10 dbus-1/session.d
drwxr-xr-x 1 root root 4096 Jul  5 12:10 dbus-1/system.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 dhcp/dhclient-enter-hooks.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 dhcp/dhclient-exit-hooks.d
drwxr-xr-x 1 root root 4096 Jun  3  2019 dpkg/dpkg.cfg.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 emacs/site-start.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 exim4/conf.d
drwxr-xr-x 1 root root 4096 Jun 18  2019 gss/mech.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 init.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 iproute2/rt_protos.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 iproute2/rt_tables.d
drwxr-xr-x 1 root root 4096 Apr 27 13:02 kernel/install.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 kernel/postinst.d
drwxr-xr-x 1 root root 4096 Aug 14 22:40 ld.so.conf.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 logrotate.d
drwxr-xr-x 1 root root 4096 Feb  9  2019 modprobe.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 modules-load.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 mysql/conf.d
drwxr-xr-x 1 root root 4096 Jul  4 08:31 mysql/mariadb.conf.d
drwxr-xr-x 1 root root 4096 Jan 28  2019 network/if-down.d
drwxr-xr-x 1 root root 4096 Jan 28  2019 network/if-post-down.d
drwxr-xr-x 1 root root 4096 Jan 28  2019 network/if-pre-up.d
drwxr-xr-x 1 root root 4096 Jan 28  2019 network/if-up.d
drwxr-xr-x 1 root root 4096 Jan 28  2019 network/interfaces.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 pam.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 ppp/ip-up.d
drwxr-xr-x 1 root root 4096 Jul 10 17:04 profile.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc0.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc1.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc2.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc3.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc4.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc5.d
drwxr-xr-x 1 root root 4096 Aug 12 18:41 rc6.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 rcS.d
drwxr-xr-x 1 root root 4096 Feb 26  2019 rsyslog.d
drwxr-xr-x 1 root root 4096 Feb 14  2019 security/limits.d
drwxr-xr-x 1 root root 4096 Feb 14  2019 security/namespace.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 sudoers.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 sysctl.d
drwxr-xr-x 1 root root 4096 Apr 27 13:02 tmpfiles.d
drwxr-xr-x 1 root root 4096 Apr 27 13:02 udev/hwdb.d
drwxr-xr-x 1 root root 4096 Apr 27 13:02 udev/rules.d
drwxr-xr-x 1 root root 4096 Aug  9 08:10 update-motd.d
debian:/etc>

… while a fresh installation of the more traditional NetBSD barely shows any:

netbsd:/etc> ls -ld *.d */*.d
drwxr-xr-x  2 root  wheel   512 Aug  9 07:43 fonts/conf.d
drwxr-xr-x  2 root  wheel   512 Aug  9 18:54 pam.d
drwxr-xr-x  2 root  wheel   512 Aug  9 07:43 rc.conf.d
drwxr-xr-x  2 root  wheel  2560 Aug  9 18:54 rc.d
drwxr-xr-x  4 root  wheel   512 Aug  9 07:43 saslc.d
netbsd:/etc>

What’s up with that? The reason is simple:

Supporting configuration updates as part of first-class package management.

But why do .d directories help? Let’s first look at a couple of examples to understand how these directories work and then we’ll get into the why.

An example: sysctl.conf and sysctl.d

To understand how configuration directories work, let’s look at a particular example that exists across the two systems we are looking at. That is the mechanism to apply sysctl configuration changes.

For those not familiar with it, sysctl is a tool and an interface to tune kernel-level settings. On Linux, the sysctl(8) tool is a simple wrapper over the contents of /proc/sys/, but on NetBSD it’s a direct client to the kernel’s sysctl(3) API. For a sneak peek of what these kernel settings involve, type sysctl -a to get a list of all available knobs and their current values.

Because sysctl offers access to tuning kernel-level knobs for the whole system, it seems natural to have a mechanism to configure those at system boot time. After all, if you want to modify how e.g. the virtual memory subsystem works, you will want do that during the early stages of the system’s boot process.

Looking at a NetBSD system, here is what we have:

netbsd:/etc> ls -l sysctl*
-rw-r--r--  1 root  wheel  797 Oct 12  2018 sysctl.conf
netbsd:/etc>

In this scenario, all kernel settings must be put in the single /etc/sysctl.conf text file, which is then trivially read at boot time by the /etc/rc.d/sysctl script. Both human administrators and automated processes have to update this single file if they want to effect change on sysctl settings at boot time.

Let’s compare this to a what Debian system has:

debian:/etc> ls -l sysctl*
-rw-r--r-- 1 root root 2351 May 31  2018 sysctl.conf

sysctl.d:
total 4K
lrwxrwxrwx 1 root root  14 Apr 27 13:02 99-sysctl.conf -> ../sysctl.conf
-rw-r--r-- 1 root root 639 May 31  2018 README.sysctl
-rw-r--r-- 1 root root 324 May 31  2018 protect-links.conf
debian:/etc>

We still have the traditional /etc/sysctl.conf file but we also have an /etc/sysctl.d/ subdirectory. In this case, the systemd-sysctl boot service—because it had to be systemd, of course—iterates over all files under /etc/sysctl.d/ in lexicographical order and applies their settings.

Note that I did not mention /etc/sysctl.conf and that’s because systemd-sysctl couldn’t care less about it. That file location exists for the administrator’s convenience as the traditional place to perform manual edits. But the way systemd-sysctl finds this file is thanks to its /etc/sysctl.d/99-sysctl.conf symlink. The 99 prefix is important, as this causes the administrator-controlled file to sort after all others. In this way, its contents override any other settings that might have been put under the sysctl.d directory by automated tools.

But that’s not the only way configuration directories interact with their corresponding legacy file.

Another example: ld.so.conf and ld.so.conf.d

In our previous example, we saw that the /etc/sysctl.conf file on a Debian system exists purely for the convenience of the administrator. The system does not care about this file directly so its existence in this specific location might as well be removed.

The alternative is to go the complete opposite route: keep the configuration file as the primary location for the configuration, but then make this file explicitly include all contents of a subdirectory.

An obvious example of this behavior is the configuration for the dynamic linker: the ldconfig tool reads the /etc/ld.so.conf file to discover which directories contain dynamic libraries, but is otherwise unaware of the existence of the /etc/ld.so.conf.d/ directory. So how is the latter read then? Easy:

debian:/etc> cat ld.so.conf
include /etc/ld.so.conf.d/*.conf
debian:/etc>

As we can see, /etc/ld.so.conf continues to be the primary file, but it then includes all individual files from the configuration directory. This requires native support in the tool to allow file inclusions and also makes the semantics of glob expansion tool-specific (are they in lexicographic order?).

Note that, in this case, the administrator has the freedom to append custom configuration directly to /etc/ld.so.conf or to create a new unmanaged file under /etc/ld.so.conf.d/ (say, 99-zzz.conf).

The real purpose behind .d directories

Once you start looking, as we did in the first listing, you’ll notice that this pattern repeats everywhere: for any given configuration that needs edits from multiple independent sources, you will likely find a traditional configuration file accompanied by a configuration subdirectory.

So now that we have all necessary knowledge, we are ready to tackle the questions: What are configuration directories for? Why have these configuration directories proliferated? What benefit do they bring? And the answer to them all is the same: they make automated edits straightforward, which is important when you want a package manager to perform and own such edits.

You see: if you have a plain text file and you want to edit it programmatically, you have two options. You can have a parser that understands the format of the file and is able to modify it, but then preserving comments and formatting (both of which are important for configuration!) is difficult. Or you can add comment-like markers to the file and perform unstructured edits, but then those edits break if the user happens to modify the file in an unexpected manner.

Contrariwise, if you have a directory where you can put individual files in, and the whole configuration is determined to be the logical concatenation of all these files, all of a sudden automated edits become trivial. A package wanting to add new settings to the system will only need to add a new file in that directory, and the package manager will be able to track the provenance of that file. And if you uninstall the package, the package manager will be able to remove its configuration without issues (possibly leaving behind the files if they were manually modified).

Parting words

So, are configuration directories better than files?

Well… they are different. With directories, we trade the ability to read and modify a single plain text file by hand with a scheme that is maybe-trickier to hand-tune but that is amenable to automated edits. As a consequence, we shift the complexity from a package installer’s scripts to the tools that read the configuration.

And I think this tradeoff is well worth it. A package manager that can track the provenance of almost all files in a system and that can perform automated reconfigurations in a robust manner is a blessing. In particular because, you, as the system administrator, will have increased confidence that the package manager will not revert any “99” local-only files created on your own. And also, shifting this logic out of the package manager and putting it into the tools ensures consistency across different systems (especially across different Linux distributions).

One last thought: is this concept of configuration directories restricted to configuration? Nope! As we shall see in the next post, this can be applied to system-wide databases as well.