The NetBSD vs. the FreeBSD test suite

This post had been sitting in my draft queue for almost a year. It is not fresh any longer, but the information in it is still worth posting, so here we go.

In my last post, you saw what my plans for the FreeBSD test suite are and may have noticed that they differ significantly from what currently exists in NetBSD.

Somebody rightfully pointed out in Twitter that the test suite has been under development for 7 years already and we are "not there yet". There is no point in taking anger at that comment because it is a good reality check. It is very true that things have stagnated in various areas and that no real activity has happened in the underlying framework. Things have moved forward significantly in the NetBSD test suite itself (i.e. more tests), but that's about it. Unfortunately, this work all happens in spare time and requires a great deal of energy and motivation — all of which fluctuate easily and were at minimums for me over last year.

Status of the NetBSD test suite

Let me start by describing the highlights of the NetBSD test suite:

• Vast collection of tests. Although it wasn't trivial at first, the NetBSD community has embraced testing. The result of this is a pretty extensive and ever-growing test suite that covers many areas of the system, including (and most importantly) the kernel.
• Various continuous testing machines up and running. Even though this is the point I will self-criticize the most below, a few NetBSD developers have done a great job at taking the lacking ATF toolchain and building continuous testing machines and services for result publishing on top of them.
• The Anykernel and RUMP systems bundled in the upstream NetBSD tree provide a mechanism to build tests for kernel-level code that run without privileges, in user-space, without a chance of crashing the kernel. These tests —and, mind you, the whole Anykernel philosophy— is invaluable to kernel developers because the code–build–test cycle is shortened to the very minimum.

So what are the lowlights?

• The engine that executes the test suite bundled with NetBSD is the deprecated ATF tools.
• The build infrastructure —that is, the bsd.test.mk file— supports building ATF-based tests and those tests only. This is because the runtime in use only understands how to run its own tests.
• Running the tests "out of the box" from the point of view of a developer or a system administrator is an easy enough thing to do. However, setting up a continuous testing system is really difficult because ATF does not provide any facilities for doing so.
• Every person that has set up a continuous testing system has had to implement wrapper scripts to build the system, collect build logs, set up a VM with the results, run the installed tests within the VM, extract the tests, format the tests as HTML, and, after all this dance, glue all the logs and result pages together into a customized web server. All this wheel-reinvention is suboptimal, especially when it is completely tied to NetBSD.

Fixing NetBSD

In order to fix all the shortcomings of the NetBSD test suite above, my plan was to switch NetBSD from the deprecated ATF tools to Kyua. Kyua currently does not solve the problems above but has feature-parity with the ATF tools and thus is a viable replacement. Missing features would be built on top of Kyua later, iteratively, as the design makes it easier.

However, because Kyua only has feature parity with ATF at the moment, the maintainers of the continuous testing machines see no benefit for switching. And, what is worse, some of the differences in Kyua apparently make the switch inviable. (These are all mostly "fixable" problems which I haven't had the time to address...)

Enter FreeBSD

The answer to all this is in FreeBSD. Because FreeBSD did not have a test suite framework in place until 2013Q4, the Test Suite project around it was not held back by the deprecated ATF tools. My first contribution to FreeBSD was actually the removal of the ATF tools from the tree before any formal release ever shipped with them — precisely to prevent having to remain compatible in the future!

With the ATF tools gone from the tree, Kyua is the runtime engine of choice for FreeBSD. From the ground up, new features can be introduced that are not tied up to backwards compatibility with NetBSD. This is a path to tread with care however: blindly introducing incompatibilities in ATF or Kyua would lead to a de-facto fork in NetBSD which would be hard to reconcile.

For these reasons, you should see the FreeBSD test suite as an experimentation ground where the current Kyua seed is planted and slowly built upon to reach all the goals of the project. This is not an easy process and will indeed take time. But, with that and energy, we'll get there.

Once the new ideas have proven worthy under FreeBSD, we can backport them to NetBSD.