Netbsd - Julio Merino (jmmv.dev)

Hands-on graphics without X11

If you have been following the development of EndBASIC, you know its console can display both text and graphics at once. What you may not know is that, now, it can also achieve this feat on the NetBSD console without using X11 at all. This is done by directly rendering to the wsdisplay framebuffer, and this article presents a crash course on direct graphics and keyboard access via NetBSD’s wscons framework.

January 17, 2025 · Tags: blogsystem5, endbasic, netbsd
Continue reading (about 13 minutes)

Revisiting the NetBSD build system

I recently picked up an embedded project in which I needed to build a highly customized full system image with minimal boot times. As I explored my options, I came to the conclusion that NetBSD, the often-forgotten BSD variant, was the best viable choice for my project.

One reason for this choice is NetBSD’s build system. Once you look and get past the fact that it feels frozen in time since 2002, you realize it is still one of the most advanced build systems you can find for an OS. And it shows: the NetBSD build system allows you to build the full OS from scratch, on pretty much any host POSIX platform, while targeting any hardware architecture supported by NetBSD. All without root privileges.

Another reason for this choice is that NetBSD was my daily workhorse for many years and I’m quite familiar with its internals, which is useful knowledge to quickly achieve the goals I have in mind. In fact, I was a NetBSD Developer with capital D: I had commit access to the project from about 2002 through 2012 or so, and I have just revived my account in service of this project. jmmv@ is back!

So, strap onto your seats and let’s see how today’s NetBSD build system looks like and what makes it special. I’ll add my own critique at the end, because it ain’t perfect, but overall it continues to deliver on its design goals set in the late 1990s.

December 28, 2024 · Tags: blogsystem5, netbsd
Continue reading (about 16 minutes)

Kyua graduates

After years of inactivity, the Kyua project has graduated as an open source citizen and has a new home under the FreeBSD umbrella!

But uh… wait, what is Kyua and why is this exciting? To resolve confusion and celebrate this milestone, I’d like to revisit what Kyua is, how it came to be, why I stopped working on it for a while, why that was a problem for FreeBSD—and, indirectly, NetBSD—and how Kyua being free software has helped keep it alive.

August 2, 2024 · Tags: blogsystem5, freebsd, kyua, netbsd
Continue reading (about 14 minutes)

Speeding up autoconf with caching

In the recent Remembering Buildtool post, I described how setting up a cache of configuration checks was an important step in Buildtool’s installation process. The goal was to avoid pointless repetitive work on every build by performing such common checks once.

Episode 457 of BSD Now featured my post and Allan Jude wondered how much time would be saved in a bulk build of all FreeBSD packages if we could just do that same kind of caching with GNU Autoconf. And, you know what? It is indeed possible to do so. I had mentioned it en passing in my post but I guess I wasn’t clear enough, so let’s elaborate!

June 17, 2022 · Tags: freebsd, netbsd
Continue reading (about 10 minutes)

pkgdb belongs in libdata, not var

The pkgsrc package database, which by default lives in /var/db/pkg/, should not be there. Instead, it should be under /usr/pkg/libdata/pkgdb/. The same applies to FreeBSD’s and OpenBSD’s ports and also Debian’s dpkg, but I’ll focus on pkgsrc because it’s the system I know best. Let’s see why the current default is suboptimal and why libdata is a good alternative.

August 26, 2020 · Tags: netbsd, opinion, pkgsrc
Continue reading (about 7 minutes)

rc.d belongs in libexec, not etc

The scripts that live under /etc/rc.d/ in FreeBSD, NetBSD, and OpenBSD are in the wrong place. They all should live in /libexec/rc.d/ because they are code, not configuration. Let’s look at the history of these systems to see how we got here, why this is problematic, and how things would look like in a better world.

August 24, 2020 · Tags: freebsd, netbsd, opinion, sysupgrade
Continue reading (about 8 minutes)

A tour of directories as system-wide databases

In the previous post, we saw how .d directories permit programmatic edits to system-wide configuration with ease. But this same concept can be applied to other kinds of tracking. Let’s dive into a few examples ranging from desktop menu entries to the package manager’s database itself.

August 21, 2020 · Tags: debian, menu2wm, netbsd, unix
Continue reading (about 12 minutes)

Configuration files and .d directories

Have you ever wondered why an increasing number of programs are configured by placing small files in .d directories instead of by just editing a single file? Have you ever wondered why these .d directories seem to proliferate in Linux installations? Read on to understand what these are and why they are useful.

August 17, 2020 · Tags: debian, featured, netbsd, unix
Continue reading (about 9 minutes)

Keeping NetBSD up-to-date with pkg_comp 2.0

This is a tutorial to guide you through the shiny new pkg_comp 2.0 on NetBSD.

Goals: to use pkg_comp 2.0 to build a binary repository of all the packages you are interested in; to keep the repository fresh on a daily basis; and to use that repository with pkgin to maintain your NetBSD system up-to-date and secure.

February 18, 2017 · Tags: netbsd, pkg_comp, sandboxctl, software, tutorial
Continue reading (about 6 minutes)

Refocusing Kyua maybe?

The FreeBSD devsummit that just passed by gave me enough insight into Jenkins to question the long-term plans for Kyua. Uh, WHAT?! Let me explain.

In the beginning...

One of the original but unstated goals of Kyua was to fix the "mess" that is the NetBSD releng test run logs site: if you pay close attention, you will notice that various individuals have reinvented the wheel over and over again in an attempt to automate release builds and test suite runs. In other words: different parties have implemented independent continuous integration systems several times with more or less success.

May 23, 2014 · Tags: freebsd, jenkins, kyua, netbsd
Continue reading (about 6 minutes)

BSDCan 2014 summary

BSDCan 2014 and the accompanying FreeBSD devsummit are officially over. Let's recap.

FreeBSD devsummit

The FreeBSD devsumit at BSDCan is, by far, the largest of them all. It is true that I already visited a devsummit once —the one in EuroBSDCon 2013—, but this is the first time I participate in the "real deal" while also being a committer.

The first impressive thing about this devsummit is that there were about 120 attendees. The vast majority of these were developers, of course, but there was also a reasonable presence from vendors — including, for example, delegates from Netflix, Isilon, NetApp and even smaller parties like Tarsnap.

May 21, 2014 · Tags: bsdcan, conference, freebsd, netbsd, testing
Continue reading (about 6 minutes)

Killing the ATF deprecated tools code

The time to kill the deprecated tools —atf-report and atf-run principally— from the upstream ATF distribution file has come. Unfortunately, this is not the trivial task that it may seem.

But wait, "Why?" and "Why now?"

Because NetBSD still relies on the deprecated tools to run its test suite, they cannot just be killed. Removing them from the upstream distribution, however, is actually a good change for both ATF and NetBSD.

February 5, 2014 · Tags: atf, netbsd
Continue reading (about 5 minutes)

EuroBSDCon 2013 takeaways

EuroBSDCon 2013 is done. If you have been following my daily posts over the last 4 days (day 1, day 2, day 3 and day 4) as well as #EuroBSDCon updates in Twitter, you may already have a pretty good idea of what went on here. However, with the conference over, it is now a good time to recap the whole event and present the takeaways of these four days which, overall, were quite interesting and productive.

September 30, 2013 · Tags: eurobsdcon, freebsd, kyua, netbsd
Continue reading (about 6 minutes)

Live from EuroBSDCon 2013, day 3

Live from Malta today attending the EuroBSDCon 2013 conference. Today is the first day of the conference itself. Many more people have shown up as expected and there have been tons of very interesting talks all the time. It is both good and bad that there are several tracks: you can select the topic you are most interested in, but sometimes great talks overlap!

Keynote

Today's opening session was led by Theo de Raadt, the founder of OpenBSD. His keynote focused on explaining how there is no real research happening on operating systems any more and how new, risky technological changes can be tested in a real-world system like OpenBSD.

September 28, 2013 · Tags: conference, eurobsdcon, kyua, netbsd
Continue reading (about 4 minutes)

Live from EuroBSDCon 2013, day 1

Hello everyone! Live from Malta today attending the EuroBSDCon 2013 conference.

Today is the first day out of four: two days of tutorials and two days of actual conference. The tutorials are overlapped by two days of the usual FreeBSD Developer Summit (devsummit for short) and one day of the infrequent NetBSD Developer Summit.

The ambient here is pretty good already: lots of enthusiastic people catching up since the last time they met each other and, more importantly, discussing ongoing developments. Keeping in mind that this is only the first day of tutorials and not the proper conference, things look promising: many more people are expected to join on Saturday.

September 26, 2013 · Tags: conference, eurobsdcon, freebsd, netbsd
Continue reading (about 3 minutes)

Self-interview after leaving the NetBSD board

The decision to not renew my NetBSD board membership was bittersweet.

Let me put aside the Readability series posts for a moment while I recap how the two years serving the NetBSD Board of Directors have been. My term just finished a couple of weeks ago, so it is better to post this while it is still relevant.

First, let me backtrack a little bit. A couple of years ago, I was nominated to serve the NetBSD Board of Directors. Needless to say, I was flattered that this was the case so I decided to run for the position. This worked, so “soon” after I joined the board in May of 2011.

June 20, 2013 · Tags: featured, netbsd
Continue reading (about 14 minutes)

Setting up my old Mac Mini G4 as a development machine

I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below.

I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it... which is great, but when thinking about it objectively, I am reluctant to unnecessarily "punish" all NetBSD users by increasing the build time of the system significantly. Don't get me wrong: Kyua itself runs efficiently on old machines, but building the code — particularly the hundreds of tests that come with it — takes just too long. This slowdown is not too noticeable on a relatively-modern machine, but it's unacceptable on not-so-old machines.

Of course, I could proceed with the import right now (with the code disabled by default) and make it leaner later, but this would cause quite a first bad impression on our users. So it's better to delay the import a little bit until, at least, I have had a chance to simplify some core parts of the code. Mind you, this simplification work is already in progress and quite advanced: it consists on modularizing (as separate processes) some critical parts of the code and writing these with a much simpler style and in plain C.

But back to the original point of my post.

The first thing to mention about this experience is that with some effort and long waits, I've got a pretty decent setup for development even on this old machine. From time to time, I miss having a real Unix desktop at hand for development (no OS X, you are not one of those). The GUI behaves relatively well with a 1920x1200 display, running Fluxbox, traditional xterms, Mutt for GMail access and a bunch of other applications.

Unfortunately, too many things feel really sluggish. A few specific examples:

Firefox 16 is barely usable. I'm not sure there are many alternatives to decent web browsing for such an old non-Intel platform. Dillo is blazing fast and allows me to access online documentation and mailing list archives (just enough for development), but it's pretty much useless for any other purpose given the "Web 2.0".
Any operation that involves pkgsrc takes ages. Even when building the simplest packages, one can notice the system crawl through all of the pkgsrc infrastructure. Sometimes this is the fault of a bad algorithm; other times it's just sh(1) being the wrong tool for something as complex as pkgsrc internals.
Things like basic code editing in Emacs 24 are slow at responding to typing. Disabling font lock mode makes it feel fast again, but it's just surprising to see that even color coding is slow.

I still remember my old and trusty machine from 10 years ago (a Pentium II 233 MHz): with a similar setup, it was significantly snappier. Yes, software has evolved and these packages now have many more features... but really, does editing a text file have to be sluggish?

Leaving aside sluggishness, there is also the problem of instability. NetBSD/macppc is a tier 2 port, and things don't work as well as one would like. I personally would enjoy bringing this port to tier 1... but I currently lack the time (and basic knowledge of the architecture) to do so :-/

Anyway, the result of this exercise: the new code I'm writing to modularize Kyua builds damn fast in this machine, and keeping it this way is the whole point of having such an old machine as a build environment. So I'll try to keep using it to develop these new components of Kyua.

October 22, 2012 · Tags: kyua, lab-notes, mac, netbsd
Continue reading (about 3 minutes)

Testing NetBSD's share/mk

For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends.

One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story:

Other projects depend on these build files and thus rely on their behavior. This can either be by just relying on NetBSD having these files installed, or by using the separate mk-files package (available e.g. in Fedora). The functionality of the build infrastructure should not regress, or it would break these third-party projects. (Think about these files as having a public API outside of NetBSD.)
Even if the build of NetBSD is the real test of the functionality of the build infrastructure, the build infrastructure supports dozens of options and tweaks to change its behavior. One would need to rebuild NetBSD tens, if not hundreds of times, each with a different combination of build options, to ensure the files work.
Lastly, when a new functionality is added to the build infrastructure, it is because some component of the source tree needs such functionality. It may latter happen that this component disappears or stops needing such functionality. The functionality becomes unused by the source tree, and thus can regress unexpectedly (breaking third-party packages or introducing bugs, as mentioned earlier).

With this in mind, it's clear that we should have some tests for the share/mk files. Unfortunately, that's easier said than done: the files in share/mk are extremely complex and expose hundreds, if not thousands, of different behaviors each with its own subtleties. Adding tests for these is hard. The fact that this code that was never designed to be unit-tested doesn't help either.

Regardless, I have just submitted some "placeholder" tests to the tree. The major point of these new test programs is to lower the barrier of entry to writing tests for share/mk and, therefore, maybe get other people to write some tests. To predicate with the example, I have populated these skeleton test programs with a couple of test cases each, although as you will see they are very trivial tests.

And, to conclude, why have I done this now? Well: I'm working on integrating Kyua into the source tree and, to make this happen, I need to do a couple of changes to the build infrastructure files. The changes are tricky, so I want to write tests to have some assurance that my modifications work and that, specially, they do not regress over time. Having the placeholder test programs in place makes this much easier, and the real functionality changes easier to review.

August 26, 2012 · Tags: netbsd, testing
Continue reading (about 3 minutes)

Introducing shtk

Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so.

That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short. Yeah, this name sounds very pretentious but, really, I don't intend this to be anything big. The only thing I want to do is simplify my life when implementing shell scripts, and hope that other people might find the modules useful. So far, I have taken the generic (and common!) code from sysbuild and sysupgrade, reconciled a few tiny divergences, and moved it into this new shtk package.

In reality, writing something like shtk is sin-borderline. I really should not be using shell scripting for the kind of tools I am implementing (they deserve better data structures and better error checking than what shell provides, for example). However, shell scripting is incredible convenient to get reasonably-good implementations of such tools with minimal effort, and is the only scripting language available in NetBSD's base system. (Yes, yes, there is Lua, but my limited knowledge of Lua would not let me write these tools in any decent manner nor in any reasonable time.)

So, what's in shtk? There are a few functions to deal with command lines (error/warning reporting and such things), some trivial stuff to deal with lists, a bunch of code to interact with cvs and, what I like the most, a module to implement configuration files with some kind of key/value validation.

At the moment, shtk can only be found in pkgsrc under pkgsrc/devel/shtk and I don't currently have any plans to make it more widely available. If there are enough people interested in that with real needs, I could reconsider, but the maintenance effort would be non-trivial.

To showcase the features of shtk, I have updated the sysbuild and sysupgrade packages to depend on this new toolkit while at the same time dropping all this duplicate supporting code. It's a good thing that I wrote exhaustive tests for all possible code paths, because the migration from the built-in modules to shtk was riddled with subtleties that would have impacted end users otherwise.

Now... time to really consider taking the task of rewriting pkg_comp in a more maintainable style so that I can add the features I have wished for for many years (like OS X support).

August 15, 2012 · Tags: announce, netbsd, shell, shtk, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Introducing sysupgrade for NetBSD

Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version?

Upgrading NetBSD to a newer version from distribution sets generally looks like the following;

Fetch new distribution sets (or roll your own).
Upgrade the kernel.
Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.tgz along the way.
Use etcupdate to merge new changes to configuration files.
Use postinstall to validate the upgraded system.

Simple? Yes. Easy? No. The above procedure is obscure to anyone new to NetBSD. (Actually, if you tell anybody that the way to upgrade your machine is by unpacking tarballs over / will stare at you thinking you are kidding. It's 2012.) "Jokes" aside, what is worse is that the procedure is quite monotonous and, therefore, it is very easy for the administrator to make a trivial mistake along the way and screw up a running system. (Been there, done that... multiple times.)

Machines are made to automate trivial and repetitive tasks like the above, and they are actually very good at that. Over the years, I have performed NetBSD updates manually and later written crappy, unreliable scripts to do the upgrades for me. These scripts have never been reusable and they haven't dealt with error conditions gracefully. Furthermore, because these scripts live in my home directory, I have to remember to carry them around every time I set up a new NetBSD box.

It was about time I sat down and rewrote my custom scripts into something more "decent". Something with documentation, with a configuration file, and with tests.

So today, and for all the reason above, I am introducing sysupgrade.

sysupgrade is a script that automates (trivializes) the whole process of upgrading an existing NetBSD installation to a newer release, be it the currently-running system or a non-live system. sysupgrade does so by following the steps outlined above and is coordinated by a configuration file. You can find the tool in pkgsrc/sysutils/sysupgrade, next to sysbuild. The bundled sysupgrade(8) and sysupgrade.conf(5) manual pages, and the default sysupgrade.conf configuration file should get you started and hopefully answer most of your questions.

For the impatient, the following command would upgrade your machine to the specified version target:

$ sysupgrade auto
ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-<X.Y.Z>/$(uname -m)

At the moment, please consider sysupgrade to be experimental. It works well for me on my various machines (running both NetBSD 6.0 BETA and -current), and I have been using the upgrade procedure outlined above for years without issues. However, as with any shiny-new software, be careful. If you use NetBSD on a virtual machine, take a snapshot before running this tool; I don't think your machine is going to blow up, but better safe than sorry!

Enjoy, and feedback very welcome!

PS: There is lots of room for improvement. The TODO file in the package directory includes some of the ideas I'd like to work on later.

August 6, 2012 · Tags: announce, netbsd, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Introducing sysbuild for NetBSD

NetBSD's build system is close to awesome: after checking a source tree out from CVS on virtually any Unix-like operating sytem, building a full NetBSD release for any of the supported platforms is as simple as running the build.sh script with the right arguments.

There are, however, a few things that would deserve automation in this process, but that are not in build.sh's domain to solve. These are:

Fetching and keeping the source tree up to date: interacting with CVS is still the responsibility of the user.
Simplifying the call to build.sh: this script takes a ton of arguments, and running it by hand quickly becomes "annoying". It would be nice if all the desired values to its arguments where stored elsewhere and picked up automatically. (mk.conf may not always be the right choice.)
Plugging into your crontab(5) for easy periodic rebuilds of NetBSD, with proper reporting of failures. Upgrading a live system from source is a major supported mechanism (particularly for NetBSD-current), so having a way to keep fresh release files around is welcome.
Performing such daily rebuilds of NetBSD as a dedicated unprivileged user.

To anybody familiar with NetBSD, it is obvious that all the above items are "simple enough" to solve one by one; however, doing them by hand gets old very quickly — machines exist to make our life easier, don't they? I know I am not the only one that has custom local wrapper scripts around build.sh, and I also know that my scripts were ad-hoc, ugly and non-reusable. (Also, of course, anybody new to NetBSD will certainly not find any of the above trivial, but that's secondary.)

To this end, I have written a script that automates all the aforementioned steps in a single tool, driven by configuration files that specify what to do. Enter sysbuild.

My setup

These days, my main NetBSD development box is a virtual machine. For a couple of years, I have been carrying around this virtual machine across machines and slowly tuning its configuration to suit my needs. The way I work is the following:

I keep a read-write copy of the sources under my home directory, which is the one I use for my development work. This copy is accompanied by a script in my home directory that, given a machine type, updates the sources and rebuilds NetBSD for that machine type, using a known directory layout within my home directory.

I also keep a read-only copy of the sources under /usr/src, checked out from anoncvs. This copy is used by the dedicated "builder" system user to perform dedicated rebuilds of NetBSD. The "builder" user also has a custom script (but different than the other one!) that updates /usr/src, rebuilds NetBSD for the machine types I am interested in, records full logs to files and places the build results into a shared network drive. Lastly, the dedicated "builder" user has a cron job that runs the previous script overnight.

The purpose of this dual setup is to always have a fresh release build somewhere in the system, built from pristine sources, while at the same time allowing me to do my development work in a tree that I could break at any time.

Unfortunately, the size of the virtual disk of my development box has become too small for my current needs, and in order to fix this I prefer to start afresh. The thought of having to set up this whole scheme all over again, by hand, is what triggered the creation of sysbuild.

Getting started

sysbuild lives in pkgsrc under the pkgsrc/sysutils/sysbuild directory. This package provides the main script, tests, sample configuration files and the extensive manual page. Installing the script is as easy as installing any other package, and I would recommend reading its documentation now.

Once installed, sysbuild should be ready to use out of the box assuming that you have write access to /usr/src. Simply typing sysbuild build from anywhere in the system will cause the source tree to be updated and a release for your current platform to be built.

If you wish to customize the behavior of sysbuild, copy /usr/pkg/etc/sysbuild/default.conf to ~/.sysbuild/default.conf and edit the latter. The tool will pick it up on the next execution and use your new configuration settings. You can, of course, manage a variety of configurations in case you want to do different builds (say -current and 6.x).

Setting up a daily cron job

The pkgsrc/sysutils/sysbuild-user package is a convenience bundle that manages a "sysbuild" unprivileged user in the system and installs a sample crontab to perform the desired periodic rebuilds of NetBSD. This crontab uses the sysbuild4cron script, which is a very simple utility to run sysbuild, store its output in a file, and send an error report any time the execution fails.

Simply installing the package will cause the creation of the "sysbuild" user and the configuration of this sample cron job. Follow the instructions provided by the package in its MESSAGE file to tune the behavior of any of these.

Concluding trivia

Yes, these scripts are extremely tied to my particular workflow and use cases, although I don't think my needs are that special. However, I have attempted to come up with a generic-enough script and configuration file to allow the addition of new features.

You may notice that sysbuild's version is 2.0 and not 1.0. While preparing the script for addition to pkgsrc, cvs add barfed that I was re-adding previously-deleted files. Uh hum, what a surprise! Upon reading the CVS log of the package's Makefile, I found that 10 years ago I already wrote this exact same script and that two years after it got deleted because it stopped working. I had completely forgotten about this! However, seeing pkg_comp's messy code (which was written around the same time), I imagine that the previous implementation of this idea was messy; I don't dare to look at the previous code.

Enjoy!

July 25, 2012 · Tags: announce, netbsd, sysbuild
Continue reading (about 5 minutes)

Encrypted disk images in NetBSD

When I joined the NetBSD Board of Directors, I was trusted with access to private information and one of the prerequisites for downloading such data was to be able to store it safely: i.e. inside an encrypted volume. I initially went the easy route by putting the data in my laptop, which already uses FileVault 2 to encrypt the whole disk. But soon after, I decided that it would be nicer to put this data in my NetBSD home server, which is the machine that is perpetually connected to IRC and thus the one I use to log into the weekly meetings.

At that point, I was faced with the "challenge" to set up an encrypted volume under NetBSD. I knew of the existence of cgd(4): cryptographic disk driver but had never used it. It really is not that hard, but there are a bunch of manual steps involved so I've been meaning to document them for almost a year already.

Because my home server is a NetBSD/macppc and reinstalling the system is quite a nightmare, I did not want to go through the hassle of repartitioning the disk just to have an encrypted partition of 1GB. The choice was to create a disk image instead, store it in /secure.img and mount it on /secure. Let's see how to do this.

The first step is to create the disk image itself and "loop-mount" it (in Linux terms). We do this the traditional way, by creating an empty file and setting up a vnd(4): vnode disk driver device to access it:

# dd if=/dev/zero of=/secure.img bs=1m count=1024
# vnconfig -c /dev/vnd2 /secure.img

The next step is to configure a new cgd(4) device. This block-level device is a layer placed on top of a regular storage device, and is in charge of encrypting the data to the lower-level device transparently. cgdconfig(8), the tool used to configure cgd(4) devices, stores control information on a device basis under /etc/cgd/. The control information can be generated as follows:

# cgdconfig -g -o /etc/cgd/vnd2c aes-cbc 192

This creates the /etc/cgd/vnd2c configuration file, which you can choose to inspect now. With the file in place, configuring the cryptographic device for the first time is easy:

# cgdconfig cgd0 /dev/vnd2c

This command will inspect the underlying /dev/vnd2c device looking for a signature indicating that the volume is valid and already encrypted. Because the volume is still empty, the tool will proceed to ask us for an encryption key to initialize the volume. We enter our key, wait a couple of seconds, and /dev/cgd0 will be ready to be used as a regular disk device. With that in mind, we can proceed to create a new file system and mount it on the desired location:

# newfs -O 2 /dev/rcgd0c
# mkdir /secure
# mount /dev/cgd0c /secure

To simplify access to the device for your users, I did something like this:

# user=your-user-name
# mkdir /secure/${user}
# chown ${user}:users /secure/${user}
# chmod 700 /secure/${user}
# ln -s /home/${user}/secure /secure/${user}

And we are done. Now, depending on the situation, we could choose to get /secure automatically mounted during boot, but that would involve having to type the decryption key. This is OK for a laptop, but not for a headless home server. Because I don't really need to have this secure device mounted all the time, I have the following script in /root/secure.sh that I can use to mount and unmount the device at will:

#! /bin/sh

set -e -x

case "${1:-mount}" in
mount)
    vnconfig -c /dev/vnd2 /secure.img
    cgdconfig cgd0 /dev/vnd2c
    fsck -y /dev/cgd0c
    mount /dev/cgd0c /secure
    ;;
unmount)
    umount /secure
    cgdconfig -u cgd0
    vnconfig -u /dev/vnd2
    ;;
esac

Be aware that cgdconfig(1) has many more options! Take a look at the manual page and choose the ones that best suit your use case.

Random idea: instead of using a vnd(4) device, plug in a USB stick and use that instead for your secure data! (I actually do this too to back up sensitive information like private keys.)

February 17, 2012 · Tags: howto, netbsd
Continue reading (about 4 minutes)

NetBSD 6.0 BETA tagged

Dear users of NetBSD,

I am pleased to announce that we (well, the release engineering team!) have just tagged the netbsd-6 branch in the CVS repository and thus opened the gate for testing of NetBSD 6.0_BETA. New binary snapshots should start appearing in the daily FTP archive soon. You can, of course, perform a cvs update -r netbsd-6 on your existing source tree and roll your own binaries (as I'm already doing on my home server).

Please help us make NetBSD 6.0 the best release ever! As you may already know, it will provide tons of new features compared to the ancient 5.x release series that will make your user experience much better. The branch just needs a bit of love to shake out any critical bugs that may be left and to ensure that all those annoying cosmetic issues are corrected. So, if you find any of those, do not hesitate to file a problem report (PR).

Thank you!

February 15, 2012 · Tags: netbsd, release
Continue reading (about 1 minute)

Joining the Board of Directors of The NetBSD Foundation

Dear readers,

It is a great pleasure for me to announce that I have just joined the Board of Directors of The NetBSD Foundation.

If you are curious about how this all happened, here it goes: as described in the election procedure, someone who I don't know nominated me back in November of 2010 to become part of the new board composition. After the Nomination Committee made their way through the long list of nominees, interviews and deliberation, they proposed a slate for the new members of the board. This slate included two people who were renewing their previous term (tron@ and reed@) and two new members (spz@ and jmmv@) to replace the two members stepping down (agc@ and david@). The final approval of the proposed slate happened in April 2011 and, yesterday, it became official.

Looking back, I can't believe it has been already ~10 years since I first started using NetBSD. Ah, those were the times of 1.5. My responsibilities within the project have shifted a lot during this time, ranging from the maintenance of GNOME and several web site tasks, the development of the testing framework (which is still ongoing today), and, from today, to my new duties within the board.

I'm looking forward to working on this new area of the project and I hope to meet the requirements of such position. The two members being replaced will be missed, and keeping up to the high bar they left behind will be tough. That said, I hope spz@ and I will be able to meet the expectations.

If you have any ideas or concerns regarding the direction of The NetBSD Project, don't hesitate to send them to the board@!

May 22, 2011 · Tags: netbsd
Continue reading (about 2 minutes)

Sticky bit trivia

Did you ever wonder where the "sticky" part of the "sticky bit" name comes from? I actually didn't, but I just came across the Sticky bit page on Wikipedia through a tweet from @AaronToponce and discovered why.

If you have used any "recent" (the quotes are important) Unix-like system, you probably know what the sticky bit is used for: to restrict the deletion of files in a directory to, basically, the owner of such files. The sticky bit is used on /tmp (among other directories) so that anyone can create temporary files in it but only those that created the temporary files can delete them.

But that's not all that is to know.

The original purpose of the sticky bit was to mark frequently-used executable files so that the operating system kept their text segment on swap space. This speeded up subsequent executions of such programs because the system would not need to access the file system to load the binary: it could just use the image already kept in the swap space. This behavior became obsolete with the advent of memory-mapped executables.

Now it's clear why the sticky bit has the name it has, isn't it? But still, that's not all that is to know.

SunOS 4 introduced a new behavior for the sticky bit on regular, non-executable files: reads and writes to such files would bypass the buffer cache, thus basically telling the system to perform raw I/O on those files. This was particularly useful on swap files when NFS was involved.

With the above, I have just tried to summarize you the information that is in NetBSD's chmod(2) and sticky(7) manual pages; they contain much more detailed information. (And yep, that's right: contrary to what the Wikipedia article says, NetBSD does not support this behavior any more.) Hope you found this insightful if you did not know this little piece of history!

December 21, 2010 · Tags: netbsd
Continue reading (about 2 minutes)

Introducing Kyua

Wow. I have just realized that I have not blogged at all about the project that has kept me busy for the past two months! Not good, not good. "What is this project?", I hear. Well, this project is Kyua.

A bit of background first: the Automated Testing Framework, or ATF for short, is a project that I started during the Summer of Code of 2007. The major goal of ATF was, and still is, to provide a testing framework for the NetBSD operating system. The ATF framework is composed of a set of libraries to aid in the implementation of test cases in C, C++ and shell, and a set of tools to ease the execution of such test cases (atf-run) and to generate reports of the execution (atf-report).

At that point in time, I would say that the original design of ATF was nice. It made test programs intelligent enough to execute their test cases in a sandboxed environment. Such test programs could be executed on their own (without atf-run) and they exposed the same behavior as when they were run within the runtime monitor, atf-run. On paper this was nice, but in practice it has become a hassle. Additionally, some of these design decisions mean that particular features (in particular, parallel execution of tests) cannot be implemented at all. At the end of 2009 and beginning of 2010, I did some major refactorings to the code to make the test programs dumber and to move much of the common logic into atf-run, which helped a lot in fixing the major shortcomings encountered by the users... but the result is that, today, we have a huge mess.

Additionally, while ATF is composed of different modules conceptually separate from each other, there is some hard implementation couplings among them that impose severe restrictions during development. Tangentially, during the past 2 years of working at Google (and coding mainly in Python), I have been learning new neat programming techniques to make code more testable... and these are not followed at all by ATF. In fact, while the test suite of ATF seems very extensive, it definitely is not: there are many corner cases that are not tested and for which implementing tests would be very hard (which means that nasty bugs have easily sneaked in into releases).

Lastly, a very important point that affects directly the success of the project. Outsiders that want to contribute to ATF have a huge entry barrier: the source repository is managed by Monotone, the bug tracker is provided by Gnats (a truly user-unfriendly system), and the mailing lists are offered by majordomo. None of these tools is "standard" by today's common practices, and some of them are tied to NetBSD's hosting which puts some outsiders off.

For all the reasons above and as this year has been moving along, I have gotten fed up with the ATF code base. (OK, things are not that bad... but in my mind they do ;-) And here is where Kyua comes into the game.

Kyua is a project to address all the shortcomings listed above. First of all, the project uses off-the-shelf development tools that should make it much, much easier for external people to contribute. Secondly, the project intends to be much more modular, providing a clear separation between the different components and providing code that is easily testable. Lastly, Kyua intends to remain compatible with ATF so that there are no major disruptions for users. You can (and should) think of Kyua as ATF 2.0, not as a vastly different framework.

As of today, Kyua implements a runtime engine that is on par, feature-wise, to the one provided by atf-run. It is able to run test cases implemented with the ATF libraries and it is able to test itself. It currently contains 355 test cases that run in less than 20 seconds. (Compare that to the 536 test cases of ATF, which take over a minute to run, and Kyua is still really far from catching up with all the functionality of ATF.) Next actions involve implementing reports generation and configuration files.

Anyway. For more details on the project, I recommend you to read the original posting to atf-devel or the project's main page and wiki. And of course, you can also download the preliminary source code to take a look!

Enjoy :-)

December 16, 2010 · Tags: atf, kyua, netbsd
Continue reading (about 4 minutes)

Creating atf-based tests for NetBSD src

Thanks to Antti Kantee's efforts, atf is seeing increasing visibility in the NetBSD community during the past few months. But one of the major concerns that we keep hearing from our developers is "Where is the documentation?". Certainly I have been doing a pretty bad job at that, and the current in-tree documents are a bit disorganized.

To fix the short-term problem, I have written a little tutorial that covers pretty much every aspect that you need to know to write atf tests and, in particular, how to write such tests for the NetBSD source tree. Please refer to the official announcement for more details.

Comments are, of course, welcome! And if you can use this tutorial to write your first tests for NetBSD, let me know :-)

September 3, 2010 · Tags: atf, netbsd
Continue reading (about 1 minute)

Testing NetBSD: Easy Does It

Antti Kantee has been, for a while, writing unit/integration tests for the puffs and rump systems (for which he is the author) shipped with NetBSD. Recently, he has been working on fixing the NetBSD test suite to report 0 failures in the i386 platform so as to encourage developers to keep it that way while doing changes to the tree. The goal is to require developers to run the tests themselves before submitting code.

Antti has just published an introductory article, titled Testing NetBSD: Easy Does It, that describes what ATF and Anita are, how to use them and how they can help in NetBSD development and deployment. Nice work!

June 24, 2010 · Tags: atf, netbsd
Continue reading (about 1 minute)

ATF 0.9 released (late announcement)

Oops! Looks like I forgot to announce the release of ATF 0.9 here a couple of weeks ago. Just a short notice that the formal release has been available since June 3rd and that 0.9 has been in NetBSD since June 4th!

You can also enjoy a shiny-new web site! It even includes a FAQ!

And, as a side note: I have added a test target to the NetBSD Makefiles, so now it's possible to just do make test within any subdirectory of src/tests/ and get what you expect.

June 18, 2010 · Tags: atf, netbsd
Continue reading (about 1 minute)

ATF 0.8 imported into NetBSD

Finished importing ATF 0.8 into the NetBSD source tree. Wow, the CVS import plus merge was much easier than I expected.

Note that, while the NetBSD test suite should continue to work as usual, there are some backwards incompatible changes in the command line interface of test programs. If you are used to run them by hand, expect different results. Please read the release news for details.

Now let's wait for complaints about broken builds! And enjoy this new release in your NetBSD-current system!

May 8, 2010 · Tags: atf, netbsd
Continue reading (about 1 minute)

New gio-fam package

As briefly outlined in the previous post, new versions of Glib provide GIO, a library that intends to be a low-level file system API on top of the POSIX interface. This library provides an interface to asynchronously wait for file system change notifications including the creation, deletion and modification of files.

The monitoring functionality in GIO is modular: it is backed by different loadable plugins that implement OS-specific functionality. In particular, GIO uses an inotify module in Linux and a FAM module everywhere else.

Up until now, the devel/glib2 package in pkgsrc provided a build-time option to specify whether to build the GIO FAM plugin or not. Given that this plugin is built as a shared object that is loaded dynamically at run-time, having a build-time option for this is clearly wrong: it gives no choice to those relying on binary packages (e.g. end/new users). Furthermore, it adds a dependency on the ugly-FAM at the very bottom of the huge Gnome dependency chain. (As already stated, FAM is outdated and hard to set up.)

So, based on this, I've just removed all FAM support from devel/glib2 altogether and packaged its loadable module as sysutils/gio-fam.

Now waiting for a clean rebuild of the Gnome packages to see if the desktop now works on my machine by avoiding FAM/Gamin.

April 20, 2010 · Tags: gnome, netbsd, pkgsrc
Continue reading (about 2 minutes)

File system monitoring, Gnome and NetBSD

A few days ago, I decided to start using NetBSD, as well as Gnome on NetBSD once again, mostly because the lack of their use makes my skills feel rusty in many different areas. While NetBSD has surprised me in a good way (I am running it on a Macbook Pro and things like wireless and DRI work), Gnome has not. There are tons of broken things that prevent a smooth user experience.

One of these broken things is the monitoring of changes in the file system. Actually, this has never worked 100%. But what is this and why does it matter, you ask? Well, file system monitoring is an internal component of the Gnome infrastructure that allows the desktop to receive notifications when files or directories change. This way, if, say, you are viewing the Downloads folder in Nautilus and you start downloading a file from Epiphany into that folder, Nautilus will realize the new file and show it immediately without requiring a manual refresh.

How to monitor the file system depends on the operating system. There are basically two approaches: polling and asynchronous notifications. Polling is suboptimal because the notifications are usually delayed. Asynchronous notifications are tied to the operating system: Linux provides inotify, NetBSD provides kqueue and other systems provide their own APIs.
In the past, Gnome monitored the file system by a combination of FAM, a system-level service that provides an API to file system monitoring, and GNOME VFS, a high-level layer that hides the interaction with FAM. This approach was good in spirit (client/server separation) but didn't work well:

FAM is abandoned.
Does not support kqueue out of the box.
FAM runs as root.
FAM is too hard to set up: it requires rpcbind, an addition to /etc/services, a sysctl tweak, and the configuration of a system-level daemon.

To solve some of these problems, a drop-in replacement for FAM was started. Gamin, as it is known, still does not fix everything:

Gamin is abandoned.
Supports kqueue out of the box, but does not work very well.
Actually, Gamin itself does not work. Running the tests provided by the distfile in a modern Linux system results in several test failures.

Anyway. Did you notice the abandoned pattern above? This is important: in the new world order, Gnome does not use FAM any more.

The new structure to monitor files is: the low-level glib library provides the gio module, which has some file system monitoring APIs. The GVFS module provides higher level abstractions to file system management, and relies on gio for file system monitoring. There is no more GNOME VFS any more.

The key point is: gio uses inotify directly; no abstraction layers in between. FAM support is still there for platforms without inotify, but as it is not used in Linux any more, it rots. Linux developers will never experience what it is to have a system that needs to use FAM to get this functionality to work.

At last, let's look at the status of all this in NetBSD:

The FAM package was patched to support kqueue. Although this kinda works, it is not perfect. Also, as mentioned above, FAM is, I'd say, the package with the hardest installation procedure of the whole Gnome platform.
The Gamin packages are nicer than the FAM package regarding their configuration. However, when using Gamin instead of FAM, all sorts of bugs appear in Gnome (it actually gets stuck during startup for me). The breakage of the unit tests does not provide any confidence, and the fact that Gamin is abandoned, the idea of fixing it doesn't make me thrive.
The glib2 package depends on FAM. This is ugly; really ugly. I had to shout WTF when I saw this, seriously.
Seeing the direction gio/gvfs take, it is obvious that things can only get worse in the future.

If time permits, I'm planning to work on improving this whole situation. Ideas include:

Splitting the FAM gio module out of the glib2 package. Ideally, this would happen upstream.
Implement a gio backend for kqueue.
Check if the core packages still using gnome-vfs have a more recent version that uses gvfs instead and, if so, update them.

Can't promise you anything other than, if I get to work on it, I will keep you posted!

April 20, 2010 · Tags: gnome, netbsd
Continue reading (about 4 minutes)

NetBSD in Google Summer of Code 2010

For the 6th year in a row, NetBSD is a mentoring organization for Google Summer of Code 2010!

If you are a bright student willing to develop full-time for an open source project during this coming summer, consider applying with us! You will have a chance to work with very smart people and, most likely, in the area that you are most passionate about. NetBSD, being an operating system project, has offers for project ideas at all levels: from the kernel to the packaging system, passing by drivers, networking tools, user-space utilities, the system installer, automation tools and more!

I would like to point you at the 3 project proposals I'm willing to directly mentor:

Optimize and speed-up ATF: Make the testing framework blazing fast so that running the NetBSD automated tests does not take ages on slow platforms.
Reorganize ATF to improve modularity: Refactor pieces of the testing framework so that it is easier to redistribute, has cleaner interfaces and is easier to depend on from third-party projects.
Rewrite pkg_comp with portability as a major goal: Use Python to create a tool to automatically build binary packages from within a sandbox.

If you find any of the above projects interesting, or if you have any other project proposal that you think I could mentor, do not hesitate to contact me. Feel free to send me a draft of your application, together with a bit of information about you, so that we can discuss your proposal and make sure it gets selected!

Or, if none of the projects above interests you, please do check out the full list of NetBSD project proposals. I'm sure you will find something that suits your interests :-)

March 19, 2010 · Tags: atf, netbsd, pkgsrc, soc
Continue reading (about 2 minutes)

New version of the monotone-server package in pkgsrc

Wow, it has been a long time... 5 years ago, I created the monotone-server package in pkgsrc, a package that provided an interactive script to set up a monotone server from scratch with, what I though, minimal hassle.

My package did the job just fine, but past year I was blown away by the simplicity of the same package in Fedora: their init.d script provides a set of extra commands to initialize the server before starting it up, and that is it. No need to mess with a separate interactive script; no need to create and memorize passphrases that you will never use; and, what's more, all integrated in the only single place that makes sense: in the init.d "service management" script.

It has been a while since I became jealous of their approach, but I've finally got to it: I've spent the last few days rewriting the monotone-server package in pkgsrc and came up with a similar scheme. And this new package just made its way to pkgsrc-HEAD! The new package comes with what I think is a detailed manual page that explains how to configure the server from scratch. Take a look and, if you find any mistakes, inconsistencies or improvements to be done, let me know!

In the meantime, I will log into my home server, rebuild the updated package and put it in production :-)

March 12, 2010 · Tags: monotone, netbsd, pkgsrc
Continue reading (about 2 minutes)

Installing NetBSD/macppc on a Mac Mini G4

Yesterday, I spent a while installing NetBSD/macppc 5.0.1 on a Mac Mini G4. The process wasn't easy, as it involved the following steps. I'm omitting many details, as they are "common knowledge" to Mac users (or otherwise can be easily found on the net):

After booting the installer from the CD image, drop into the shell.
Use pdisk to create an Apple_HFS partition for the boot loader and two Apple_UNIX_SVR2 partitions, one for the root file system and another for swap.
Run sysinst and install the system. When asked to repartition the disk, just say Use existing partition sizes.
Once the system is installed, drop again into the shell before rebooting.
Mount your hard disk into /mnt and chroot into it.
Fetch a copy of pkgsrc.
Install the sysutils/hfsutils package.
Use hformat to create a new HFS file system in the Apple_HFS partition we created.
Mount the installation CD.
Copy, using hcopy, the ofwboot.xcf file from the CD to the boot partition.
Reboot.
Drop into the OpenFirmware setup (Command+Option+P+R).
Set boot-device to hd:,ofwboot.xcf.
Set boot-file to netbsd.
And here is the tricky thing to get the machine to auto-boot: Set boot-command to ." hello" cr " screen" output boot, not mac-boot.

I found the last command somewhere on the Internet (dunno where now), but, supposedly, a regular mac-boot should have worked. In fact, it works if you call this command from the prompt, but not during automatic boot. (It turns out to be a problem with the version of OpenFirmware I have.)

Just writing down the steps in case I need them later on. Installing Debian stable was much, much easier, but the installer for testing crashes every day with a different error, so I gave up.

(Oh, by the way, I did the same installation into an old PowerMac G3 and that was really painful. The machine refused to boot from any of the CDs I tried and the prebuilt kernels hang during initialization due to a bogus driver. In the end: netbooting and using custom kernels.)

January 11, 2010 · Tags: mac, netbsd
Continue reading (about 2 minutes)

The NetBSD Blog

The NetBSD Project recently launched a new official blog for NetBSD. From here, I'd like to invite you to visit it and subscribe to it. It's only with your support (through reading and, specially, commenting) that developers will post more entries! Enjoy :-)

May 4, 2009 · Tags: netbsd
Continue reading (about 1 minute)

NetBSD-SoC needs your application!

The Google Summer of Code 2009 application deadline for students is tomorrow and NetBSD has got very few applications so far. If you have the interest in working on a cool operating system project, where almost any project idea can fit, take the time to read our proposals and apply! New, original ideas not listed there will also be considered.

It'd be a pity if the number of assigned slots to NetBSD was small due to the low number of applications! We did much better past year.

Note that there are a couple of ATF-related proposals in there. Help will be certainly welcome (by me ;-) in those areas!

April 2, 2009 · Tags: netbsd, soc
Continue reading (about 1 minute)

userconf support for the boot loader

I have a machine at work, a Dell Optiplex 745, that cannot boot GENERIC NetBSD kernels. There is a problem in one of the uhci/ehci, bge or azalia drivers that causes a lockup at boot time because of a shared interrupt problem. Disabling ehci or azalia from the kernel lets the machine boot. In order to do that, there are two options: either you rebuild your kernel without the offending driver, or you boot into the userconf prompt with -c and, from there, manually disable the driver at each boot. None of the options are quite convincing.

Of course, disabling a faulty driver is not the correct solution, but the workaround is useful on its own. I've just added a userconf command to the boot loader and its configuration file -- /boot and /boot.cfg respectively -- so that the end user can pass random userconf commands to the kernel in an automated way. userconf is a kernel feature that lets you change the parameters of builtin drivers and enable/disable them before the hardware detection routines are run.

With this new feature in the boot loader, you can customize a GENERIC kernel without having to rebuild it! Yes, modules could help here too, but we are not there yet for hardware drivers. Note that OpenBSD has had a similar feature for a while with config -e, but they actually modify the kernel binary.

You can check the patch out and comment about it in my post to tech-kern.

January 22, 2009 · Tags: netbsd
Continue reading (about 2 minutes)

ATF 0.6 released

I am very happy to announce the availability of the 0.6 release of ATF. I have to apologize for this taking so long because the code has been mostly-ready for a while. However, doing the actual release procedure is painful. Testing the code in many different configurations to make sure it works, preparing the release files, uploading them, announcing the new release on multiple sites... not something I like doing often.

Doing some late reviews, I have to admit that the code has some rough edges, but these could not delay 0.6 any more. The reason is that this release will unblock the NetBSD-SoC atfify project, making it possible to finally integrate all the work done in it into the main NetBSD source tree.

Explicit thanks go to Lukasz Strzygowski. He was not supposed to contribute to ATF during his Summer of Code 2008 project, but he did, and he actually provided very valuable code.

The next step is to update the NetBSD source tree to ATF 0.6. I have extensive local changes for this in my working copy, but I'm very tired at the moment. I think I'll postpone their commit until tomorrow so that I don't screw up something badly.

Enjoy it and I'm looking for your feedback on the new stuff!

January 18, 2009 · Tags: atf, netbsd, soc
Continue reading (about 2 minutes)

pwd_mkdb and the new time_t

NetBSD-current has recently switched time_t to be a 64-bit type on all platforms to cope with the year-2038 problem. This is causing all sorts of trouble, and a problem I found yesterday was that, after a clean install of NetBSD/amd64, it was impossible to change the data of any user through chfn. The command failed with:

chfn: /etc/master.passwd: entry root inconsistent expire chfn: /etc/master.passwd: unchanged

Suspiciously, the data presented by chfn showed an expiration date for root set in a seemingly-random day (October 14th, 2021). That seemed like some part of the system not parsing the user database correctly and generating random values. A sample test program that walked through the passwords database with getpwent(3) showed an invalid expiration date for root, even when /etc/master.passwd had a 0 in that field.

After some debugging, I found out that libc tries to be compatible with old-format binary password databases (those generated by pwd_mkdb). In order to deal with compatibility, libc checks to see if the database has a VERSION field set in it. If not, it assumes it is an old version and thus time_t may not be 64-bit. If the VERSION field is set, it uses the new time_t.

So what was the problem? pwd_mkdb did not set the VERSION field even though it wrote a new-format database. libc assumed it was laid out according to the old format but it was not, so it got garbage data during parsing. After some hacking, I fixed it as described in a post to current-users.

Soon after, Christos Zoulas (the one who did all the time_t work) told me that he had already done these changes in his branch but forgot to merge them. He did now, and the code in the main branch should work fine. I still think there is a minor problem in it, but the major issue after installation should be gone.

January 15, 2009 · Tags: netbsd
Continue reading (about 2 minutes)

Kernel modules in NetBSD/shark (or other ARMs)

I reinstalled NetBSD-current recently on my shark (Digital DNARD) and, out of curiosity, I wanted to see if the new-style kernel modules worked fine on this platform. To test that, I attempted to load the puffs module and failed with an error message saying something like "kobj_reloc: unexpected relocation type 1". Similarly, the same error appeared when running the simpler regression tests in /usr/tests/modules.

After seeing that error message, I tracked it down in the source code and ended in src/sys/arch/arm/arm32/kobj_machdep.c. A quick look at it and at src/sys/arch/arm/include/elf_machdep.h revealed that the kernel was lacking support for the R_ARM_PC24 relocation type. "It can't be too difficult to implement", I thought. Hah!

Based on documentation, I understood that R_ARM_PC24 is used in "short" jumps. This relocation is used to signal the runtime system that the offset to the target address of a branch instruction has to be relocated. This offset is a 24-bit number and, when loaded, it has to be shifted two bits to the left to accommodate for the fact that instructions are 32-bit aligned. Before the relocation, there is some addend encoded in the instruction that has to be loaded, sign-extended and shifted two bits to the left and, after all that, added to the calculated address.

I spent hours trying to implement support for the R_ARM_PC24 relocation type because it didn't want to work as expected. I even ended up looking at the Linux code to see how they dealt with it, and I found out that I was doing exactly the same as them. So what was the problem? A while later I realized that this whole thing wasn't working because the relocated address to be stored in the branch instruction didn't fit in the 24 bits! That makes things harder to solve.

At that point, I looked at the port-arm mailing list and found that several other people were looking at this same issue. Great, some time "wasted" but a lot of new stuff learnt. Anyway, it turns out there are basically two solutions to the problem described above. The first involves generating jump trampolines for the addresses that fall too far away. The second one is simpler: just change the kernel to load the modules closer to the kernel text, and thus make the jump offsets fit into the 24 bits of the instructions. Effectively, there is a guy that has got almost everything working already.

Let's see if they can get it working soon!

January 2, 2009 · Tags: netbsd
Continue reading (about 2 minutes)

ATF talk at NYCBSDCon 2008

NYCBSDCon 2008 will take place in New York City on October 11th and 12th. Given that I am already in NYC and will still be by that time, I submitted a presentation proposal about ATF. I have just been notified that my proposal has been accepted and, therefore, I will be giving a talk on ATF itself and how it relates to NetBSD on one of those two days. The conference program and schedule have not been published yet, though, so keep tuned. Hope to see you there! :)

July 30, 2008 · Tags: atf, netbsd, nyc
Continue reading (about 1 minute)

NetBSD talk at Isla Cristina

Yesterday night, I got back from the "I Jornadas Tecnológicas Isla Cristina", a small technological conference organized at Isla Cristina, a little town in Huelva, Spain.

The main organizers were the teachers of a local technical school (the IES Padre José Miravent), and they invited me to give a talk about NetBSD development. I will publish the slides soon, but I have to warn you that you will not like the source format, aka PowerPoint. Being part of the university personnel, I was given a copy of Office 2008 for Mac and I wanted to give it a serious try before judging it. It is certainly more powerful (or easy to use) than OpenOffice Impress, but it is also a lot slower; I don't know what they have done there, but the application feels really really sluggish.

Anyway, back to the point of the conference. It has been great and surpassed all the expectations I had. The organization was excellent, the people was very nice, the food was (very) abundant and the talks were interesting (except for a couple of exceptions). What else could you ask for?

Just as a point of fact, there were around 300 registered people, and I guess around 100 of them came to my talk (it was first hour in the morning); that's a lot more public than I have ever had before, and it was a really exciting thing. I hope the listeners enjoyed it as much as I did.

The only thing I regret was not staying there one more day (after the conference) so I could go around the town and take some cool photos. Maybe next year :-) Ah, speaking of next year: if you get invited to give a talk, don't think twice and accept the offer!

April 5, 2008 · Tags: conference, netbsd
Continue reading (about 2 minutes)

Google Summer of Code 2008 and NetBSD

Google has launched the Summer of Code program once again this year, and NetBSD is a mentoring organization for the fourth time as announced in a netbsd-announce post. Unless things go very wrong in the following days, I will not take part this year as a student because I will be intering at Google SRE during the Summer!

However, I will try to become a mentor for the "Convert all remaining regression tests to ATF" project. If you are looking for some interesting idea to apply for, this is a good one! Why?

It will let you get into NetBSD internals in almost all areas of the system: you'll need to understand how the source tree is organized, how to add new components to it (because tests are almost in all aspects regular programs), how the current pieces of the system interact with each other...
You will need to gain knowledge in some areas (such as the kernel or the libraries) to be able to port tests from the old framework (if it deserves that name ;-) to the new one and, if you are really up to it, even add new tests for functionality that is currently uncovered by the test suite. But adding new tests is something you will not be required to do, because the sole task of migrating the existing ones is a huge task already.
Get involved in ATF's development because, as you study the existing test cases and their requirements, you will most likely find that it lacks some important functionality to make things really straightforward.
And, of course, make a unvaluable contribution to the NetBSD operating system. Having a public test suite with high coverage means that the system will gain quality. Yes, you will most likely uncover bugs in many areas of the system and give them enough exposure so that someone else may fix them.

Note that this project is really a Summer of Code project. It does not have a long design phase on its own so, once you have got used to the system and ATF, you'll just code and immediately make useful contributions. In the past, projects that had a heavy design phase involved were not good because, in the end, the student did not finish the code on time.

So... don't hesitate to apply! I'm looking forward to see your applications for this project :-)

March 19, 2008 · Tags: atf, netbsd, soc
Continue reading (about 2 minutes)

Implementing timeouts for test cases

One of the pending to-do entries for ATF 0.4 is (was, mostly) the ability to define a timeout for a test case after which it is forcibly terminated. The idea behind this feature is to prevent broken tests from stalling the whole test suite run, something that is already needed by the factor(6) tests in NetBSD. Given that I want to release this version past weekend, I decided to work on this instead of delaying it because... you know, this sounds pretty simple, right? Hah!

What I did first was to implement this feature for C++ test programs and added tests for it. So far, so good. It effectively was easy to do: just program an alarm in the test program driver and, when it fires, kill the subprocess that is executing the current test case. Then log an appropriate error message.

The tests for this feature deserve some explanation. What I do is: program a timeout and then make the test case's body sleep for a period of time. I try different values for the two timers and if the timeout is smaller than the sleeping period, then the test must fail or otherwise there is a problem.

The next step was to implement this in the shell interface, and this is where things got tricky. I did a quick and dirty implementation, and it seemed to make the same tests I added for the C++ interface pass. However, when running the bootstrap testsuite, it got stalled at the cleanup part. Upon further investigation, I noticed that there were quite a lot of sleep(1) processes running when the testsuite was stalled, and killing them explicitly let the process continue. You probably noticed were the problem was already.

When writing a shell program, you are forking and executing external utilities constantly, and sleep(1) is one of them. It turns out that in my specific test case, the shell interpreter is just waiting for the sleep subprocess to finish (whereas in the C++ version everything happens in a single process). And, killing a process does not kill its children. There you go. My driver was just killing the main process of the test case, but not everything else that was running; hence, it did not die as expected, and things got stalled until the subprocesses also died.

Solving this was the fun part. The only effective way to make this work is to kill the test case's main process and, recursively, all of its children. But killing a tree of processes is not an easy thing to do: there is no system interface to do it, there is no portable interface to get a list of children and I'm yet unsure if this can be done without race conditions. I reserve the explanation of the recursive-kill algorithm I'm using for a future post.

After some days of work, I've got this working under Mac OS X and also have got automated tests to ensure that it effectively works (which were the hardest part by far). But as I foresaw, it fails miserably under NetBSD: the build was broken, which was easy to fix, but now it also fails at runtime, something that I have not diagnosed yet. Aah, the joys of Unix...

January 15, 2008 · Tags: atf, netbsd, process
Continue reading (about 3 minutes)

Fixing id's command line parsing

Today's work: been fixing NetBSD's id(1)'s command line parsing to match the documented syntax. Let's explain.

Yesterday, for some unknown reason, I ended up running id(1) with two different user names as its arguments. Mysteriously, I only got the details for the first user back and no error for the second one. After looking at the manual page and what the GNU implementation did, I realized that the command is only supposed to take a single user or none at all as part of its arguments.

OK, so "let's add a simple argc check to the code and raise the appropriate error when it is greater than 2". Yeah, right. If you look at id(1)'s main routine, you'll find an undecipherable piece of spaghetti code — have you ever thought about adding multiple ?flag variables and checking the result of the sum? — that comes from the fact that id(1)'s code is shared across three different programs: id(1), groups(1) and whoami(1).

After spending some time trying to understand the rationale behind the code, I concluded that I could not safely fix the problem as easily as I first thought. And, most likely, touching the logic in there would most likely result in a regression somewhere else, basically because id(1) has multiple primary, mutually-exclusive options and groups(1) and whoami(1) are supposed to have their own syntax. Same unsafety as for refactoring it.

So what did I do? Thanks to ATF being already in NetBSD, I spent the day writing tests for all possible usages of the three commands (which was not trivial at all) and, of course, added stronger tests to ensure that the documented command line syntax was enforced by the programs. After that, I was fairly confident that if I changed the code and all the new tests passed afterwards (specially those that did before), I had not broken it. So I did the change only after the tests were done.

I know it will be hard to "impose" such testing/bug-fixing procedure to other developers, but I would really like them to consider extensive testing... even for obvious changes or for trivial tools such as these ones. You never know when you break something until someone else complains later.

November 16, 2007 · Tags: atf, netbsd
Continue reading (about 2 minutes)

ATF 0.3 released

I've just published the 0.3 release of ATF. I could have delayed it indefinitely (basically because my time is limited now), so I decided it was time to do it even if it did not include some things I wanted.

The important thing here is that this release will most likely be the one to be merged into the NetBSD source tree. If all goes well, this will happen during this week. Which finally will give a lot of exposure to the project :-)

November 11, 2007 · Tags: atf, netbsd
Continue reading (about 1 minute)

SoC: Second preview of NetBSD with ATF

Reposting from the original ATF news entry:

I have just updated the first preview of NetBSD-current release builds with ATF merged in to match the ATF 0.1 release published today. As already stated in the old news item: These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF is imported into it; we are not sure when that will happen, though.
August 20, 2007 · Tags: atf, netbsd, soc
Continue reading (about 1 minute)

SoC: First preview of NetBSD with ATF

Reposting from the original ATF news entry:

I have just uploaded some NetBSD-current release builds with ATF merged in. These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation. They will give you a preview of how a NetBSD installation will look like once ATF 0.1 is made public, which should happen later this month.

For more details see my post to the NetBSD's current-users mailing list.

Waiting for your feedback :-)

Edit (Aug 20th): Fixed a link.

August 8, 2007 · Tags: atf, netbsd, soc
Continue reading (about 1 minute)

SoC: Getting started

This weekend I have finally been able to start coding for my SoC project: the Automated Testing Framework for NetBSD. To my disliking, this has been delayed too much... but I was so busy with my PFC that I couldn't find any other chance to get my hands on it.

I've started by working on two core components:

libatf: The C/C++ library that will provide the interface to easily write test cases and test suites. Among its features will be the ability to report the results of each test, a way to define meta-data for each test, etc.
libatfmain: A library that provides a default entry point for test programs that takes care to run the test cases in a controlled environment — e.g. it captures all signals and deals with them gracefully — and provides a standard command-line interface for them.

Soon after I started coding, I realized that I could need to write my own C code to deal with data collections and safe strings. I hate that, because that's a very boring task — it is not related to the problem at hand at all — and because it involves reinventing the wheel: virtually all other languages provide these two features for-free. But wait! NetBSD has a C++ compiler, and atf won't be a host tool¹. So... I can take advantage of C++, and I'll try to. Calm down! I'll try to avoid some of the "complex" C++ features as much as possible to keep the executables' size small enough. You know how the binaries' size blows up when using templates... Oh, and by the way: keep in mind that test cases will be typically written in POSIX shell script, so in general you won't need to deal with the C++ interface.

Furthermore, I see no reason for atf to be tied to NetBSD. The test cases will surely be, but the framework needn't. Thus I'm thinking of creating a standalone package for atf itself and distributing it as a completely independent project (under the TNF² umbrella), which will later be imported into the NetBSD source tree as we currently do for other third-party projects such as Postfix. In fact, I've already started work on this direction by creating the typical infrastructure to use the GNU auto-tools. Of course this separation could always be done at a later step in the development, but doing it from the very beginning ensures the code is free of NetBSD-isms, emphasizes the portability desire and keeps the framework self-contained.

I'd like to hear your comments about these "decisions" :-)

¹ A host tool is a utility that is built with the operating system's native tools instead of with the NetBSD's tool-chain: i.e. host tools are what build.sh tools builds. Such tools need to be highly portable because they have to be accepted by old compilers and bizarre build environments.

² TNF = The NetBSD Foundation.

June 24, 2007 · Tags: atf, netbsd, soc
Continue reading (about 3 minutes)

NetBSD's website redesign

Even though I don't usually repost general NetBSD news, I would like to mention this one: the NetBSD web site has got a severe facelifting aiming at improving its usability and increasing the consistency among its pages.

Many thanks to Daniel Sieger for his perseverance and precious work. This is something that had been attempted in the past many times but raised so many bikesheds that it was never accomplished.

In case you would like to contribute to the project doing something relatively easy, you can do so now. It could be interesting to revamp many of the existing pages to be more user friendly by reorganizing their contents (simplification is good sometimes!), their explanations and making better use of our XML-based infrastructure. Keep in mind that the web site is the main "entry point" to a project and newcomers should feel very comfortable with it; otherwise they will go away in less than a minute!

Furthermore, it'd be nice to see if there are any plain HTML pages left and convert them to XML. This could make all those pages automatically use the new look of the site and integrate better with it. (If you don't know what I mean, just click, for example, on the Report or query a bug at the top of the front page. It looks ugly; very ugly. But unfortunately, this is not as simple as to convert the page to XML because it is automatically generated by some script.)

Send your feedback to www AT NetBSD.org or to the netbsd-docs AT NetBSD.org public list.

June 12, 2007 · Tags: netbsd
Continue reading (about 2 minutes)

pkgsrcCon 2007 report

pkgsrcCon 2007 is over. The conference started around 1:00pm on the 27th and has lasted until today^Wyesterday (the 29th) at around 7:00pm. There have been 10 different talks as planned, although we weren't able to follow the proposed schedule. Most of the presentations were delayed and some were shifted because the speakers could not arrive on time. Not a big deal though.

We have been, more or less, around 20-25 people. There were 30 registered, but a couple had to withdraw and some others did not come (for unknown reasons to us). Maybe the conferences were too technical or the schedule did not meet their expectations.

The thing is I originally intended to write a short report after each day, but I have had literally no life outside pkgsrcCon since Thursday night. Taking care of the organization has been time consuming (and I have done almost nothing!), not to mention that recording all the presentations was exhausting; more on this below.

Anyway, it has been an excellent experience. Meeting other pkgsrc developers in person has been very nice, plus they are all also very nice people too. Of course, their presentations were also interesting, and they showed extremely interesting ideas from their creators. I'm specially looking forward to seeing the preliminary results of one of the detailed projects (won't tell you which ;-).

So, what did we do? We started with a pre-pkgsrcCon dinner on Thursday night, which was the first encounter among all developers. The talks started the day afterwards in the afternoon, and when they were over we had some beers and dinner together. Saturday was more of the same, even though it had more talks. We then had dinner on a nicer restaurant; great, great time. The last talks happened on Sunday, after which we had some more beers and the last dinner all together.

I now have to compress all the video recordings and we have yet to decide if we will make them public and how. Stay tuned! (And don't hesitate to join for pkgsrcCon 2008; I'm sure you will enjoy it, at the very least as much as I did!)

April 29, 2007 · Tags: netbsd, pkgsrc, pkgsrccon
Continue reading (about 2 minutes)

SoC: Selected again!

Yes, Google Summer of Code (SoC) 2007 is back and I'm in once again! This means I'll be able to spend another summer working on free software and deliver some useful contributions by the end of it.

This time I sent just one proposal, choosing NetBSD as the mentoring organization. The project is entitled Automated testing framework and is mentored by Martin Husemann. This framework is something I've had in mind for a long time already; in fact, I also applied for this in SoC 2006 and attempted to develop this project as my undergraduate thesis.

For more details on what the project is about, check out these notes.

At last, take a look at the full list of accepted projects for NetBSD. It is rather short unfortunately, but they all look very promising. It is a pity ext3 support is not in them, but getting ZFS instead will be good too.

Edit (April 19th): Fixed a link.

April 12, 2007 · Tags: netbsd, soc
Continue reading (about 1 minute)

X11 mode-line generator

I recently installed NetBSD-current (4.99.12 at the time I did this) inside Parallels Desktop for Mac. Everything went fine except for the configuration of the XFree86 shipped with the base system: I was unable to get high resolutions to work (over 1024x768 if I recall correctly), and I wanted to configure a full-screen desktop. In my specific case, this is 1440x900, the MacBook Pro's native resolution.

It turns out I had to manually add a mode line to the XF86Config file to get that resolution detected. I bet recent X.Org versions do not need this as, e.g. Fedora Core 6 works fine without manual fiddling.

Writing mode lines is not fun, but fortunately I came across an automated generator. In fact, this seems to be just a web-based frontend to the gtf tool provided by NVIDIA. So I entered the appropriate details (x = 1440, y = 900, refresh = 60), hit the Generate modeline button and got:

# 1440x900 @ 60.00 Hz (GTF) hsync: 55.92 kHz; pclk: 106.47 MHz Modeline "1440x900_60.00" 106.47 1440 1520 1672 1904 900 901 904 932 -HSync +Vsync

After that I had to make the HorizSync and VertRefresh values in my configuration file a bit wider to fulfill this mode's request and everything worked fine. Be extremely careful if you mess with synchronization values though; incorrect ones can physically damage a monitor, although I think this is not a problem for LCDs.

March 15, 2007 · Tags: netbsd, parallels, x11
Continue reading (about 2 minutes)

NetBSD and SoC 2007

Yes, ladies and gentlemen: Google Summer of Code 2007 is here and NetBSD is going to become a mentoring organization again (unless Google rejected the application, that is)!

We are preparing a list of projects suitable for SoC; spend some time looking for one that interests you (I'm sure there is something) and get ready to send your proposal between the 14th and 24th of this same month. I've already made my choice :-)

See the official announcement for more details.

March 8, 2007 · Tags: netbsd, soc
Continue reading (about 1 minute)

Article on Multiboot and NetBSD

A bit more than a year ago I started working on Multiboot support for NetBSD. This work was completed by the end of past year and was integrated into the main source tree, allowing any user to boot his NetBSD installation by using GRUB alone, without having to chainload different boot loaders.

I've written an introductory article on Multiboot and how NetBSD was converted to support it. It has just been published at ONLamp. Enjoy reading!

March 2, 2007 · Tags: article, multiboot, netbsd
Continue reading (about 1 minute)

Making NetBSD Multiboot-Compatible

This article first appeared on this date in O’Reilly’s ONLamp.com online publication. The content was deleted sometime in 2019 but I was lucky enough to find a copy in the WayBack Machine. I reformatted the text to fit the style of this site and fixed broken links, but otherwise the content is a verbatim reproduction of what was originally published.

The i386 architecture is full of cruft required to maintain compatibility with old machines that go back as far as the 8086 series. Technically speaking, these features aren’t necessary anymore because any recent computer based on this architecture uses a full 32-bit operating system that could work perfectly fine without the legacy code. Unfortunately, the compatibility hacks remain in place and hurt the development of new software.

March 1, 2007 · Tags: featured, netbsd, onlamp, os, programming
Continue reading (about 13 minutes)

A subject for my undergraduate thesis

It has finally come the time when I have to choose a subject for my undergraduate thesis on which I'll be working on full time next semester. My first idea was to make a contribution to NetBSD by developing an automated testing framework. I have had interest in this for a long while (I even proposed it as part of this year's SoC), and there is a lot of interest in it within the project too.

However, this specific project does not fit correctly into the current research groups at my faculty. This wouldn't be a problem if I wasn't thinking in taking a CS Master or Ph.D. later on. But as I'm seriously considering this possibility, it'd be better if I worked on a project that lets me integrate into an existing research group as early as possible. This could also teach me several new stuff that I'd not learn otherwise: if you look at the paper linked above, you can see I already have several ideas for the testing framework. That is, I already know how I'd address most of it, hence there'd not be a lot of "research". Furthermore, the teacher I talked to about this project felt the core of the project could not be long enough to cover a full semester.

So what are the other possible ideas? I went to talk to a teacher that currently directs some of the research groups and he proposed me several ideas, organized in three areas:

Code analysis and optimization: Here I'd work on tools to analyze existing code and binaries to understand how they work internally; this way one could later generate a better binary by reorganizing related code and/or removing dead bits. They already have done a lot of work on this subject, so I'd be working on a tiny part of it. No matter what, dealing with the compiler/linker and the resulting binaries sounds quite well.

Improve heterogeneous multiprocessor support: This group contains ideas to improve the management of heterogeneous systems such as those based on the Cell processor. I'm "afraid" any project here would be completely Linux-based, but the background idea also feels interesting. Haven't got too much details yet, though.

Distributed systems: This doesn't interest me as much as the other two, but this may be because there was not enough time during the meeting to learn about this group. However, next week we are taking a guided visit to the BSC which will hopefully clear some of my doubts and let me decide if I'm really interested in this area.

I shall make a decision as soon as possible, but this is hard!

Oh, and don't worry about the testing framework project. I'll try to work on this in my spare time because I feel it's something NetBSD really needs and I'm sure I'll enjoy coding it. Not to mention that nowadays, whenever I try to apply any fix to the tree, I feel I should be adding some regression test for it! Plus... I already have a tiny, tiny bit of code :-)

December 15, 2006 · Tags: netbsd, pfc
Continue reading (about 3 minutes)

tmpfs marked non-experimental

The implementation of an efficient memory-based file system (tmpfs) for NetBSD was my Google Summer of Code 2005 project. After the program was over, the code was committed to the repository and some other developers (specially YAMAMOTO Takashi) did several fixes and improvements in it. However, several problems remained in it that prevented tagging it release quality (see this thread).

Finally I found some time to deal with most of them, something that has kept me busy for around three weeks (and which I should have done much, much earlier). All the issues that were resolved are detailed in this other post.

There still are some problems in the code (which code doesn't have any?) but these do not prevent tmpfs from working fine. Of course they should be addressed in the future but people is already enjoying tmpfs in their installations and have been requesting its activation by default for a long time.

Hence, after core@'s blessing, I'm proud to announce that tmpfs has been marked non-experimental and is now enabled by default in the GENERIC kernels of amd64, i386, macppc and sparc64. Other platforms will probably follow soon.

The next logical step is to replace mfs with tmpfs wherever the former is used (e.g. in sysinst) but more testing is required before this happens. And this is what 4.0_BETA will allow users to do :-) Enjoy!

November 11, 2006 · Tags: netbsd, tmpfs
Continue reading (about 2 minutes)

Making vnd(4) work with tmpfs

vnd(4) is the virtual disk driver found in NetBSD. It provides a disk-like interface to files which allows you to treat them as if they were disks. This is useful, for example, when a file holds a file system image (e.g. the typical ISO-9660 files) and you want to inspect its contents.

Up until now vnd(4) used the vnode's bmap and strategy operations to access the backing file. These operate at the block-level and therefore do not involve any system-wide caches; this is why they were used (see below). Unfortunately, some file systems (e.g. tmpfs and smbfs) do not implement these operations so vnd could not work with files stored inside them.

One of the possible fixes to resolve this problem was to make vnd(4) use the regular read and write operations; these act on a higher (byte) level and are so fundamental that must be implemented by all file systems. The disadvantage is that all data that flows through these two methods ends up in the buffer cache. (If I understand it correctly, this is problematic because vnd itself will also push a copy of the same data into the cache thus ending up with duplicates in there.)

Despite that minor problem, I believe it is better to have vnd(4) working in all cases even if that involves some performance penalty in some situations (which can be fixed anyway by implementing the missing operations later on). So this is what I have done: vnd(4) will now use read and write for those files stored in file systems where bmap and strategy are not available and continue to use the latter two if they are present (as it has always done).

Some more information can be found in the CVS commit and its corresponding bug report.

November 9, 2006 · Tags: netbsd, tmpfs, vnd
Continue reading (about 2 minutes)

A couple of Ext2/Ext3 project proposals

I've just added a couple of project proposals related to improving Ext2/Ext3 file system support in the NetBSD Operating System. These are:

If you are interested in getting into file system development — a very interesting research area, believe me! ;-) — this is probably a safe bet. These two projects are not very complex but can quickly benefit NetBSD for different reasons (not only better Linux compatibility). Check out their descriptions for more details!

November 7, 2006 · Tags: netbsd
Continue reading (about 1 minute)

A letter to NetBSD

Dear NetBSD,

It is almost five years since we first met and I still remember how much I liked you at that time. Despite your 1.5 release had slow disk performance when compared to the other BSDs, I found in you an operating system that just felt right. You focused on clean and well designed code among many other goals; sincerely, I didn’t come to you looking for portability because I never had anything else than i386 machines. All these feelings turned into love after installing and experimenting with you: the system was minimal, well documented and made sense. As you know, I soon left FreeBSD and migrated my machines to you.

September 23, 2006 · Tags: gnome, netbsd, pkgsrc
Continue reading (about 5 minutes)

GNOME on NetBSD needs YOU!

A few pkgsrc developers and I have been working hard for years to bring the GNOME Desktop to this packaging system and make it work under NetBSD. We are quite happy with the current results because the packages are updated very frequently and everything works. Well, almost. There are still several missing details that really hurt the end user experience and need fixing.

If things continue as have gone until now, we will always be one step (or more!) behind other operating systems such as Linux and FreeBSD. Linux, of course, gets full support from the GNOME developers because they do their daily work on it. FreeBSD, on the other hand, has more people working on the port and therefore have more manpower to resolve all the portability problems; they are doing a nice work.

In our case, it is clear that we do not have enough manpower to keep up with the huge task that porting GNOME is — we are very few people working on it. Just consider that GNOME is composed of around 100 packages and that there are new major releases every 6 months (minor ones coming much faster). This updating task imposes a lot of stress on us that prevents us from working on the remaining pending items.

If we were able to work on all these issues, we could have a fully-functional GNOME Desktop on top of NetBSD. I belive this is a key area to improve NetBSD's visibility: if we had a complete desktop evnrionment, more users could come and use NetBSD for their daily tasks. Eventually, this could attract more developers who would start contributing to the system itself.

So... I've prepared a list of GNOME-related projects that details the major items that need to be addressed to have a complete GNOME Desktop installation on top of NetBSD and pkgsrc. I've tried to detail each project as much as possible, explaining the current problem, why solving it could benefit NetBSD and how to get started.

Should you need more information, I've also written some generic guidelines about GNOME packaging and porting. And, of course, you can contact me to get more details and take one of the projects! I'm willing to mentor you to make them success. You can certainly make a difference to the current status of things.

Let me add that I've have learned a lot about many different areas from my contributions to pkgsrc and NetBSD. You can seize the opportunity to learn new exciting stuff too; don't be shy! Oh... and if GNOME on NetBSD is not of your preference, please see our complete list of proposed projects; I bet you will find something of your interest!

September 1, 2006 · Tags: gnome, netbsd, pkgsrc
Continue reading (about 3 minutes)

Recent GNOME fixes

A week has almost passed since someone told me that D-Bus' session daemon was broken in NetBSD. I curse that day! ;-) I've been investigating that problem since then and (very) beleatedly fixing some issues in other GNOME programs during the process.

D-Bus' session daemon did not work under NetBSD because it couldn't authenticate incoming connections; that was due to the lack of socket credentials. After some of days of investigation — which included discovering that NetBSD does indeed support socket credentials through LOCAL_CREDS — and multiple attempts to implement them, I finally got D-Bus session daemon to authenticate appropriately.

This served me to fix gnome-keyring too, which was broken for the exact same reason, and gnome-keyring-manager, the application I was using to check whether gnome-keyring worked or not.

At last I also finally sat down and solved an annoying problem in the gnome-applets package that caused the Sticky Notes applet to crash when adding a new note; this had been happening since 2.12.0 if I recall correctly. I am sure that the root of this problem was also producing incorrect behavior in other panel applets.

For more details check these out:

dbus: #7798 - Generalize kqueue support
dbus: #8037 - Improve debugging messages in exchange_credentials
dbus: #8041 - Add LOCAL_CREDS socket credentials support
gnome-keyring: #353105 - Implement LOCAL_CREDS socket credentials
gnome-applets: #353239 - Get rid of AC_DEFINE_DIR
gnome-keyring-manager: #353251 - Better handling of null paths

Ouch... and GNOME 2.16 is around the corner... I'm afraid of all the new problems to come!

August 28, 2006 · Tags: gnome, netbsd
Continue reading (about 2 minutes)

More on LOCAL_CREDS

One of the problems of learning new stuff based on trial-and-error iterations is that it is very easy to miss important details... but that's the price to pay when there is no decent documentation available for a given feature. We saw yesterday multiple details about LOCAL_CREDS socket credentials and, as you may deduce, I missed some.

First of all I assumed that setting the LOCAL_CREDS option only affected the next received message (I didn't mention this explicitly in the post though). It turns out that this is incorrect: enabling this option makes the socket transmit credentials information on each message until the option is disabled again.

Secondly, setting the LOCAL_CREDS option on a server socket (one configured with the listen(2) call) results in all sockets created from it through accept(2) to also carry the flag enabled. In other words, it is inherited.

These features are interesting because, when using combined, avoid the need for the synchronization protocol outlined in the previous post — in some cases only. If the credentials are to be transmitted at the very beginning of the connection, the server can follow these steps:

Create the server socket and configure it with bind(2) and listen(2).
Before entering the accept(2) loop, set the LOCAL_CREDS option on the server socket.
Enter the accept(2) loop and start accepting clients.
For each new client:

Receive its first message.
Get the credentials from it.
Disable the LOCAL_CREDS option from the socket used to communicate with that specific client.

It couldn't be easier! This is still different to all other socket credentials methods I know of but can be easily adapted to protocols that were not designed to support LOCAL_CREDS (i.e. that do not implement the synchronization explained in the previous post).

August 28, 2006 · Tags: credentials, netbsd, sockets
Continue reading (about 2 minutes)

LOCAL_CREDS socket credentials

Socket credentials is a feature that allows a user process to receive the credentials (UID, GID, etc.) of the process at the other end of a communication socket in a safe way. The operating system is in charge of managing this information, sent separately from the data flow, so that the user processes cannot fake it. There are many different implementations of this concept out there as you can imagine.

For some reason I assumed for a long time that NetBSD didn't support any kind of socket credentials. However, I recently discovered that it indeed supports them through the LOCAL_CREDS socket option. Unfortunately it behaves quite differently from other methods. This poses some annoying portability problems in applications not designed in the first place to support it (e.g. D-Bus, the specific program I'm fighting right now).

LOCAL_CREDS works as follows:

The receiver interested in remote credentials uses setsockopt(2) to enable the LOCAL_CREDS option in the socket.
The sender sends a message through the channel either with write(2) or sendmsg(2). It needn't do anything special other than ensuring that the message is sent after the receiver has enabled the LOCAL_CREDS option.
The receiver gets the message using recvmsg(2) and parses the out of band data stored in the control buffer: a struct sockcred message that contains the remote credentials (UID, GID, etc.). This does not provide the PID of the remote process though, as other implementations do.

The tricky part here is to ensure that the sender writes the message after the receiver has enabled the LOCAL_CREDS option. If this is not guaranteed, a race condition appears and the behavior becomes random: some times the receiver will get socket credentials, some times it will not.

To ensure this restriction there needs to be some kind of synchronization protocol between the two peers. This is illustrated in the following example: it assumes a client/server model and a "go on" message used to synchronize. The server could do:

Wait for client connection.
Set LOCAL_CREDS option on remote socket.
Send a "go on" message to client.
Wait for a response, which carries the credentials.
Parse the credentials.

And the client could do:

Connect to server.
Wait until "go on" message.
Send any message to the server.

To conclude, a sample example program that shows how to manage the LOCAL_CREDS option. socketpair(2) is used for simplicity, but this can easily be extrapolated to two independent programs.

#include <sys/param.h>
#include <sys/types.h>
#include <sys/inttypes.h>
#include <sys/socket.h>
#include <sys/un.h>

#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int
main(void)
{
    int sv[2];
    int on = 1;
    ssize_t len;
    struct iovec iov;
    struct msghdr msg;
    struct {
        struct cmsghdr hdr;
        struct sockcred cred;
        gid_t groups[NGROUPS - 1];
    } cmsg;

    /*
     * Create a pair of interconnected sockets for simplicity:
     * sv[0] - Receive end (this program).
     * sv[1] - Write end (the remote program, theorically).
     */
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == -1)
        err(EXIT_FAILURE, "socketpair");

    /*
     * Enable the LOCAL_CREDS option on the reception socket.
     */
    if (setsockopt(sv[0], 0, LOCAL_CREDS, &on, sizeof(on)) == -1)
        err(EXIT_FAILURE, "setsockopt");

    /*
     * The remote application writes the message AFTER setsockopt
     * has been used by the receiver.  If you move this above the
     * setsockopt call, you will see how it does not work as
     * expected.
     */
    if (write(sv[1], &on, sizeof(on)) == -1)
        err(EXIT_FAILURE, "write");

    /*
     * Prepare space to receive the credentials message.
     */
    iov.iov_base = &on;
    iov.iov_len = 1;
    memset(&msg, 0, sizeof(msg));
    msg.msg_iov = &iov;
    msg.msg_iovlen = 1;
    msg.msg_control = &cmsg;
    msg.msg_controllen = sizeof(struct cmsghdr) +
        SOCKCREDSIZE(NGROUPS);
    memset(&cmsg, 0, sizeof(cmsg));

    /*
     * Receive the message.
     */
    len = recvmsg(sv[0], &msg, 0);
    if (len < 0)
        err(EXIT_FAILURE, "recvmsg");
    printf("Got %zu bytesn", len);

    /*
     * Print out credentials information, if received
     * appropriately.
     */
    if (cmsg.hdr.cmsg_type == SCM_CREDS) {
        printf("UID: %" PRIdMAX "n",
               (intmax_t)cmsg.cred.sc_uid);
        printf("EUID: %" PRIdMAX "n",
               (intmax_t)cmsg.cred.sc_euid);
        printf("GID: %" PRIdMAX "n",
               (intmax_t)cmsg.cred.sc_gid);
        printf("EGID: %" PRIdMAX "n",
               (intmax_t)cmsg.cred.sc_egid);
        if (cmsg.cred.sc_ngroups > 0) {
            int i;
            printf("Supplementary groups:");
            for (i = 0; i < cmsg.cred.sc_ngroups; i++)
                printf(" %" PRIdMAX,
                       (intmax_t)cmsg.cred.sc_groups[i]);
            printf("n");
        }
    } else
        errx(EXIT_FAILURE, "Message did not include credentials");

    close(sv[0]);
    close(sv[1]);

    return EXIT_SUCCESS;
}

August 27, 2006 · Tags: credentials, netbsd, sockets
Continue reading (about 4 minutes)

Making Packager-Friendly Software (part 2)

My previous article, Making Packager-Friendly Software (part 1), explains why software packaging is sometimes problematic due to real problems in the mainstream sources. It also discusses many issues that affect the distribution files and the configuration scripts (the most visible items when trying out a new program). This part explores the problems found in the build infrastructure and the code itself.

April 28, 2005 · Tags: featured, netbsd, onlamp, programming, unix
Continue reading (about 15 minutes)

Making Packager-Friendly Software (part 1)

A package maintainer, or packager, is a person who creates packages for software projects. He eventually finds common problems in these projects, resulting in a complex packaging process and a final package that is a nightmare to maintain. These little flaws exist because in most cases the original developers are not packagers, so they are not aware of them. In other words, if you do not know something is wrong, you cannot fix it.

March 31, 2005 · Tags: featured, netbsd, onlamp, programming, unix
Continue reading (about 20 minutes)

Posts: Netbsd

My setup

Getting started

Setting up a daily cron job

Concluding trivia

Featured software

Featured posts