Sysupgrade - Julio Merino (jmmv.dev)

rc.d belongs in libexec, not etc

The scripts that live under /etc/rc.d/ in FreeBSD, NetBSD, and OpenBSD are in the wrong place. They all should live in /libexec/rc.d/ because they are code, not configuration. Let’s look at the history of these systems to see how we got here, why this is problematic, and how things would look like in a better world.

August 24, 2020 · Tags: freebsd, netbsd, opinion, sysupgrade
Continue reading (about 8 minutes)

Moving projects to GitHub

For a couple of years or so, I have been hosting my open source projects in Google Code. The time to jump ship has come.

The major reason for this move is that Google Code stopped supporting file downloads three weeks ago. This is unfortunate given that "binary" releases are a must for proper software distribution. Sure, I could use a third-party service like Bintray to offer the downloads, but I'd rather consolidate all project data in a single location.

February 6, 2014 · Tags: github, oss, sysbuild, sysupgrade
Continue reading (about 2 minutes)

'Hello GitHub!' say shtk, sysbuild and sysupgrade

Over a year ago, I developed two command line utilities for NetBSD (sysbuild and sysupgrade) and a supporting library for the two tools (shtk). These were all introduced in their corresponding blog posts —Introducing sysbuild for NetBSD, Introducing sysupgrade for NetBSD and Introducing shtk— and since then I have heard good comments about them.

About a couple of weeks ago, I started working on the much-needed rewrite of pkg_comp and a supporting standalone tool. I was using the same development methodology as with the other three projects: putting all the code in pkgsrc and implementing the build system from the package's Makefile. Along the way, this became increasingly annoying to the point where I could not stand it any more. pkgsrc is a packaging system, not a development platform. Developing a project within it is difficult due to the indirections between the outer Makefile and the work directory, and the expected workflow of working with packages.

A separate issue is that having the source code in pkgsrc prevents the distribution of the packages as standalone tools for third-party operating systems. (Mind you: I've recently started building NetBSD from Linux and FreeBSD and missed sysbuild dearly.) It's hard and ugly (but not impossible) to generate tarballs for the sources in pkgsrc that can later be hosted elsewhere... and, even if doing that, using pkgsrc as the master tree for the code would seem backwards.

July 28, 2013 · Tags: github, release, shtk, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Introducing shtk

Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so.

That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short. Yeah, this name sounds very pretentious but, really, I don't intend this to be anything big. The only thing I want to do is simplify my life when implementing shell scripts, and hope that other people might find the modules useful. So far, I have taken the generic (and common!) code from sysbuild and sysupgrade, reconciled a few tiny divergences, and moved it into this new shtk package.

In reality, writing something like shtk is sin-borderline. I really should not be using shell scripting for the kind of tools I am implementing (they deserve better data structures and better error checking than what shell provides, for example). However, shell scripting is incredible convenient to get reasonably-good implementations of such tools with minimal effort, and is the only scripting language available in NetBSD's base system. (Yes, yes, there is Lua, but my limited knowledge of Lua would not let me write these tools in any decent manner nor in any reasonable time.)

So, what's in shtk? There are a few functions to deal with command lines (error/warning reporting and such things), some trivial stuff to deal with lists, a bunch of code to interact with cvs and, what I like the most, a module to implement configuration files with some kind of key/value validation.

At the moment, shtk can only be found in pkgsrc under pkgsrc/devel/shtk and I don't currently have any plans to make it more widely available. If there are enough people interested in that with real needs, I could reconsider, but the maintenance effort would be non-trivial.

To showcase the features of shtk, I have updated the sysbuild and sysupgrade packages to depend on this new toolkit while at the same time dropping all this duplicate supporting code. It's a good thing that I wrote exhaustive tests for all possible code paths, because the migration from the built-in modules to shtk was riddled with subtleties that would have impacted end users otherwise.

Now... time to really consider taking the task of rewriting pkg_comp in a more maintainable style so that I can add the features I have wished for for many years (like OS X support).

August 15, 2012 · Tags: announce, netbsd, shell, shtk, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Introducing sysupgrade for NetBSD

Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version?

Upgrading NetBSD to a newer version from distribution sets generally looks like the following;

Fetch new distribution sets (or roll your own).
Upgrade the kernel.
Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.tgz along the way.
Use etcupdate to merge new changes to configuration files.
Use postinstall to validate the upgraded system.

Simple? Yes. Easy? No. The above procedure is obscure to anyone new to NetBSD. (Actually, if you tell anybody that the way to upgrade your machine is by unpacking tarballs over / will stare at you thinking you are kidding. It's 2012.) "Jokes" aside, what is worse is that the procedure is quite monotonous and, therefore, it is very easy for the administrator to make a trivial mistake along the way and screw up a running system. (Been there, done that... multiple times.)

Machines are made to automate trivial and repetitive tasks like the above, and they are actually very good at that. Over the years, I have performed NetBSD updates manually and later written crappy, unreliable scripts to do the upgrades for me. These scripts have never been reusable and they haven't dealt with error conditions gracefully. Furthermore, because these scripts live in my home directory, I have to remember to carry them around every time I set up a new NetBSD box.

It was about time I sat down and rewrote my custom scripts into something more "decent". Something with documentation, with a configuration file, and with tests.

So today, and for all the reason above, I am introducing sysupgrade.

sysupgrade is a script that automates (trivializes) the whole process of upgrading an existing NetBSD installation to a newer release, be it the currently-running system or a non-live system. sysupgrade does so by following the steps outlined above and is coordinated by a configuration file. You can find the tool in pkgsrc/sysutils/sysupgrade, next to sysbuild. The bundled sysupgrade(8) and sysupgrade.conf(5) manual pages, and the default sysupgrade.conf configuration file should get you started and hopefully answer most of your questions.

For the impatient, the following command would upgrade your machine to the specified version target:

$ sysupgrade auto
ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-<X.Y.Z>/$(uname -m)

At the moment, please consider sysupgrade to be experimental. It works well for me on my various machines (running both NetBSD 6.0 BETA and -current), and I have been using the upgrade procedure outlined above for years without issues. However, as with any shiny-new software, be careful. If you use NetBSD on a virtual machine, take a snapshot before running this tool; I don't think your machine is going to blow up, but better safe than sorry!

Enjoy, and feedback very welcome!

PS: There is lots of room for improvement. The TODO file in the package directory includes some of the ideas I'd like to work on later.

August 6, 2012 · Tags: announce, netbsd, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Posts: Sysupgrade

rc.d belongs in libexec, not etc

Moving projects to GitHub

'Hello GitHub!' say shtk, sysbuild and sysupgrade

Introducing shtk

Introducing sysupgrade for NetBSD

Featured software

Featured posts