Announce - Julio Merino (jmmv.dev)

Kyua turns parallel

After three months of intensive work on Kyua's executor Git branch, I am happy to announce that the new execution engine, whose crown feature is the ability to run test cases in parallel, has just landed in master and passes all self-tests!

You can head over to the commit message for more details on the merge, read the NEWS entries, and skim throught the history of the executor branch to understand how this feature has been built.

February 28, 2015 · Tags: announce, kyua
Continue reading (about 5 minutes)

shtk 1.6 is now available

The Shell Toolkit, or shtk for short, is a little project I introduced back in August of 2008 to support other tools such as sysbuild and sysupgrade. Since that time, the project has seen little activity because it did not have much to offer and because shtk's public interface was not documented (hence making it impossible for developers to get started with shtk).

Well, both are changing today with the brand-new release of shtk 1.6:

November 17, 2014 · Tags: announce, release, shtk
Continue reading (about 2 minutes)

Joining the FreeBSD committer ranks

About 11 years ago, I had the honor of becoming part of the NetBSD developer team. A week ago, the same offer was extended to me for FreeBSD and I could not refuse it... so, as of two days ago, I am jmmv at FreeBSD and will be working in src!

But how have I ended up here? The story goes back in time...

My beginnings with FreeBSD

FreeBSD was the first BSD system I tried after spending a couple of years with Linux. I was able to do so because one of the PC magazines that my father used to buy in 1999 came with a copy of FreeBSD 3.2 — pretty advanced stuff for a magazine compared to today's standards, huh? However, that first experience did not go very well: the system felt really sluggish... only because I did not know that I had to enable the magic softdep option. So I gave up.

November 4, 2013 · Tags: announce, freebsd
Continue reading (about 5 minutes)

Introducing shtk

Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so.

That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short. Yeah, this name sounds very pretentious but, really, I don't intend this to be anything big. The only thing I want to do is simplify my life when implementing shell scripts, and hope that other people might find the modules useful. So far, I have taken the generic (and common!) code from sysbuild and sysupgrade, reconciled a few tiny divergences, and moved it into this new shtk package.

In reality, writing something like shtk is sin-borderline. I really should not be using shell scripting for the kind of tools I am implementing (they deserve better data structures and better error checking than what shell provides, for example). However, shell scripting is incredible convenient to get reasonably-good implementations of such tools with minimal effort, and is the only scripting language available in NetBSD's base system. (Yes, yes, there is Lua, but my limited knowledge of Lua would not let me write these tools in any decent manner nor in any reasonable time.)

So, what's in shtk? There are a few functions to deal with command lines (error/warning reporting and such things), some trivial stuff to deal with lists, a bunch of code to interact with cvs and, what I like the most, a module to implement configuration files with some kind of key/value validation.

At the moment, shtk can only be found in pkgsrc under pkgsrc/devel/shtk and I don't currently have any plans to make it more widely available. If there are enough people interested in that with real needs, I could reconsider, but the maintenance effort would be non-trivial.

To showcase the features of shtk, I have updated the sysbuild and sysupgrade packages to depend on this new toolkit while at the same time dropping all this duplicate supporting code. It's a good thing that I wrote exhaustive tests for all possible code paths, because the migration from the built-in modules to shtk was riddled with subtleties that would have impacted end users otherwise.

Now... time to really consider taking the task of rewriting pkg_comp in a more maintainable style so that I can add the features I have wished for for many years (like OS X support).

August 15, 2012 · Tags: announce, netbsd, shell, shtk, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Introducing sysupgrade for NetBSD

Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version?

Upgrading NetBSD to a newer version from distribution sets generally looks like the following;

Fetch new distribution sets (or roll your own).
Upgrade the kernel.
Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.tgz along the way.
Use etcupdate to merge new changes to configuration files.
Use postinstall to validate the upgraded system.

Simple? Yes. Easy? No. The above procedure is obscure to anyone new to NetBSD. (Actually, if you tell anybody that the way to upgrade your machine is by unpacking tarballs over / will stare at you thinking you are kidding. It's 2012.) "Jokes" aside, what is worse is that the procedure is quite monotonous and, therefore, it is very easy for the administrator to make a trivial mistake along the way and screw up a running system. (Been there, done that... multiple times.)

Machines are made to automate trivial and repetitive tasks like the above, and they are actually very good at that. Over the years, I have performed NetBSD updates manually and later written crappy, unreliable scripts to do the upgrades for me. These scripts have never been reusable and they haven't dealt with error conditions gracefully. Furthermore, because these scripts live in my home directory, I have to remember to carry them around every time I set up a new NetBSD box.

It was about time I sat down and rewrote my custom scripts into something more "decent". Something with documentation, with a configuration file, and with tests.

So today, and for all the reason above, I am introducing sysupgrade.

sysupgrade is a script that automates (trivializes) the whole process of upgrading an existing NetBSD installation to a newer release, be it the currently-running system or a non-live system. sysupgrade does so by following the steps outlined above and is coordinated by a configuration file. You can find the tool in pkgsrc/sysutils/sysupgrade, next to sysbuild. The bundled sysupgrade(8) and sysupgrade.conf(5) manual pages, and the default sysupgrade.conf configuration file should get you started and hopefully answer most of your questions.

For the impatient, the following command would upgrade your machine to the specified version target:

$ sysupgrade auto
ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-<X.Y.Z>/$(uname -m)

At the moment, please consider sysupgrade to be experimental. It works well for me on my various machines (running both NetBSD 6.0 BETA and -current), and I have been using the upgrade procedure outlined above for years without issues. However, as with any shiny-new software, be careful. If you use NetBSD on a virtual machine, take a snapshot before running this tool; I don't think your machine is going to blow up, but better safe than sorry!

Enjoy, and feedback very welcome!

PS: There is lots of room for improvement. The TODO file in the package directory includes some of the ideas I'd like to work on later.

August 6, 2012 · Tags: announce, netbsd, sysbuild, sysupgrade
Continue reading (about 3 minutes)

Introducing sysbuild for NetBSD

NetBSD's build system is close to awesome: after checking a source tree out from CVS on virtually any Unix-like operating sytem, building a full NetBSD release for any of the supported platforms is as simple as running the build.sh script with the right arguments.

There are, however, a few things that would deserve automation in this process, but that are not in build.sh's domain to solve. These are:

Fetching and keeping the source tree up to date: interacting with CVS is still the responsibility of the user.
Simplifying the call to build.sh: this script takes a ton of arguments, and running it by hand quickly becomes "annoying". It would be nice if all the desired values to its arguments where stored elsewhere and picked up automatically. (mk.conf may not always be the right choice.)
Plugging into your crontab(5) for easy periodic rebuilds of NetBSD, with proper reporting of failures. Upgrading a live system from source is a major supported mechanism (particularly for NetBSD-current), so having a way to keep fresh release files around is welcome.
Performing such daily rebuilds of NetBSD as a dedicated unprivileged user.

To anybody familiar with NetBSD, it is obvious that all the above items are "simple enough" to solve one by one; however, doing them by hand gets old very quickly — machines exist to make our life easier, don't they? I know I am not the only one that has custom local wrapper scripts around build.sh, and I also know that my scripts were ad-hoc, ugly and non-reusable. (Also, of course, anybody new to NetBSD will certainly not find any of the above trivial, but that's secondary.)

To this end, I have written a script that automates all the aforementioned steps in a single tool, driven by configuration files that specify what to do. Enter sysbuild.

My setup

These days, my main NetBSD development box is a virtual machine. For a couple of years, I have been carrying around this virtual machine across machines and slowly tuning its configuration to suit my needs. The way I work is the following:

I keep a read-write copy of the sources under my home directory, which is the one I use for my development work. This copy is accompanied by a script in my home directory that, given a machine type, updates the sources and rebuilds NetBSD for that machine type, using a known directory layout within my home directory.

I also keep a read-only copy of the sources under /usr/src, checked out from anoncvs. This copy is used by the dedicated "builder" system user to perform dedicated rebuilds of NetBSD. The "builder" user also has a custom script (but different than the other one!) that updates /usr/src, rebuilds NetBSD for the machine types I am interested in, records full logs to files and places the build results into a shared network drive. Lastly, the dedicated "builder" user has a cron job that runs the previous script overnight.

The purpose of this dual setup is to always have a fresh release build somewhere in the system, built from pristine sources, while at the same time allowing me to do my development work in a tree that I could break at any time.

Unfortunately, the size of the virtual disk of my development box has become too small for my current needs, and in order to fix this I prefer to start afresh. The thought of having to set up this whole scheme all over again, by hand, is what triggered the creation of sysbuild.

Getting started

sysbuild lives in pkgsrc under the pkgsrc/sysutils/sysbuild directory. This package provides the main script, tests, sample configuration files and the extensive manual page. Installing the script is as easy as installing any other package, and I would recommend reading its documentation now.

Once installed, sysbuild should be ready to use out of the box assuming that you have write access to /usr/src. Simply typing sysbuild build from anywhere in the system will cause the source tree to be updated and a release for your current platform to be built.

If you wish to customize the behavior of sysbuild, copy /usr/pkg/etc/sysbuild/default.conf to ~/.sysbuild/default.conf and edit the latter. The tool will pick it up on the next execution and use your new configuration settings. You can, of course, manage a variety of configurations in case you want to do different builds (say -current and 6.x).

Setting up a daily cron job

The pkgsrc/sysutils/sysbuild-user package is a convenience bundle that manages a "sysbuild" unprivileged user in the system and installs a sample crontab to perform the desired periodic rebuilds of NetBSD. This crontab uses the sysbuild4cron script, which is a very simple utility to run sysbuild, store its output in a file, and send an error report any time the execution fails.

Simply installing the package will cause the creation of the "sysbuild" user and the configuration of this sample cron job. Follow the instructions provided by the package in its MESSAGE file to tune the behavior of any of these.

Concluding trivia

Yes, these scripts are extremely tied to my particular workflow and use cases, although I don't think my needs are that special. However, I have attempted to come up with a generic-enough script and configuration file to allow the addition of new features.

You may notice that sysbuild's version is 2.0 and not 1.0. While preparing the script for addition to pkgsrc, cvs add barfed that I was re-adding previously-deleted files. Uh hum, what a surprise! Upon reading the CVS log of the package's Makefile, I found that 10 years ago I already wrote this exact same script and that two years after it got deleted because it stopped working. I had completely forgotten about this! However, seeing pkg_comp's messy code (which was written around the same time), I imagine that the previous implementation of this idea was messy; I don't dare to look at the previous code.

Enjoy!

July 25, 2012 · Tags: announce, netbsd, sysbuild
Continue reading (about 5 minutes)

Introducing Lutok: A lightweight C++ API for Lua

It has finally happened. Lutok is the result of what was promised in the "Splitting utils::lua from Kyua" web post.

Quoting the project web page:

Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a small subset of miscellaneous utility functions built on top of the wrappers.

Lutok focuses on providing a clean and safe C++ interface; the drawback is that it is not suitable for performance-critical environments. In order to implement error-safe C++ wrappers on top of a Lua C binary library, Lutok adds several layers or abstraction and error checking that go against the original spirit of the Lua C API and thus degrade performance.

Lutok was originally developed within Kyua but was later split into its own project to make it available to general developers.

Coming up with a name for this project was quite an odyssey, and is what has delayed is release more than I wanted. My original candidate was "luawrap" which, although not very original, was to-the-point and easy to understand. Unfortunately, that name did not clear with the legal department and I had to propose several other names, some of which were not acceptable either. Eventually, I settled with "Lutok", which comes from "LUa TOolKit".

At this point, the source tree of Lutok provides pretty much the same code as the utils::lua module of Kyua. While it may be enough to get you started, I'm pretty sure you will lack some functions in the state class. If that is the case, don't hesitate to file a bug report to let me know what is missing.

In case you missed the link above, the project page is here: Lutok in Google Code.

September 15, 2011 · Tags: announce, cxx, kyua, lua, lutok
Continue reading (about 2 minutes)