Moving projects to GitHub
For a couple of years or so, I have been hosting my open source projects in Google Code. The time to jump ship has come. The major reason for this move is that Google Code stopped supporting file downloads three weeks ago. This is unfortunate given that "binary" releases are a must for proper software distribution. Sure, I could use a third-party service like Bintray to offer the downloads, but I'd rather consolidate all project data in a single location.
February 6, 2014
·
Tags:
<a href="/tags/github">github</a>, <a href="/tags/oss">oss</a>, <a href="/tags/sysbuild">sysbuild</a>, <a href="/tags/sysupgrade">sysupgrade</a>
Continue reading (about
2 minutes)
'Hello GitHub!' say shtk, sysbuild and sysupgrade
Over a year ago, I developed two command line utilities for NetBSD (sysbuild and sysupgrade) and a supporting library for the two tools (shtk). These were all introduced in their corresponding blog posts —Introducing sysbuild for NetBSD, Introducing sysupgrade for NetBSD and Introducing shtk— and since then I have heard good comments about them. About a couple of weeks ago, I started working on the much-needed rewrite of pkg_comp and a supporting standalone tool. I was using the same development methodology as with the other three projects: putting all the code in pkgsrc and implementing the build system from the package's Makefile. Along the way, this became increasingly annoying to the point where I could not stand it any more. pkgsrc is a packaging system, not a development platform. Developing a project within it is difficult due to the indirections between the outer Makefile and the work directory, and the expected workflow of working with packages. A separate issue is that having the source code in pkgsrc prevents the distribution of the packages as standalone tools for third-party operating systems. (Mind you: I've recently started building NetBSD from Linux and FreeBSD and missed sysbuild dearly.) It's hard and ugly (but not impossible) to generate tarballs for the sources in pkgsrc that can later be hosted elsewhere... and, even if doing that, using pkgsrc as the master tree for the code would seem backwards.
July 28, 2013
·
Tags:
<a href="/tags/github">github</a>, <a href="/tags/release">release</a>, <a href="/tags/shtk">shtk</a>, <a href="/tags/sysbuild">sysbuild</a>, <a href="/tags/sysupgrade">sysupgrade</a>
Continue reading (about
3 minutes)
Introducing shtk
Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so. That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short. Yeah, this name sounds very pretentious but, really, I don't intend this to be anything big. The only thing I want to do is simplify my life when implementing shell scripts, and hope that other people might find the modules useful. So far, I have taken the generic (and common!) code from sysbuild and sysupgrade, reconciled a few tiny divergences, and moved it into this new shtk package. In reality, writing something like shtk is sin-borderline. I really should not be using shell scripting for the kind of tools I am implementing (they deserve better data structures and better error checking than what shell provides, for example). However, shell scripting is incredible convenient to get reasonably-good implementations of such tools with minimal effort, and is the only scripting language available in NetBSD's base system. (Yes, yes, there is Lua, but my limited knowledge of Lua would not let me write these tools in any decent manner nor in any reasonable time.) So, what's in shtk? There are a few functions to deal with command lines (error/warning reporting and such things), some trivial stuff to deal with lists, a bunch of code to interact with cvs and, what I like the most, a module to implement configuration files with some kind of key/value validation. At the moment, shtk can only be found in pkgsrc under pkgsrc/devel/shtk and I don't currently have any plans to make it more widely available. If there are enough people interested in that with real needs, I could reconsider, but the maintenance effort would be non-trivial. To showcase the features of shtk, I have updated the sysbuild and sysupgrade packages to depend on this new toolkit while at the same time dropping all this duplicate supporting code. It's a good thing that I wrote exhaustive tests for all possible code paths, because the migration from the built-in modules to shtk was riddled with subtleties that would have impacted end users otherwise. Now... time to really consider taking the task of rewriting pkg_comp in a more maintainable style so that I can add the features I have wished for for many years (like OS X support).
August 15, 2012
·
Tags:
<a href="/tags/announce">announce</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/shell">shell</a>, <a href="/tags/shtk">shtk</a>, <a href="/tags/sysbuild">sysbuild</a>, <a href="/tags/sysupgrade">sysupgrade</a>
Continue reading (about
3 minutes)
Introducing sysupgrade for NetBSD
Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version? Upgrading NetBSD to a newer version from distribution sets generally looks like the following; Fetch new distribution sets (or roll your own).Upgrade the kernel.Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.tgz along the way.Use etcupdate to merge new changes to configuration files.Use postinstall to validate the upgraded system.Simple? Yes. Easy? No. The above procedure is obscure to anyone new to NetBSD. (Actually, if you tell anybody that the way to upgrade your machine is by unpacking tarballs over / will stare at you thinking you are kidding. It's 2012.) "Jokes" aside, what is worse is that the procedure is quite monotonous and, therefore, it is very easy for the administrator to make a trivial mistake along the way and screw up a running system. (Been there, done that... multiple times.) Machines are made to automate trivial and repetitive tasks like the above, and they are actually very good at that. Over the years, I have performed NetBSD updates manually and later written crappy, unreliable scripts to do the upgrades for me. These scripts have never been reusable and they haven't dealt with error conditions gracefully. Furthermore, because these scripts live in my home directory, I have to remember to carry them around every time I set up a new NetBSD box. It was about time I sat down and rewrote my custom scripts into something more "decent". Something with documentation, with a configuration file, and with tests. So today, and for all the reason above, I am introducing sysupgrade. sysupgrade is a script that automates (trivializes) the whole process of upgrading an existing NetBSD installation to a newer release, be it the currently-running system or a non-live system. sysupgrade does so by following the steps outlined above and is coordinated by a configuration file. You can find the tool in pkgsrc/sysutils/sysupgrade, next to sysbuild. The bundled sysupgrade(8) and sysupgrade.conf(5) manual pages, and the default sysupgrade.conf configuration file should get you started and hopefully answer most of your questions. For the impatient, the following command would upgrade your machine to the specified version target: $ sysupgrade auto ftp://ftp.NetBSD.org/pub/NetBSD/NetBSD-<X.Y.Z>/$(uname -m) At the moment, please consider sysupgrade to be experimental. It works well for me on my various machines (running both NetBSD 6.0 BETA and -current), and I have been using the upgrade procedure outlined above for years without issues. However, as with any shiny-new software, be careful. If you use NetBSD on a virtual machine, take a snapshot before running this tool; I don't think your machine is going to blow up, but better safe than sorry! Enjoy, and feedback very welcome! PS: There is lots of room for improvement. The TODO file in the package directory includes some of the ideas I'd like to work on later.
August 6, 2012
·
Tags:
<a href="/tags/announce">announce</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/sysbuild">sysbuild</a>, <a href="/tags/sysupgrade">sysupgrade</a>
Continue reading (about
3 minutes)
Introducing sysbuild for NetBSD
NetBSD's build system is close to awesome: after checking a source tree out from CVS on virtually any Unix-like operating sytem, building a full NetBSD release for any of the supported platforms is as simple as running the build.sh script with the right arguments. There are, however, a few things that would deserve automation in this process, but that are not in build.sh's domain to solve. These are: Fetching and keeping the source tree up to date: interacting with CVS is still the responsibility of the user.Simplifying the call to build.sh: this script takes a ton of arguments, and running it by hand quickly becomes "annoying". It would be nice if all the desired values to its arguments where stored elsewhere and picked up automatically. (mk.conf may not always be the right choice.)Plugging into your crontab(5) for easy periodic rebuilds of NetBSD, with proper reporting of failures. Upgrading a live system from source is a major supported mechanism (particularly for NetBSD-current), so having a way to keep fresh release files around is welcome.Performing such daily rebuilds of NetBSD as a dedicated unprivileged user.To anybody familiar with NetBSD, it is obvious that all the above items are "simple enough" to solve one by one; however, doing them by hand gets old very quickly — machines exist to make our life easier, don't they? I know I am not the only one that has custom local wrapper scripts around build.sh, and I also know that my scripts were ad-hoc, ugly and non-reusable. (Also, of course, anybody new to NetBSD will certainly not find any of the above trivial, but that's secondary.) To this end, I have written a script that automates all the aforementioned steps in a single tool, driven by configuration files that specify what to do. Enter sysbuild. My setupThese days, my main NetBSD development box is a virtual machine. For a couple of years, I have been carrying around this virtual machine across machines and slowly tuning its configuration to suit my needs. The way I work is the following: I keep a read-write copy of the sources under my home directory, which is the one I use for my development work. This copy is accompanied by a script in my home directory that, given a machine type, updates the sources and rebuilds NetBSD for that machine type, using a known directory layout within my home directory. I also keep a read-only copy of the sources under /usr/src, checked out from anoncvs. This copy is used by the dedicated "builder" system user to perform dedicated rebuilds of NetBSD. The "builder" user also has a custom script (but different than the other one!) that updates /usr/src, rebuilds NetBSD for the machine types I am interested in, records full logs to files and places the build results into a shared network drive. Lastly, the dedicated "builder" user has a cron job that runs the previous script overnight. The purpose of this dual setup is to always have a fresh release build somewhere in the system, built from pristine sources, while at the same time allowing me to do my development work in a tree that I could break at any time. Unfortunately, the size of the virtual disk of my development box has become too small for my current needs, and in order to fix this I prefer to start afresh. The thought of having to set up this whole scheme all over again, by hand, is what triggered the creation of sysbuild. Getting startedsysbuild lives in pkgsrc under the pkgsrc/sysutils/sysbuild directory. This package provides the main script, tests, sample configuration files and the extensive manual page. Installing the script is as easy as installing any other package, and I would recommend reading its documentation now. Once installed, sysbuild should be ready to use out of the box assuming that you have write access to /usr/src. Simply typing sysbuild build from anywhere in the system will cause the source tree to be updated and a release for your current platform to be built. If you wish to customize the behavior of sysbuild, copy /usr/pkg/etc/sysbuild/default.conf to ~/.sysbuild/default.conf and edit the latter. The tool will pick it up on the next execution and use your new configuration settings. You can, of course, manage a variety of configurations in case you want to do different builds (say -current and 6.x). Setting up a daily cron jobThe pkgsrc/sysutils/sysbuild-user package is a convenience bundle that manages a "sysbuild" unprivileged user in the system and installs a sample crontab to perform the desired periodic rebuilds of NetBSD. This crontab uses the sysbuild4cron script, which is a very simple utility to run sysbuild, store its output in a file, and send an error report any time the execution fails. Simply installing the package will cause the creation of the "sysbuild" user and the configuration of this sample cron job. Follow the instructions provided by the package in its MESSAGE file to tune the behavior of any of these. Concluding triviaYes, these scripts are extremely tied to my particular workflow and use cases, although I don't think my needs are that special. However, I have attempted to come up with a generic-enough script and configuration file to allow the addition of new features. You may notice that sysbuild's version is 2.0 and not 1.0. While preparing the script for addition to pkgsrc, cvs add barfed that I was re-adding previously-deleted files. Uh hum, what a surprise! Upon reading the CVS log of the package's Makefile, I found that 10 years ago I already wrote this exact same script and that two years after it got deleted because it stopped working. I had completely forgotten about this! However, seeing pkg_comp's messy code (which was written around the same time), I imagine that the previous implementation of this idea was messy; I don't dare to look at the previous code. Enjoy!
July 25, 2012
·
Tags:
<a href="/tags/announce">announce</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/sysbuild">sysbuild</a>
Continue reading (about
5 minutes)
Featured software
Featured posts
- Fast machines, slow machines
- EndBASIC 0.10: Core language, evolved
- Farewell, Microsoft; hello, Snowflake!
- Rust is hard, yes, but does it matter?
- Rust traits and dependency injection
- A year on Windows: Introduction
- Always be quitting
- How does Google keep build times low?
- How does Google avoid clean builds?
- Unit-testing a console app (a text editor)
- More...
