Showing 65 posts
In the recent Remembering Buildtool post, I described how setting up a cache of configuration checks was an important step in Buildtool’s installation process. The goal was to avoid pointless repetitive work on every build by performing such common checks once. Episode 457 of BSD Now featured my post and Allan Jude wondered how much time would be saved in a bulk build of all FreeBSD packages if we could just do that same kind of caching with GNU Autoconf. And, you know what? It is indeed possible to do so. I had mentioned it en passing in my post but I guess I wasn’t clear enough, so let’s elaborate!
The pkgsrc package database, which by default lives in /var/db/pkg/, should not be there. Instead, it should be under /usr/pkg/libdata/pkgdb/. The same applies to FreeBSD’s and OpenBSD’s ports and also Debian’s dpkg, but I’ll focus on pkgsrc because it’s the system I know best. Let’s see why the current default is suboptimal and why libdata is a good alternative.
The scripts that live under /etc/rc.d/ in FreeBSD, NetBSD, and OpenBSD are in the wrong place. They all should live in /libexec/rc.d/ because they are code, not configuration. Let’s look at the history of these systems to see how we got here, why this is problematic, and how things would look like in a better world.
In the previous post, we saw how .d directories permit programmatic edits to system-wide configuration with ease. But this same concept can be applied to other kinds of tracking. Let’s dive into a few examples ranging from desktop menu entries to the package manager’s database itself.
Have you ever wondered why an increasing number of programs are configured by placing small files in .d directories instead of by just editing a single file? Have you ever wondered why these .d directories seem to proliferate in Linux installations? Read on to understand what these are and why they are useful.
This is a tutorial to guide you through the shiny new pkg_comp 2.0 on NetBSD. Goals: to use pkg_comp 2.0 to build a binary repository of all the packages you are interested in; to keep the repository fresh on a daily basis; and to use that repository with pkgin to maintain your NetBSD system up-to-date and secure.
The FreeBSD devsummit that just passed by gave me enough insight into Jenkins to question the long-term plans for Kyua. Uh, WHAT?! Let me explain. In the beginning... One of the original but unstated goals of Kyua was to fix the "mess" that is the NetBSD releng test run logs site: if you pay close attention, you will notice that various individuals have reinvented the wheel over and over again in an attempt to automate release builds and test suite runs.
BSDCan 2014 and the accompanying FreeBSD devsummit are officially over. Let's recap. FreeBSD devsummit The FreeBSD devsumit at BSDCan is, by far, the largest of them all. It is true that I already visited a devsummit once —the one in EuroBSDCon 2013—, but this is the first time I participate in the "real deal" while also being a committer. The first impressive thing about this devsummit is that there were about 120 attendees.
The time to kill the deprecated tools —atf-report and atf-run principally— from the upstream ATF distribution file has come. Unfortunately, this is not the trivial task that it may seem. But wait, "Why?" and "Why now?" Because NetBSD still relies on the deprecated tools to run its test suite, they cannot just be killed. Removing them from the upstream distribution, however, is actually a good change for both ATF and NetBSD.
EuroBSDCon 2013 is done. If you have been following my daily posts over the last 4 days (day 1, day 2, day 3 and day 4) as well as #EuroBSDCon updates in Twitter, you may already have a pretty good idea of what went on here. However, with the conference over, it is now a good time to recap the whole event and present the takeaways of these four days which, overall, were quite interesting and productive.
Live from Malta today attending the EuroBSDCon 2013 conference. Today is the first day of the conference itself. Many more people have shown up as expected and there have been tons of very interesting talks all the time. It is both good and bad that there are several tracks: you can select the topic you are most interested in, but sometimes great talks overlap! Keynote Today's opening session was led by Theo de Raadt, the founder of OpenBSD.
Hello everyone! Live from Malta today attending the EuroBSDCon 2013 conference. Today is the first day out of four: two days of tutorials and two days of actual conference. The tutorials are overlapped by two days of the usual FreeBSD Developer Summit (devsummit for short) and one day of the infrequent NetBSD Developer Summit. The ambient here is pretty good already: lots of enthusiastic people catching up since the last time they met each other and, more importantly, discussing ongoing developments.
The decision to not renew my NetBSD board membership was bittersweet. Let me put aside the Readability series posts for a moment while I recap how the two years serving the NetBSD Board of Directors have been. My term just finished a couple of weeks ago, so it is better to post this while it is still relevant. First, let me backtrack a little bit. A couple of years ago, I was nominated to serve the NetBSD Board of Directors.
I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below. I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it.
For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends. One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story:
Have you ever wanted to have a collection of ready-to-use modules for shell scripts? I have, particularly because I keep reimplementing the same functions over and over and over and over again whenever I write non-trivial shell scripts, and I'm tired of doing so. That's why I have just abstracted all the common code in the aforementioned tools and put it into a new package called the "Shell Toolkit", or shtk for short.
Over the last two weeks, you might have had fun rolling your own NetBSD binary releases with sysbuild. But what fun is that if you have no trivial way of upgrading your existing NetBSD installation to a newer version? Upgrading NetBSD to a newer version from distribution sets generally looks like the following; Fetch new distribution sets (or roll your own).Upgrade the kernel.Unpack the distribution sets over the root directory, without fat-fingering the command and unpacking etc.
NetBSD's build system is close to awesome: after checking a source tree out from CVS on virtually any Unix-like operating sytem, building a full NetBSD release for any of the supported platforms is as simple as running the build.sh script with the right arguments. There are, however, a few things that would deserve automation in this process, but that are not in build.sh's domain to solve. These are: Fetching and keeping the source tree up to date: interacting with CVS is still the responsibility of the user.
When I joined the NetBSD Board of Directors, I was trusted with access to private information and one of the prerequisites for downloading such data was to be able to store it safely: i.e. inside an encrypted volume. I initially went the easy route by putting the data in my laptop, which already uses FileVault 2 to encrypt the whole disk. But soon after, I decided that it would be nicer to put this data in my NetBSD home server, which is the machine that is perpetually connected to IRC and thus the one I use to log into the weekly meetings.
Dear users of NetBSD, I am pleased to announce that we (well, the release engineering team!) have just tagged the netbsd-6 branch in the CVS repository and thus opened the gate for testing of NetBSD 6.0_BETA. New binary snapshots should start appearing in the daily FTP archive soon. You can, of course, perform a cvs update -r netbsd-6 on your existing source tree and roll your own binaries (as I'm already doing on my home server).
Dear readers, It is a great pleasure for me to announce that I have just joined the Board of Directors of The NetBSD Foundation. If you are curious about how this all happened, here it goes: as described in the election procedure, someone who I don't know nominated me back in November of 2010 to become part of the new board composition. After the Nomination Committee made their way through the long list of nominees, interviews and deliberation, they proposed a slate for the new members of the board.
Did you ever wonder where the "sticky" part of the "sticky bit" name comes from? I actually didn't, but I just came across the Sticky bit page on Wikipedia through a tweet from @AaronToponce and discovered why. If you have used any "recent" (the quotes are important) Unix-like system, you probably know what the sticky bit is used for: to restrict the deletion of files in a directory to, basically, the owner of such files.
Wow. I have just realized that I have not blogged at all about the project that has kept me busy for the past two months! Not good, not good. "What is this project?", I hear. Well, this project is Kyua. A bit of background first: the Automated Testing Framework, or ATF for short, is a project that I started during the Summer of Code of 2007. The major goal of ATF was, and still is, to provide a testing framework for the NetBSD operating system.
Thanks to Antti Kantee's efforts, atf is seeing increasing visibility in the NetBSD community during the past few months. But one of the major concerns that we keep hearing from our developers is "Where is the documentation?". Certainly I have been doing a pretty bad job at that, and the current in-tree documents are a bit disorganized. To fix the short-term problem, I have written a little tutorial that covers pretty much every aspect that you need to know to write atf tests and, in particular, how to write such tests for the NetBSD source tree.
Antti Kantee has been, for a while, writing unit/integration tests for the puffs and rump systems (for which he is the author) shipped with NetBSD. Recently, he has been working on fixing the NetBSD test suite to report 0 failures in the i386 platform so as to encourage developers to keep it that way while doing changes to the tree. The goal is to require developers to run the tests themselves before submitting code.
Oops! Looks like I forgot to announce the release of ATF 0.9 here a couple of weeks ago. Just a short notice that the formal release has been available since June 3rd and that 0.9 has been in NetBSD since June 4th! You can also enjoy a shiny-new web site! It even includes a FAQ! And, as a side note: I have added a test target to the NetBSD Makefiles, so now it's possible to just do make test within any subdirectory of src/tests/ and get what you expect.
Finished importing ATF 0.8 into the NetBSD source tree. Wow, the CVS import plus merge was much easier than I expected. Note that, while the NetBSD test suite should continue to work as usual, there are some backwards incompatible changes in the command line interface of test programs. If you are used to run them by hand, expect different results. Please read the release news for details. Now let's wait for complaints about broken builds!
As briefly outlined in the previous post, new versions of Glib provide GIO, a library that intends to be a low-level file system API on top of the POSIX interface. This library provides an interface to asynchronously wait for file system change notifications including the creation, deletion and modification of files. The monitoring functionality in GIO is modular: it is backed by different loadable plugins that implement OS-specific functionality. In particular, GIO uses an inotify module in Linux and a FAM module everywhere else.
A few days ago, I decided to start using NetBSD, as well as Gnome on NetBSD once again, mostly because the lack of their use makes my skills feel rusty in many different areas. While NetBSD has surprised me in a good way (I am running it on a Macbook Pro and things like wireless and DRI work), Gnome has not. There are tons of broken things that prevent a smooth user experience.
For the 6th year in a row, NetBSD is a mentoring organization for Google Summer of Code 2010! If you are a bright student willing to develop full-time for an open source project during this coming summer, consider applying with us! You will have a chance to work with very smart people and, most likely, in the area that you are most passionate about. NetBSD, being an operating system project, has offers for project ideas at all levels: from the kernel to the packaging system, passing by drivers, networking tools, user-space utilities, the system installer, automation tools and more!
Wow, it has been a long time... 5 years ago, I created the monotone-server package in pkgsrc, a package that provided an interactive script to set up a monotone server from scratch with, what I though, minimal hassle. My package did the job just fine, but past year I was blown away by the simplicity of the same package in Fedora: their init.d script provides a set of extra commands to initialize the server before starting it up, and that is it.
Yesterday, I spent a while installing NetBSD/macppc 5.0.1 on a Mac Mini G4. The process wasn't easy, as it involved the following steps. I'm omitting many details, as they are "common knowledge" to Mac users (or otherwise can be easily found on the net): After booting the installer from the CD image, drop into the shell.Use pdisk to create an Apple_HFS partition for the boot loader and two Apple_UNIX_SVR2 partitions, one for the root file system and another for swap.
The NetBSD Project recently launched a new official blog for NetBSD. From here, I'd like to invite you to visit it and subscribe to it. It's only with your support (through reading and, specially, commenting) that developers will post more entries! Enjoy :-)
The Google Summer of Code 2009 application deadline for students is tomorrow and NetBSD has got very few applications so far. If you have the interest in working on a cool operating system project, where almost any project idea can fit, take the time to read our proposals and apply! New, original ideas not listed there will also be considered. It'd be a pity if the number of assigned slots to NetBSD was small due to the low number of applications!
I have a machine at work, a Dell Optiplex 745, that cannot boot GENERIC NetBSD kernels. There is a problem in one of the uhci/ehci, bge or azalia drivers that causes a lockup at boot time because of a shared interrupt problem. Disabling ehci or azalia from the kernel lets the machine boot. In order to do that, there are two options: either you rebuild your kernel without the offending driver, or you boot into the userconf prompt with -c and, from there, manually disable the driver at each boot.
I am very happy to announce the availability of the 0.6 release of ATF. I have to apologize for this taking so long because the code has been mostly-ready for a while. However, doing the actual release procedure is painful. Testing the code in many different configurations to make sure it works, preparing the release files, uploading them, announcing the new release on multiple sites... not something I like doing often.
NetBSD-current has recently switched time_t to be a 64-bit type on all platforms to cope with the year-2038 problem. This is causing all sorts of trouble, and a problem I found yesterday was that, after a clean install of NetBSD/amd64, it was impossible to change the data of any user through chfn. The command failed with: chfn: /etc/master.passwd: entry root inconsistent expire chfn: /etc/master.passwd: unchanged Suspiciously, the data presented by chfn showed an expiration date for root set in a seemingly-random day (October 14th, 2021).
I reinstalled NetBSD-current recently on my shark (Digital DNARD) and, out of curiosity, I wanted to see if the new-style kernel modules worked fine on this platform. To test that, I attempted to load the puffs module and failed with an error message saying something like "kobj_reloc: unexpected relocation type 1". Similarly, the same error appeared when running the simpler regression tests in /usr/tests/modules. After seeing that error message, I tracked it down in the source code and ended in src/sys/arch/arm/arm32/kobj_machdep.
NYCBSDCon 2008 will take place in New York City on October 11th and 12th. Given that I am already in NYC and will still be by that time, I submitted a presentation proposal about ATF. I have just been notified that my proposal has been accepted and, therefore, I will be giving a talk on ATF itself and how it relates to NetBSD on one of those two days. The conference program and schedule have not been published yet, though, so keep tuned.
Yesterday night, I got back from the "I Jornadas Tecnológicas Isla Cristina", a small technological conference organized at Isla Cristina, a little town in Huelva, Spain. The main organizers were the teachers of a local technical school (the IES Padre José Miravent), and they invited me to give a talk about NetBSD development. I will publish the slides soon, but I have to warn you that you will not like the source format, aka PowerPoint.
Google has launched the Summer of Code program once again this year, and NetBSD is a mentoring organization for the fourth time as announced in a netbsd-announce post. Unless things go very wrong in the following days, I will not take part this year as a student because I will be intering at Google SRE during the Summer! However, I will try to become a mentor for the "Convert all remaining regression tests to ATF"
One of the pending to-do entries for ATF 0.4 is (was, mostly) the ability to define a timeout for a test case after which it is forcibly terminated. The idea behind this feature is to prevent broken tests from stalling the whole test suite run, something that is already needed by the factor(6) tests in NetBSD. Given that I want to release this version past weekend, I decided to work on this instead of delaying it because.
Today's work: been fixing NetBSD's id(1)'s command line parsing to match the documented syntax. Let's explain. Yesterday, for some unknown reason, I ended up running id(1) with two different user names as its arguments. Mysteriously, I only got the details for the first user back and no error for the second one. After looking at the manual page and what the GNU implementation did, I realized that the command is only supposed to take a single user or none at all as part of its arguments.
I've just published the 0.3 release of ATF. I could have delayed it indefinitely (basically because my time is limited now), so I decided it was time to do it even if it did not include some things I wanted. The important thing here is that this release will most likely be the one to be merged into the NetBSD source tree. If all goes well, this will happen during this week.
Reposting from the original ATF news entry: I have just updated the first preview of NetBSD-current release builds with ATF merged in to match the ATF 0.1 release published today. As already stated in the old news item: These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process.
Reposting from the original ATF news entry: I have just uploaded some NetBSD-current release builds with ATF merged in. These will ease testing to the casual user who is interested in this project because he will not need to mess with patches to the NetBSD source tree nor rebuild a full release, which is a delicate and slow process. For the best experience, these releases are meant to be installed from scratch even though you can also do an upgrade of a current installation.
This weekend I have finally been able to start coding for my SoC project: the Automated Testing Framework for NetBSD. To my disliking, this has been delayed too much... but I was so busy with my PFC that I couldn't find any other chance to get my hands on it. I've started by working on two core components: libatf: The C/C++ library that will provide the interface to easily write test cases and test suites.
Even though I don't usually repost general NetBSD news, I would like to mention this one: the NetBSD web site has got a severe facelifting aiming at improving its usability and increasing the consistency among its pages. Many thanks to Daniel Sieger for his perseverance and precious work. This is something that had been attempted in the past many times but raised so many bikesheds that it was never accomplished.
pkgsrcCon 2007 is over. The conference started around 1:00pm on the 27th and has lasted until today^Wyesterday (the 29th) at around 7:00pm. There have been 10 different talks as planned, although we weren't able to follow the proposed schedule. Most of the presentations were delayed and some were shifted because the speakers could not arrive on time. Not a big deal though. We have been, more or less, around 20-25 people.
Yes, Google Summer of Code (SoC) 2007 is back and I'm in once again! This means I'll be able to spend another summer working on free software and deliver some useful contributions by the end of it. This time I sent just one proposal, choosing NetBSD as the mentoring organization. The project is entitled Automated testing framework and is mentored by Martin Husemann. This framework is something I've had in mind for a long time already; in fact, I also applied for this in SoC 2006 and attempted to develop this project as my undergraduate thesis.
I recently installed NetBSD-current (4.99.12 at the time I did this) inside Parallels Desktop for Mac. Everything went fine except for the configuration of the XFree86 shipped with the base system: I was unable to get high resolutions to work (over 1024x768 if I recall correctly), and I wanted to configure a full-screen desktop. In my specific case, this is 1440x900, the MacBook Pro's native resolution. It turns out I had to manually add a mode line to the XF86Config file to get that resolution detected.
Yes, ladies and gentlemen: Google Summer of Code 2007 is here and NetBSD is going to become a mentoring organization again (unless Google rejected the application, that is)! We are preparing a list of projects suitable for SoC; spend some time looking for one that interests you (I'm sure there is something) and get ready to send your proposal between the 14th and 24th of this same month. I've already made my choice :-)
A bit more than a year ago I started working on Multiboot support for NetBSD. This work was completed by the end of past year and was integrated into the main source tree, allowing any user to boot his NetBSD installation by using GRUB alone, without having to chainload different boot loaders. I've written an introductory article on Multiboot and how NetBSD was converted to support it. It has just been published at ONLamp.
This article first appeared on this date in O’Reilly’s ONLamp.com online publication. The content was deleted sometime in 2019 but I was lucky enough to find a copy in the WayBack Machine. I reformatted the text to fit the style of this site and fixed broken links, but otherwise the content is a verbatim reproduction of what was originally published. The i386 architecture is full of cruft required to maintain compatibility with old machines that go back as far as the 8086 series.
It has finally come the time when I have to choose a subject for my undergraduate thesis on which I'll be working on full time next semester. My first idea was to make a contribution to NetBSD by developing an automated testing framework. I have had interest in this for a long while (I even proposed it as part of this year's SoC), and there is a lot of interest in it within the project too.
The implementation of an efficient memory-based file system (tmpfs) for NetBSD was my Google Summer of Code 2005 project. After the program was over, the code was committed to the repository and some other developers (specially YAMAMOTO Takashi) did several fixes and improvements in it. However, several problems remained in it that prevented tagging it release quality (see this thread). Finally I found some time to deal with most of them, something that has kept me busy for around three weeks (and which I should have done much, much earlier).
vnd(4) is the virtual disk driver found in NetBSD. It provides a disk-like interface to files which allows you to treat them as if they were disks. This is useful, for example, when a file holds a file system image (e.g. the typical ISO-9660 files) and you want to inspect its contents. Up until now vnd(4) used the vnode's bmap and strategy operations to access the backing file. These operate at the block-level and therefore do not involve any system-wide caches; this is why they were used (see below).
I've just added a couple of project proposals related to improving Ext2/Ext3 file system support in the NetBSD Operating System. These are: Implement Ext3 file system supportImprove support for Ext2 root filesystemIf you are interested in getting into file system development — a very interesting research area, believe me! ;-) — this is probably a safe bet. These two projects are not very complex but can quickly benefit NetBSD for different reasons (not only better Linux compatibility).
Dear NetBSD, It is almost five years since we first met and I still remember how much I liked you at that time. Despite your 1.5 release had slow disk performance when compared to the other BSDs, I found in you an operating system that just felt right. You focused on clean and well designed code among many other goals; sincerely, I didn't come to you looking for portability because I never had anything else than i386 machines.
A few pkgsrc developers and I have been working hard for years to bring the GNOME Desktop to this packaging system and make it work under NetBSD. We are quite happy with the current results because the packages are updated very frequently and everything works. Well, almost. There are still several missing details that really hurt the end user experience and need fixing. If things continue as have gone until now, we will always be one step (or more!
A week has almost passed since someone told me that D-Bus' session daemon was broken in NetBSD. I curse that day! ;-) I've been investigating that problem since then and (very) beleatedly fixing some issues in other GNOME programs during the process. D-Bus' session daemon did not work under NetBSD because it couldn't authenticate incoming connections; that was due to the lack of socket credentials. After some of days of investigation — which included discovering that NetBSD does indeed support socket credentials through LOCAL_CREDS — and multiple attempts to implement them, I finally got D-Bus session daemon to authenticate appropriately.
One of the problems of learning new stuff based on trial-and-error iterations is that it is very easy to miss important details... but that's the price to pay when there is no decent documentation available for a given feature. We saw yesterday multiple details about LOCAL_CREDS socket credentials and, as you may deduce, I missed some. First of all I assumed that setting the LOCAL_CREDS option only affected the next received message (I didn't mention this explicitly in the post though).
Socket credentials is a feature that allows a user process to receive the credentials (UID, GID, etc.) of the process at the other end of a communication socket in a safe way. The operating system is in charge of managing this information, sent separately from the data flow, so that the user processes cannot fake it. There are many different implementations of this concept out there as you can imagine.
This article first appeared on this date in O’Reilly’s ONLamp.com online publication. The content was deleted sometime in 2019 but I was lucky enough to find a copy in the WayBack Machine. I reformatted the text to fit the style of this site and fixed broken links, but otherwise the content is a verbatim reproduction of what was originally published. My previous article, Making Packager-Friendly Software (part 1), explains why software packaging is sometimes problematic due to real problems in the mainstream sources.
This article first appeared on this date in O’Reilly’s ONLamp.com online publication. The content was deleted sometime in 2019 but I was lucky enough to find a copy in the WayBack Machine. I reformatted the text to fit the style of this site and fixed broken links, but otherwise the content is a verbatim reproduction of what was originally published. A package maintainer, or packager, is a person who creates packages for software projects.