Posts: Kyua

Kyua graduates

After years of inactivity, the Kyua project has graduated as an open source citizen and has a new home under the FreeBSD umbrella!

But uh… wait, what is Kyua and why is this exciting? To resolve confusion and celebrate this milestone, I’d like to revisit what Kyua is, how it came to be, why I stopped working on it for a while, why that was a problem for FreeBSD—and, indirectly, NetBSD—and how Kyua being free software has helped keep it alive.

August 2, 2024 · Tags: blogsystem5, freebsd, kyua, netbsd
Continue reading (about 14 minutes)

Offloading maintenance tasks to Travis CI

From day one, the Kyua source tree has had docstring annotations for all of its symbols. The goal of such docstrings is to document the code for the developers of Kyua: these docstrings were never intended to turn into pre-generated HTML documentation because Kyua does not offer an API once installed.

As you might have noticed, Doxygen is an optional component of the build and it used to run on each make invocation. This changed “recently”. Nowadays, Doxygen is only run asynchronously on Travis CI to report docstring inconsistencies post-submission (see the DO=apidocs matrix entry if you are impatient). Combined with feature branches that are only merged into master when green, this is as good as the previous approach of running Doxygen along the build. Scratch that: this is even better because running Doxygen locally on each build took significant resources and penalized edit/build/test cycles.

May 21, 2015 · Tags: automation, ci, kyua, travis
Continue reading (about 8 minutes)

Kyua turns parallel

After three months of intensive work on Kyua's executor Git branch, I am happy to announce that the new execution engine, whose crown feature is the ability to run test cases in parallel, has just landed in master and passes all self-tests!

You can head over to the commit message for more details on the merge, read the NEWS entries, and skim throught the history of the executor branch to understand how this feature has been built.

February 28, 2015 · Tags: announce, kyua
Continue reading (about 5 minutes)

Refocusing Kyua maybe?

The FreeBSD devsummit that just passed by gave me enough insight into Jenkins to question the long-term plans for Kyua. Uh, WHAT?! Let me explain.

In the beginning...

One of the original but unstated goals of Kyua was to fix the "mess" that is the NetBSD releng test run logs site: if you pay close attention, you will notice that various individuals have reinvented the wheel over and over again in an attempt to automate release builds and test suite runs. In other words: different parties have implemented independent continuous integration systems several times with more or less success.

May 23, 2014 · Tags: freebsd, jenkins, kyua, netbsd
Continue reading (about 6 minutes)

How to merge multiple Git repositories into one

Are you looking for a method to merge multiple Git repositories into a single one? If so, you have reached the right tutorial!

Please bear with me for a second while I provide you with background information and introduce the subject of our experiments. We’ll get to the actual procedure soon and you will be able to apply it to any repository of your choice.

In the Kyua project, and with the introduction of the kyua-atf-compat component in the Summer of 2012, I decided to create independent Git repositories for each component. The rationale was that, because each component would be shipped as a standalone distfile, they ought to live in their own repositories.

February 15, 2014 · Tags: featured, git, kyua
Continue reading (about 4 minutes)

FreeBSD Test Suite tutorial at AsiaBSDCon 2014

I am pleased to announce that the tutorial on the FreeBSD Test Suite that I proposed for AsiaBSDCon 2014 has been accepted!

The conference website will soon include more details, but allow me to spoil your wait:

• Goals: Learn how to use the test suite, how it is internally organized and how new tests can be written.
• Stretch goal: Get attendees to contribute one or more tests to the project.
• Audience: Mostly developers of FreeBSD that want to learn how the new test suite plugs into the system. That said, and because a major part of the tutorial will revolve around using the test suite for one's own benefit, everyone is welcome really. In particular, system administrators may get a useful tool out of this.

The main surprise in the acceptance confirmation email is that materials are due by January 20th... which is around the corner! Time to rush in getting things ready. In the meantime, you can find more details on the tutorial by reading the proposal itself.

January 8, 2014 · Tags: conference, freebsd, kyua, testing
Continue reading (about 1 minute)

FreeBSD Test Suite goals and planning

I was really impressed and delighted to see how popular my previous (and first) post on FreeBSD testing, titled Introducing the FreeBSD Test Suite, was. Looks like this project may be of great interest to developers and users out there (not unsurprisingly) so I'll try to keep you all up-to-date with any key developments.

A first question that arises from the announcement is: where are the test suite and infrastructure headed? After all, the continuous testing machines for amd64 are already up and running, so what else is there to do?

January 5, 2014 · Tags: freebsd, kyua, testing
Continue reading (about 2 minutes)

Introducing the FreeBSD Test Suite

I joined the FreeBSD committer ranks a couple of months ago with the intention to equip FreeBSD with an out-of-the-box test suite and with a testing infrastructure. The time until now has been quite fruitful and I have been rushing to get something ready for you before the year end.

With that, I am very pleased to announce that the first mockup of the FreeBSD testing cluster is up and running! Point your browser at:

December 31, 2013 · Tags: featured, freebsd, kyua, testing
Continue reading (about 4 minutes)

lutok-0.3, kyua-testers-0.2 and kyua-cli-0.8

Yesterday was release day: I pushed out Lutok 0.3, Kyua Testers 0.2 and Kyua CLI 0.8.

There are not a lot of changes in these new releases. The reason I cut them was to publish the new TAP-compliant tester and make it available for use in FreeBSD as soon as possible. I will be using this new feature as part of the FreeBSD Test Suite in order to hook existing test programs without having to rewrite them to use the ATF libraries (or at least not as a first step).

December 8, 2013 · Tags: kyua, lutok, release
Continue reading (about 2 minutes)

Three productive days on the Kyua front

This being Thanksgiving week in the U.S. and Google giving us Thursday and Friday off, I decided to take Monday to Wednesday off as well to spend some time hacking on Kyua — yes, finally, after months of being inactive. And what a three productive days!

Here comes a little briefing on the three fronts in which I made progress. (This puts on hold the header files series until next Monday... but most of you are probably away anyway. Enjoy the holidays if they apply to you!)

November 28, 2013 · Tags: atf, fedora, freebsd, kyua, lua
Continue reading (about 6 minutes)

EuroBSDCon 2013 takeaways

EuroBSDCon 2013 is done. If you have been following my daily posts over the last 4 days (day 1, day 2, day 3 and day 4) as well as #EuroBSDCon updates in Twitter, you may already have a pretty good idea of what went on here. However, with the conference over, it is now a good time to recap the whole event and present the takeaways of these four days which, overall, were quite interesting and productive.

September 30, 2013 · Tags: eurobsdcon, freebsd, kyua, netbsd
Continue reading (about 6 minutes)

Live from EuroBSDCon 2013, day 4

Live from Malta today attending the EuroBSDCon 2013 conference. The conference is over; today was the second and last day and it has just finished.

Hardware and virtualization

One of the three tracks today included a lot of talks on hardware, porting of BSDs to new hardware and virtualization techniques. Of all these, the few talks I attended covered the topics in great detail and proved to be very interesting.

September 29, 2013 · Tags: conference, eurobsdcon, kyua, virtualization
Continue reading (about 4 minutes)

Live from EuroBSDCon 2013, day 3

Live from Malta today attending the EuroBSDCon 2013 conference. Today is the first day of the conference itself. Many more people have shown up as expected and there have been tons of very interesting talks all the time. It is both good and bad that there are several tracks: you can select the topic you are most interested in, but sometimes great talks overlap!

Keynote

Today's opening session was led by Theo de Raadt, the founder of OpenBSD. His keynote focused on explaining how there is no real research happening on operating systems any more and how new, risky technological changes can be tested in a real-world system like OpenBSD.

September 28, 2013 · Tags: conference, eurobsdcon, kyua, netbsd
Continue reading (about 4 minutes)

Setting up my old Mac Mini G4 as a development machine

I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below.

I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it... which is great, but when thinking about it objectively, I am reluctant to unnecessarily "punish" all NetBSD users by increasing the build time of the system significantly. Don't get me wrong: Kyua itself runs efficiently on old machines, but building the code — particularly the hundreds of tests that come with it — takes just too long. This slowdown is not too noticeable on a relatively-modern machine, but it's unacceptable on not-so-old machines.

Of course, I could proceed with the import right now (with the code disabled by default) and make it leaner later, but this would cause quite a first bad impression on our users. So it's better to delay the import a little bit until, at least, I have had a chance to simplify some core parts of the code. Mind you, this simplification work is already in progress and quite advanced: it consists on modularizing (as separate processes) some critical parts of the code and writing these with a much simpler style and in plain C.

But back to the original point of my post.

The first thing to mention about this experience is that with some effort and long waits, I've got a pretty decent setup for development even on this old machine. From time to time, I miss having a real Unix desktop at hand for development (no OS X, you are not one of those). The GUI behaves relatively well with a 1920x1200 display, running Fluxbox, traditional xterms, Mutt for GMail access and a bunch of other applications.

Unfortunately, too many things feel really sluggish. A few specific examples:

• Firefox 16 is barely usable. I'm not sure there are many alternatives to decent web browsing for such an old non-Intel platform. Dillo is blazing fast and allows me to access online documentation and mailing list archives (just enough for development), but it's pretty much useless for any other purpose given the "Web 2.0".
• Any operation that involves pkgsrc takes ages. Even when building the simplest packages, one can notice the system crawl through all of the pkgsrc infrastructure. Sometimes this is the fault of a bad algorithm; other times it's just sh(1) being the wrong tool for something as complex as pkgsrc internals.
• Things like basic code editing in Emacs 24 are slow at responding to typing. Disabling font lock mode makes it feel fast again, but it's just surprising to see that even color coding is slow.

October 22, 2012 · Tags: kyua, lab-notes, mac, netbsd
Continue reading (about 3 minutes)

Kyua gets its own blog

For the last couple of weeks, I have been pondering the creation of a Kyua-specific blog. And, after a lot of consideration, I have finally taken the plunge. Say hello to Engineering Kyua!

June 4, 2012 · Tags: blog, kyua
Continue reading (about 3 minutes)

Exposing a configuration tree through Lua

In the previous post, I discussed the type-safe tree data structure that is now in the Kyua codebase, aimed at representing the configuration of the program. In this post, we'll see how this data structure ties to the parsing of the configuration file.

One goal in the design of the configuration file was to make its contents a simple key/value association (i.e. assigning values to predetermined configuration variables). Of course, the fact that the configuration file is just a Lua script means that additional constructions (conditionals, functions, etc.) can be used to compute these values before assignment, but in the end all we want to have is a collection of values for known keys. The tree data structure does exactly the latter: maintain the mapping of keys to values, and ensuring that only a set of "valid" keys can be set. But, as a data structure, it does not contain any of the "logic" involved in computing those values: that is the job of the script.

Now, consider that we have the possible following syntaxes in the configuration file:

simple_variable = "the value"
complex.nested.variable = "some other value"

These assignments map, exactly, to a tree::set() function call: the name of the key is passed as the first argument to tree::set() and the value is passed as the second argument. (Let's omit types for simplicity.) What we want to do is modify the Lua environment so that these assignments are possible, and that when such assignments happen, the internal tree object gets updated with the new values.

In order to achieve this, the configuration library modifies the Lua environment as follows:

• The newindex metatable method of _G is overridden so that an assignment causes a direct call to the set method of the referenced key. The key name is readily available in the newindex arguments, so no further magic is needed. This handles the case of "a = b" (top-level variables).
• The index metatable method of _G is overridden so that, if the indexed element is not found, a new table is generated and injected into _G. This new table has a metatable of its own that performs the same operations as the newindex and index herein described. This handles the case of "a.b = c", as this trick causes the intermediate tables (in this case "a") to be transparently created.
• Each of the tables created by index has a "key" metatable field that contains the fully qualified key of the node the table corresponds to. This is necessary to be able to construct the full key to pass to the set method.
• There is further magic to ensure that values pre-populated in the tree (aka default values) can be queried from within Lua, and that variables can be set more than once. These details are uninteresting though.

June 2, 2012 · Tags: cxx, kyua, lua
Continue reading (about 3 minutes)

Type-safe, dynamic tree data type

The core component of the new configuration library in Kyua is the utils::config::tree class: a type-safe, dynamic tree data type. This class provides a mapping of string keys to arbitrary types: all the nodes of the tree have a textual name, and they can either be inner nodes (no value attached to them), or leaf nodes (an arbitrary type attached as a value to them). The keys represent traversals through such tree, and do this by separating the node names with dots (things of the form root.inner1.innerN.leaf).

The tree class is the in-memory representation of a configuration file, and is the data structure passed around methods and algorithms to tune their behavior. It replaces the previous config static structure.

The following highlights describe the tree class:

• Keys are (and thus the tree layout is) pre-registered. One side-effect of moving away from a static C++ structure as the representation of the configuration to a dynamic structure such as a tree is that the compiler cannot longer validate the name of the configuration settings when they are queried. In the past, doing something like config.architecture would only compile if architecture was a valid structure defined... but now, code like config["architecture"] cannot be validated during the build.
  In order to overcome this limitation, trees must have their keys pre-defined. Pre-defining the keys declares their type within the tree.  Accesses to unknown keys results in an error right away, and accesses to pre-defined keys must always happen with their pre-recorded types.
  Note that pre-defined nodes can, or cannot, hold a value. The concept of "being set" is different than "being defined".
• Some nodes can be dynamic. Sometimes we do not know what particular keys are valid within a context. For example, the test_suites subtree of the configuration can contain arbitrary test suite names and properties within it, and there is no way for Kyua (at the moment) to know what keys are valid or not.
  As a result, the tree class allows defining a particular node as "dynamic", at which point accesses to any undefined keys below that node result in the creation of the node.
• Type safety. Every node has a type attached to it. The base configuration library provides common types such as bool_node, int_node and string_node, but the consumer can define its own node types to hold any other kind of data type. (It'd be possible, for example, to define a map_node to hold a full map as a tree leaf.)
  The "tricky" (and cool) part of type safety in this context is to avoid exposing type casts to the caller: the caller always knows what type corresponds to every key (because, remember, the caller had to predefine them!), so it knows what type to expect from every node. The tree class achieves this by using template methods, which just query the generic internal nodes and cast them out (after validation) to the requested type.
• Plain string representations. The end user has to be able to provide overrides to configuration properties through the command line... and the command line is untyped: everything is a string. The tree library, therefore, needs a mechanism to internalize strings (after validation) and convert them to the particular node types. Similarly, it is interesting to have a way to export the contents of a tree to strings so that they can be shown to the user.

config::tree tree;

// Predefine the valid keys.
tree.define< config::string_node >("kyua.architecture");
tree.define< config::int_node >("kyua.timeout");

// Populate the tree with some sample values.
tree.set< config::string_node >("kyua.architecture", "powerpc");
tree.set< config::int_node >("kyua.timeout", 300);

// Query the sample values.
const std::string architecture =
    tree.lookup< config::string_node >("kyua.architecture");
const int timeout =
    tree.lookup< config::int_node >("kyua.timeout");

tree.set_string("kyua.architecture", "powerpc");
tree.set_string("kyua.timeout", "300");

config::tree tree;

// Predefine a subtree as dynamic.
tree.define_dynamic("test_suites");

// Populate the subtree with fictitious values.
tree.set< config::string_node >("test_suites.NetBSD.ffs", "ext2fs");
tree.set< config::int_node >("test_suites.NetBSD.iterations", 5);

// And the querying would happen exactly as above with lookup().

May 29, 2012 · Tags: cxx, kyua
Continue reading (about 5 minutes)

Rethinking Kyua's configuration system

In the previous blog post, I described the problems that the implementation of the Kyua configuration file parsing and in-memory representation posed. I also hinted that some new code was coming and, after weeks of work, I'm happy to say that it has just landed in the tree!

May 28, 2012 · Tags: cxx, kyua, lua
Continue reading (about 4 minutes)

Kyua's configuration system showing its age

A couple of years ago, when Kyua was still a newborn, I wrote a very ad-hoc solution for the parsing and representation of its configuration files. The requirements for the configuration were minimal, as there were very few parameters to be exposed to the user. The implementation was quick and simple to allow further progress on other more-important parts of the project. (Yep, quick is an euphemism for dirty: the implementation of the "configuration class" has to special-case properties everywhere to deal with their particular types... just as the Lua script has to do too.)

May 26, 2012 · Tags: cxx, kyua
Continue reading (about 3 minutes)

Kyua gets nicer console messages

For the last couple of weeks, particularly during a bunch of long flights, I have been improving the command-line user interface of Kyua by implementing controlled line wrappings on screen boundaries: messages that are too long to fit on the screen are preprocessed and split into multiple lines at word boundaries. This affects informational messages, error messages and, specially, the output of the built-in help command.

I originally got this idea from Monotone and later implemented it into ATF but, when writing Kyua's code, I decided to postpone its implementation until a later stage. Reusing the code from ATF was not "nice" because the API of the formatting code was quite nasty, and reimplementing this feature during the initial stages of Kyua felt like a waste of time.

However, because controlled line wrapping is crucial to having readable built-in help messages, I have had to do this eventually and the time finally came.

The ATF approach

Why did I say that the ATF code for line wrapping was quite nasty? The main reason is that the printing of messages was incredibly tied to their wrapping. All the code in ATF that prints a message to the screen has to deal with the line wrapping itself, which involves dealing with too many presentation details.

For example, consider the help routine that prints the table of options and their descriptions. This routine has to calculate the width of the longest option first and then, for every option, output its name, output some padding, and output the description properly refilled so that subsequent lines are properly arranged with respect to the previous one. While this may not sound too bad in writing, it actually is in code.

Furthermore, because all this formatting logic is spread out throughout the code, there is no way to perform decent unit testing. The unit testing did some basic tests on input text, but could not validate that more complex constructions were working right.

The Kyua approach

In the Kyua codebase, I decided to take a more declarative and functional approach. Smaller, pure building blocks that can be combined to achieve more complex constructions and that can be easily tested for correctness individually or in combination.

I started by implementing a simple function that takes a paragraph and reformats it to any given length. This simple function alone gives full flexibility to the caller to decide how to later merge this reformatted text with other text: e.g. place a line prefix or bundle such text inside a table cell.

The next step was to implement tables. Code wishing to print, e.g. the collection of options/commands along their descriptions only cares about declaring a table of two columns and N rows; why should it bother about properly lining up the two columns and printing them? It doesn't, hence the table approach. With tables, the caller can just decide which particular column needs to be wrapped if the table does not fit on the screen, and allow the formatting code to do this. Plus, having this higher level constructs means that we can eventually print the textual reports in a nicer, tabulated way (not done yet).

And the last step was to mix all these higher level constructs into the console frontend class. This class (the ui) knows how to query the width of the terminal and knows how to fit certain kinds of text and/or tables within such width. For example, error messages are not tables: they are messages prefixed with the command name; only the message has to be reformatted if it does not fit while the rest of the text has to flow after the command name. Or, for tables, the maximum width of the terminal determines how wide the table can be and thus how much one of its columns has to be refilled.

Getting this whole thing right working has proven to be extremely tricky and I'm sure there are still quite a few rough edges to be discovered. That said, it has been fun enough :-)  But, after this experience, I certainly don't want to imagine the pain that the writers of HTML/CSS renderers have endured... this text-based table-rendering is trivial compared to what web browsers do!

April 2, 2012 · Tags: kyua
Continue reading (about 4 minutes)

Kyua generates its first public HTML report

Lately, three long trips (5 hours in a bus, and 6 and 10 hours in two planes) have allowed me to work on the long-promised HTML reporting feature of Kyua. The result of these three trips is, effectively, the ability to generate HTML reports for specific test actions!

The current results are extremely rudimentary (they lack tons of would-be-useful information) and not that aesthetically pleasing. However, the database already records enough information to make these reports more useful and pretty, so "all that is left" is coming up with the necessary code to extract such information in an efficient way and spending time creating a visually-nicer appearance. None of these are as trivial as they sound, but I prefer to work one step at a time (i.e. coming up first with a very rough draft and improve it later) rather than keeping the feature private until it is "perfect".

Without further ado, you can take a look at the report of the execution of the NetBSD test suite. This output comes from my NetBSD-current virtual machine, and I've set up a couple of cron jobs to keep it up to date. (If the "action X" in the title does not increase periodically, you will know that something is broken on my side. I found that the VM already crashed, so I don't know for how long it will run now after a restart...)

March 13, 2012 · Tags: kyua
Continue reading (about 2 minutes)

Projects migrated to Git

I finally took the plunge. Yesterday night, I migrated the Kyua and Lutok repositories from Subversion to Git. And this morning I migrated ATF from Monotone and custom hosting to Git and Google Code; oh, and this took way longer than expected.

Migration of Kyua and Lutok

Migrating these two projects was straightforward. After preparing a fresh local Git repository following the instructions posted yesterday, pushing to Google Code is a simple matter:

$ git remote add googlecode https://code.google.com/p/your-project
$ git push googlecode --all
$ git push googlecode --tags

February 26, 2012 · Tags: atf, git, kyua, lutok
Continue reading (about 3 minutes)

Kyua 0.3 released!

Dear readers,

I am pleased to announce that Kyua 0.3 is available!

The major feature in this release is the introduction of the "test results store"; i.e. a SQLite3-based database that records all test execution activity so that reports can be gathered in a central machine and reports generated out of it. This is still very experimental and the generated reports are quite rudimentary, but is a first step towards this direction.

The Kyua 0.3 release page provides links to the download as well as the list of major changes in this new release.

It has been a really long time since the 0.2 release. I wanted to postpone 0.3 until HTML reports were ready, but I have not had the time to implement this for a while already. Instead of postponing the release even further, I have decided to create it right now and then take the time to create nice reports separately. Additionally, I am planning on doing some repository changes and wanted to do them without an imminent release along the way.

The package in pkgsrc has been updated and now I'm working on bringing binary packages to Fedora.

Enjoy!

February 24, 2012 · Tags: kyua, release
Continue reading (about 1 minute)

Kyua: Weekly status report

A couple of things happened this week:

• Spent quite a few time researching the idea of moving away from Monotone and Subversion to Git. I haven't made a decision yet, but I'm pretty convinced this is the right way to go. It will simplify development significantly, it will allow me to code offline (have a bunch of really long flights coming), and it will lower the entry barrier to Kyua by making all components use the same, mainstream VCS.
• Implemented filtering by result type of the test case results in the textual reports.

February 13, 2012 · Tags: kyua, report
Continue reading (about 1 minute)

Switching projects to Git

The purpose of this post is to tell you the story of the Version Control System (VCS) choices I have made while maintaining my open source projects ATF, Kyua and Lutok. It also details where my thoughts are headed to these days.

This is not a description of centralized vs. distributed VCSs, and it does not intend to be one. This does not intend to compare Monotone to Git either, although you'll probably feel like it while reading the text. Note that I have fully known the advantages of DVCSs over centralized systems for many years, but for some reason or another I have been "forced" to use centralized systems on and off. The Subversion hiccup explained below is... well... regrettable, but it's all part of the story!

Hope you enjoy the read.


Looking back at Monotone (and ATF)

I still remember the moment I discovered Monotone in 2004: simply put, it blew my mind. It was clear to me that Distributed Version Control Systems (DVCSs) were going to be the future, and I eagerly adopted Monotone for my own projects. A year later, Git appeared and it took all the praise for DVCSs: developers all around started migrating en masse to Git, leaving behind other (D)VCSs. Many of these developers then went on to make Git usable (it certainly wasn't at first) and well-documented. (Note: I really dislike Git's origins... but I won't get into details; it has been many years since that happened.)

One of the projects in which I chose to use Monotone was ATF. That might have been a good choice at the time despite being very biased, but it has caused problems over time. These have been:

• Difficulty to get Monotone installed: While most Linux distributions come with a Monotone binary package these days, it was not the case years ago. But even nowadays if all Linux distributions have binary packages, the main consumers of ATF are NetBSD users, and their only choice is to build their own binaries. This generates discomfort because there is a lot of FUD surrounding C++ and Boost.
• High entry barrier to potential contributors: It is a fact that Monotone is not popular, which means that nobody is familiar with it. Monotone's CLI is very similar to CVS, and I'd say the knowledge transition for basic usage is trivial, but the process of cloning a remote project was really convoluted until "recently". The lack of binary packages, combined with complex instructions on just how to fetch the sources of a project only help in scaring people away.
• Missing features: Despite years have passed, Monotone still lacks some important features that impact its usability. For example, to my knowledge, it's still not possible to do work-directory merges and, while the interactive merges offered by the tool seem like a cool idea, they are not really practical as you get no chance to validate the merge. It is also not possible, for example, to reference the parent commit of any given commit without looking at the parent's ID. (Yeah, yeah, in a DAG there may be more than one parent, but that's not the common case.) Or know what a push/pull operation is going to change on both sides of the connection. And key management and trust has been broken since day one and is still not fixed. Etc, etc, etc.
• No hosting: None of the major project hosting sites support Monotone. While there are some playground hosting sites, they are toys. I have also maintained my own servers sometimes, but it's certainly inconvenient and annoying.
• No tools support: Pretty much no major development tools support Monotone as a VCS backend. Consider Ohloh, your favorite bug tracking system or your editor/IDE. (I attempted to install Trac with some alpha plugin to add Monotone support and it was a huge mess.)
• No more active development: This is the drop that spills the cup. The developers of Monotone that created the foundations of the project left years ago. While the rest of the developers did a good job in coming up with a 1.0 release by March 2011, nothing else has happened since then. To me, it looks like a dead project at this point :-(

• ATF continues to be hosted in a Monotone repository, and Kyua depends on ATF. You can spot the problem, can't you? It's a nightmare to check out all the dependencies of Kyua, using different tools, just to get the thing working.
• As of today, Git is as popular, if not more, than Subversion. All the major operating systems have binary packages for Git and/or bundle Git in their base installation (hello, OS X!). Installing Git on NetBSD is arguably easier (at least faster!) than Subversion. Developers are used to Git. Or let me fix that: developers love Git.
• Subversion gets on the way more than it helps; it really does once you have experienced what other VCSs have to offer. I currently maintain independent checkouts of the repository (appropriately named 1, 2 and 3) so that I can develop different patches on each before committing the changes. This gets old really quickly. Not to mention when I have to fly for hours, as being stuck without an internet connection and plain-old Subversion... is suboptimal. Disconnected operation is key.

February 11, 2012 · Tags: atf, git, kyua, lutok, monotone, vcs
Continue reading (about 8 minutes)

Kyua: Weekly status report

• Created an RPM package for Lutok for inclusion in Fedora.
• Created a preliminary RPM spec for ATF for Fedora. Now in discussions with the FPC to figure out how to install the tests on a Fedora system, as /usr/tests may not be appropriate.
• No activity on Kyua itself though, unfortunately.

February 7, 2012 · Tags: atf, kyua, lutok, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This comes two days late... but anyway, I felt like posting it now instead of waiting until next Sunday/Monday.

The activity past week focused mostly on implementing support for the require.memory test-case metadata property recently introduced into ATF. This was non-trivial due to the need to write some tricky Autoconf code to make this "slightly portable". Seriously: It's scary to see how hard it is to perform, in a portable manner, an operation as simple as "query the amount of physical memory"... but oh well, such are the native Unix APIs...

I later spent the weekend on Lutok, preparing and publishing the project's first release and writing my first RPM library spec for it. This was a prerequisite for the upcoming 0.3 release of Kyua and thus deserved special attention!

February 1, 2012 · Tags: kyua, lutok, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Released ATF 0.15 and imported it into NetBSD.
• Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.
• I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them. While this sounds like a cool idea (yay, code reuse!), it resulted in a ugly, nasty and horrible mess. In the end, I just ended up implementing custom parsing of the formatters, which was way easier and "good enough" for Kyua's needs.
• Started work on backporting ATF's new require.memory property into Kyua. This needs having a way to parse and format byte quantities in user-friendly forms (e.g. 1k, 2m, etc.)... hence the previous work on utils::format!
• Set up a Google+ Page for Kyua. I have no idea what to use it for yet. Maybe the status reports should go in there. Ideas?

January 23, 2012 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Finally some progress!

• Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.
• Backported local pkgsrc patches to ATF into the upstream code.
• Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.
• Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua. Not very successful yet unfortunately.

January 15, 2012 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time!

January 10, 2012 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Happy new year!

No activity past week as I was completely away from the computer.

January 3, 2012 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, not much activity this week due to travel reasons. Anyway, some work went in:

• Preliminary code to generate HTML reports from kyua report. This is easy peasy but boring. The current code was written as a proof of concept and is awful, hence why it was not committed. I'm now working in cleaning it up.
• Backported test program and test case timestamping into ATF based on a patch from Paul Goyette. This is a very useful feature to have, and it will have to be added to Kyua later. (It has always been planned to be added, but have not had the time yet.)

December 19, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Some significant improvements this week:

• Finally submitted the code to store and load full test case definitions. This is quite tricky (and currently very, very ugly) but it works and it will allow the reports to include all kinds of information from the test cases.
• Removed the Atffiles from the tree; yay! For a long time, I had been using atf-run to run broken tests because atf-run allowed me to watch the output of the test case being debugged. However, this has been unnecessary since the introduction of the debug command in late August. I now feel confident that these files can go. (And debug is much more powerful than atf-run because you can target a single test case instead of a whole test program.)
• Some crazy work attempting to hide the name of SQLite types from the sqlite::statement interface. I've been only able to do so somewhat decently for bind, but all my attempts at doing the same with column result in horrible code so far. So no, such changes have not been submitted.
• As of a few minutes ago, kyua test now records the output of the test cases (stdout and stderr) into the database. These will be invaluable for debugging of test cases, particularly when the reports are posted online.
• Some preliminary work at implementing HTML reports. This, however, has not received much progress due to the previous item requiring completion.

December 11, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Some more work towards allowing storing and loading of full test case definitions (which in turn will allow us to provide detailed HTML reports). I have local changes that do this, but they are gated by the lack of some additional tests and probably some optimizations, because they slow down kyua report significantly. Regarding commits, I have only submitted some related cleanup-changes.
• I got distracted by invalid Doxygen comments and traced down why they were not being correctly validated (which is the whole point of running Doxygen during the build). It turns out I had to enable some extra EXTRACT_* settings in the Doxyfile. After doing so, I realized there are many, many documentation problems in the code... and I have been fixing them since. It's a tough operation, but I'm more than half-way through already. To give you an idea, the current diffstat shows about 650 new comment lines (!).

December 5, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes.

Due to this, I ended up doing some cleanups. First, I reimplemented the way in which test programs that fail to list their test cases are represented. And second, I got rid of the useless test_case_id class, which exposed even further problems in the data types that represent test cases.

It's now time to sit and think if the current representations of test programs and test cases make sense and, if not, how to better redo them. Not going to be easy, but I hope to have some time for this cleanup during this upcoming week.

November 28, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori.

$ kyua test

[... wait for the tests to run ...]

Committed action 82

$ kyua report
===> Skipped tests
[... too long to show ...]
===> Expected failures
integration/cmd_report_test:output__console__change_file  ->
    expected_failure: --output not implemented yet:
    atf-check failed; see the output of the test for details
===> Failed tests
store/transaction_test:put_test_case__ok  ->
    failed: Line 663: stmt.step() not met
===> Summary
Action: 82
Test cases: 934 total, 15 skipped, 1 expected failures,

0 broken, 1 failed

November 20, 2011 · Tags: kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept.

November 14, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database.

"Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results. It's interesting to realize that most of these features are just one SQL query away. If you install Kyua, you can already run a few tests and then use kyua db-exec to issue arbitrary SQL queries against the database; the schema (see store/schema.sql) might look a bit convoluted, but a bunch of NATURAL JOINs will yield the desired output.

The feature requests that have the highest priority at this point are the ability to generate a report of the last tests run both as a text file and as an HTML dashboard, because having these features means we can finally kill the atf-run and atf-report pair. At this point I'm, once again, "stuck" while figuring out how to best organize the code to make all these things possible while still keeping a nice separation across the existing layers (cli, engine and store)... all without introducing much unnecessary complexity. But exciting times lie ahead!

November 7, 2011 · Tags: kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Submitted a placeholder implementation of the new persistence layer (store top-level directory). This only supports opening a database and ensuring its metadata is valid. See r253.
• Added a db-exec CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet. See r255.
• Miscellaneous fixes and improvements to utils::env and utils::sqlite.
• Preliminary code to support putting objects (like actions and contexts) into the database. I've been thinking about this for a while and finally came up with a design that completely decouples the persistence needs from the higher-level classes in the engine layer. I haven't submitted the code yet though, as it lacks tests. (Still thinking how to write the loading of objects though.)

October 31, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.
• Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.
• After this refactoring, the code left in the CLI modules is purely presentation-related, and the code in the engine implements all the logic.
• The goal of these changes is to be able to hide the interactions with the database in these controllers. The CLI layer has no business in dealing with the database connection (other than allowing the user to specify which database to talk to, of course).
• Implemented a very simple RAII model for SQLite transactions.
• Some additions to the utils::sqlite bindings to simplify some common calling patterns (e.g. binding statement parameters by name).
• Preliminary prototypes at database initialization. This involves creating new databases and populating them with the initial schema, plus dealing with database metadata to, e.g. detect if we are dealing with the correct schema version.
• The code for this is still too crappy to be submitted, so don't look for it in the repository just yet!
• The design document details many things that should be part of the schema (e.g. "sessions"0, but I've decided that I'll start easy with a simplified schema and later build on top of it. Otherwise there will be too many clunky moving parts to deal with while the fundamental ideas are not yet completely clear.
• Fixes to let the code build and run again in NetBSD (macppc at least).

October 23, 2011 · Tags: kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Added support to prepare and execute statements to utils::sqlite.
• Added the UTILS_PURE definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.
• Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.
• Made a quick attempt at getting kyua test record test results in a simple database; success! This was just a proof of concept, so it is not submitted yet.
• Started to refactor the code to move many logic constructions from the cli to the engine. With the need to store test results in the database, it's clear that these semantics do not belong in the CLI, but the current code structure do not easily allow this. Need to add some "controller" classes in the engine to hide all the interaction among internal components.

October 16, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Some more reading on SQLite. The Using SQLite book is close to awesome; very easy to read and full of insightful tips.
• Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep procrastinating. So, ladies and gentlemen, welcome the new utils::sqlite module in Kyua. At the moment, this just provides the barebones to open and close a database. I need to start playing around with preparing statements before I can model these in my wrapper classes. (Yes, the API structure is extremely similar to that in Lutok.)

October 9, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Unfortunately, not much time this week either :-(
• I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases.  However, I've not had the to time to finish these edits and publish them.

October 3, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, no activity this week other than some brainstorming on the database design.

Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!

September 26, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report (db designdoc edition!)

• Moved the code of utils::lua to a new project, Lutok.
• Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.
• Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.
• Been chewing through the first chapters of the "Using SQLite" book to refresh my SQL skills.
• And, most importantly, wrote a preliminary design document for the database store of Kyua and the reporting features. Comments certainly welcome! Be aware that this is how atf-report will be replaced, so once this is done we should be able to finally kill atf-run and atf-report altogether :-)

September 18, 2011 · Tags: kyua, lutok, report
Continue reading (about 1 minute)

Introducing Lutok: A lightweight C++ API for Lua

It has finally happened. Lutok is the result of what was promised in the "Splitting utils::lua from Kyua" web post.

Quoting the project web page:

Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a small subset of miscellaneous utility functions built on top of the wrappers.

Lutok focuses on providing a clean and safe C++ interface; the drawback is that it is not suitable for performance-critical environments. In order to implement error-safe C++ wrappers on top of a Lua C binary library, Lutok adds several layers or abstraction and error checking that go against the original spirit of the Lua C API and thus degrade performance.

Lutok was originally developed within Kyua but was later split into its own project to make it available to general developers.

September 15, 2011 · Tags: announce, cxx, kyua, lua, lutok
Continue reading (about 2 minutes)

Kyua: Weekly status report

• My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(
• On another order of things, I have started writing a design document for the database that will collect test case results and other information. Will share it as soon as it is readable and more or less complete.

September 12, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Slow week.  We had some team-related events at work and I have friends over from Dublin, so it has been hard to find some time to do work on Kyua.  Regardless, here comes the weekly report:

• Split utils::lua into its own package, per some user's request.  I'm still setting up the separate project and have to do lots of cleanup on the code, so nothing is available yet.
• Started experimenting on the long promised "tests results database".  So far, I have started writing a small utils::sqlite3 C++, RAII-based wrapper for SQLite 3 and a SQL schema for the database.  My rusty SQL skills don't help :-P

September 4, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Splitting utils::lua from Kyua

If you remember a post from January titled C++ interface to Lua for Kyua (wow, time flies), the Kyua codebase includes a small library to wrap the native Lua C library into a more natural C++ interface. You can take a look at the current code as of r129.

Quoting the previous post:

The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.

September 3, 2011 · Tags: cxx, kyua, lua
Continue reading (about 3 minutes)

Kyua: Weekly status report

Not a very active week: I've been on-call four days and they have been quite intense. Plus I have had to go through a "hurricane" in NYC. That said, I had some time to do a bit of work on Kyua and the results have been nice :-)

• Made calls to getopt_long(3) work with GNU Getopt by using the correct value of optind to reset option processing.
• Improved the configure script to error out in a clearer way when missing dependencies (pkg.m4 and Lua) are not found.
• Did some portability fixes.
• And released Kyua 0.2! (along with a pkgsrc package)
  

August 28, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua 0.2 released

Dear readers,

I am very proud to announce the second formal release of Kyua, 0.2.  This release comes with lots of internal changes and a few user-visible changes.  Instead of listing all the changes and news here, I'll just recommend you to visit the 0.2 release page and read all the notes.

This release has been tested under NetBSD-current, Mac OS X Snow Leopard, Debian sid and Ubuntu 10.04.1 LTS.

Please report any problems that you encounter.

Have fun!

August 24, 2011 · Tags: kyua, release
Continue reading (about 1 minute)

Kyua: Weekly status report

• Implemented the "debug" command. Still very rudimentary, this command allows the user to run a test case without capturing its stdout nor stderr to aid in debugging of failed test cases. In the future, this command will also allow things like keeping the work directory for manual inspection, or spawning a shell or a debugger in the work directory after a test case is executed.
• Many build fixes under different platforms in preparation for a 0.2 release. In particular, Kyua now builds under Ubuntu 10.04.1 LTS but some tests fail.
• Had to disable the execution of the bootstrap test suite within Kyua because it stalls in systems where the default shell is not bash.  I presume this is a bug in GNU Autotest, so I filed a report.

August 21, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Changed the --config and --variable options to be program-wide instead of command-specific. The configuration file should be able to hold properties that tune the behavior of Kyua, not just the execution of tests, so this makes sense.
• Added the config subcommand, which provides a way to inspect the configuration as read by Kyua.
• Got rid of the test_suites_var function from configuration files and replaced it by simple assignments to variables in the test_suites global table.
• Enabled detection of unused parameters by the compiler and fixed all warnings.
• Changed developer mode to only control whether warnings are enforced or not (not to enable the warnings themselves) and made developer mode be disabled on formal releases.

August 15, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Added the ability to explicitly define timeouts for plain test programs. This just completes the work from past week that made running plain test programs at all but had an ugly TODO in it to implement this missing feature.
• The bootstrap test suite now runs as a single test case within the whole Kyua test suite. Demonstrates the plain test programs interface functionality :-)
• Started reshuffling code to make the <tt>--config</tt> and related flags program-wide. The goal is to allow the configuration file to tune the behavior of all of Kyua, so these flags must be made generic. They were previously specific to the <tt>test</tt> subcommand only.

August 8, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Implemented the "plain" test interface. This allows plugging "foreign test programs" into a Kyua-based test suite. (A foreign test program is a program that does not use any testing framework: it reports success and failure by means of an exit code.)
• Generalized code between the atf and plain interfaces and did some cleanups.
• Attempted to fix the ATF_REQUIRE_EQ macros in ATF to evaluate their arguments only once. This has proven to be tricky and therefore it is not done yet. At the moment, the macros reevaluate the arguments on a failure condition, which is not too big of a deal.

August 1, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Finished splitting the atf-specific code from the generic structures in the engine. The engine now supports the addition of extra test interfaces with minimal effort.
• Started implementing a "plain" test interface for test programs that do not use any test framework. This is to allow muxing non-atf tests into atf-based test cases, which is required in the NetBSD tree.

July 25, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Slow week. I've been busy moving to NYC!

• Kept working on the splitting of ATF-specific code from the core test abstractions. The work is now focused on refactoring the results-reporting pieces of the code, which are non-trivial.

July 18, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

One of the major features I want in place for Kyua 0.2 is the ability to run "foreign" test programs as part of a test suite: i.e. to be able to plug non-ATF test programs into a Kyuafile. The rationale for this is to lower the entry barrier of newcomers to Kyua and, also, to allow running some foreign test suites that exist in the NetBSD source tree but that are currently not run.

The work this week has gone in the direction outlined above among other things:

• Created an abstract base class to represent test programs and provided an implementation for ATF.
• Did the same thing for test cases.
• Moved the kyua-cli package from pkgsrc-wip into pkgsrc head. Installing Kyua is now a breeze under NetBSD (and possibly under other platforms supported by pkgsrc!)

July 10, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Belated update:

July 5, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This has been the big week:

• Wrote some user documentation for the kyua binary.
• Fixed some distcheck problems.
• Released Kyua 0.1!

June 26, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua 0.1 released!

Dear readers,

I'm very proud to announce that the very first release of Kyua, obviously numbered 0.1, has been published! The release page for kyua-cli-0.1 contains more details about what this release offers, and don't hesitate to join the kyua-discuss mailing list and ask if you have questions.

Kyua was started sometime in past October and it has taken over six months to get the first public version. (It is true that my free time has heavily fluctuated during this period though, so it does not mean that six months of intensive coding have gone into the release ;-) Kyua 0.1 hovers at almost 40K lines of code according to Ohloh and comes with 800 test cases, yet it only implements a replacement for the atf-run component of ATF.

I have a package ready for pkgsrc, but unfortunately can't submit it because pkgsrc is in a freeze in preparation for the next stable branch. No matter what, expect detailed instructions on how to install Kyua and how to run the NetBSD-current test suite with it very soon.

Enjoy, and keep the bug reports coming!

June 23, 2011 · Tags: kyua, release
Continue reading (about 1 minute)

Kyua: Weekly status report

A couple of things have happened:

• Released ATF 0.14. This release was followed by an import into NetBSD and fixing of subsequent fallout.
• Some performance improvements to atf-sh. After killing a bunch of complex shell constructions and removing lots of obsolete functions, the performance results are significant. There is still room for improvement of course, and I still need to quantify how these optimizations behave in single-core machines.

June 19, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

June 13, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Some long-standing bug fixes / improvements have gone in this week:

• Improvements to the cleanup routine, which is used to destroy the work directory of a test case after the test case has terminated:
• Heavy refactoring to be tolerant to failures. These failures may arise when a child of the test case does not exit immediately and holds temporary files in the work directory open for longer than expected.
• Any file systems that the test case leaves mounted within the work directory will now be unmounted, just as the ATF test interface mandates. I realize that this adds a lot of complexity to the runtime engine for very little gain. If/when we revise the tests interface, it will be worth to reconsider this and maybe leave the cleanup of mounted file systems to the test case altogether.
• As a result, issue 17 has been fixed!
• Kyua now captures common termination signals (such as SIGINT) and exits in a controlled manner. What this means is that Kyua will now kill any active test programs and clean up any existing work directories before exiting. What this also means is that issue 4 is fixed.

June 5, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

Some cool stuff this week, albeit not large-scale:

• Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.
• Added support for several environment variables to allow users (and tests) to override built-in paths. I can't imagine right now any legitimate use for these variables, but hardcoded values are bad in general, atf-run provided these same variables, and these variables are very handy for testing purposes.
• Added support for the new require.files test-case metadata property to both ATF and Kyua. This new property allows tests to specify a set of files that they require in order to run, and is useful for those tests that can't run before make install is executed.
  

May 29, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This week:

May 22, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report, BSDCan 2011 edition

I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers.

Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them. (Video coming soon, I hope.)

The talk was later followed by several questions and off-session conversations about testing in general in BSDs. Among these, I gathered a few random ideas / feelings:

• The POSIX 1003.3 standard defines the particular results a test can emit (see the corresponding DejaGnu documentation). Both ATF and Kyua already implement all the results defined in the standard, but they use different names and extend the standard with many extra results. Given that the standard does not define useful concepts like "expected failures", an idea that came up is to provide a flag to force POSIX compliance at the cost of being less useful. Why? Just for the sake of saying that Kyua conforms to this standard.
• The audience seemed to like the idea of a "tests results store" quite a bit, and the sound of SQLite for the implementation was not bullied. This is something I'm eager to work on, but not before I publish a 0.1 release.
• I highlighted the possibility of allowing Kyua to run "foreign" test programs so that we could integrate the results into the database. This could be useful to run tests for which we (*BSD) have no control (e.g. gcc) in an integrated manner. The idea was not bullied by anyone either.
• FreeBSD has already been looking at ATF / Kyua and they are open to collaboration.
• OpenBSD won't import any new C++ code, and adding C-based tests to the tree while relegating the C++ runtime to the ports is not an option. Somehow I expected this.
• Junos (the FreeBSD-based operating system from Jupiter Networks) recently imported ATF and they are happy with it so far. Yay!
• Would be nice to have a feature to run tests remotely after, maybe, deploying a single particular test and its dependencies. This is gonna be tricky and not in my current immediate plans.

May 16, 2011 · Tags: atf, conference, kyua, report
Continue reading (about 3 minutes)

Kyua: Weekly status report

Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow.

May 8, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, I have had no time for coding this week. The only things I could do were:

May 1, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Ouch; I'm exhausted. I just finished a multi-hour hacking session to get the implementation of the list subcommand in control. It is now in a very nice user-facing shape, although its code deserves a little bit of house cleaning (coming soon).

Anyway, this week's progress:

• Added the kyuaify.sh script. This little tool takes a test suite (say, NetBSD's /usr/tests directory) and converts all its Atffiles into Kyuafiles. The tool is no sophisticated at all; in fact, it is a pretty simple script that I haven't tested with any other test suites so far. See the announcement of kyuaify for some extra details.
• Added logging support for the Lua code and changed the Lua modules to spit some logging information while processing Kyuafiles and configuration files.
• Added a mechanism in the user interface module to consistently print informational, warning and error messages.
• Implemented proper test filtering (after several iterations). What does proper mean? Well, for starters, test filters are always relative to the test suite's root (although we already saw this in last week's report). But the most important thing is that the filters are now validated: nice, user-friendly errors will be reported when the collection of tests is non-disjoint, when it includes duplicate names or when any of the provided filters does not match any test case. I really need to document the rationale of these in the manual, but for now the r118 commit message includes a few details.
• Drafted some notes for the BSDCan 2011 conference. I am quite tempted to reuse parts of the presentation from NYCBSDCon 2010, but I really want to give more emphasis on Kyua this time. In case you don't know, Kyua was first announced in NYCBSDCon 2010 and it was still a very immature project. The project has changed a lot since then.

April 24, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Added integration tests for the about and help subcommands. These were pretty easy to do.
• Added integration tests for the list subcommand. I initially added these tests as expected failures to reason about the appearance and behavior of this command from the point of view of the user before actually working on the code... and I am still writing such code to make these tests pass!

April 17, 2011 · Tags: atf, kyua, report
Continue reading (about 3 minutes)

Kyua: Weekly status report

Few things worth mentioning this week as reviewing Summer of Code student applications has taken priority. The good thing is that there are several strong applications for NetBSD; the bad thing is that none relate directly to testing. Anyway, the work this week:

• Added a pkg-config file for atf-sh as well as an Autoconf macro to detect its presence. This is needed by Kyua to easily find atf-sh. (Yes, I know: this is an abuse of pkg-config, but it works pretty well and is consistent with atf-c and atf-c++.)
• Implemented basic integration tests for Kyua in r98 using atf-sh. These tests are still very simple but provide a placeholder into which new tests will be plugged. Having good integration test coverage is key in preparation for a 0.1 release. Oh, and by the way, this revision has bumped the number of tests to 601, crossing the 600 barrier :-)

April 10, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This week's work has been quite active on the ATF front but not so much in the Kyua one. I keep being incredibly busy on the weekends (read: traveling!) so it's hard to get any serious development work done.

What has happened?

• Finally tracked down and fixed some random atf-run crashes that had been hunting the NetBSD test suite for months (see PR bin/44176). The fix is in reality an ugly workaround for the fact that a work directory cannot be considered "stable" even after the test case terminates. There may be dying processes around that touch the work directory contents, and the cleanup code in atf-run was not coping well with those. As it turns out, this problem also exists in Kyua (even though it's not as pronounced because arbitrary failures when running a test case do not crash the runtime engine) so I filed issue 17 to address it.
• Released ATF 0.13 and imported it both to NetBSD-current and pkgsrc. As a side note, Kyua requires the new features in this release, so putting it out there is a requirement to release Kyua 0.1. This new release does not have a big effect on NetBSD though, because the copy of ATF in NetBSD has been constantly receiving cherry-picks of the upstream fixes.
• Replaced several TODO items in the Kyua code with proper calls to the logging subsystem. These TODO items were referring to conditions in the code that should not happen, but for which we cannot do any proper recovery (like errors in a destructor). Sure, these could be better signaled as an assertion... but these code paths can be triggered in extremely-tricky conditions and having Kyua crash because of them is not nice (particularly when the side-effects of executing that code paths are non-critical).

April 3, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

This has been a slow week. In the previous report, I set the goal of getting Kyua to run the NetBSD test suite accurately (i.e. to report the same results as atf-run), and this has been accomplished. Actually, the changes required in Kyua to make this happen were minimal, but I got side-tracked fixing issues in NetBSD itself (both in the test suite and in the kernel!). So, the things done:

• Fixed Kyua to correctly kill any dangling subprocesses of a test case and thus match the behavior of atf-run. This was the only change required to make issue 16 happen: i.e. to get Kyua to report the same results as atf-run for the NetBSD test suite.
• Based on a suggestion from Antti Kantee, an alternative way to handle this would be to not kill any processes and just report the test case as broken if it fails to clean itself up. The rationale being that the runtime engine can kill dangling subprocesses in 99% of the occasions, but not always. The exception are those subprocesses that change their process group. It'd be better to make all cleanups explicit instead of hiding this corner case, as it can lead to confusion. Addressing this will have to wait though, as it is a pretty invasive change.
• Before closing issue 16, I want to implement some integration tests for Kyua to ensure that the whole system behaves as we expect (which is what the NetBSD test suite is currently doing implicitly).
• Kyua is pickier than atf-run: if a cleanup routine of a test case fails or crashes, Kyua will (correctly) report the test case as broken while atf-run will silently ignore this situation. Some NetBSD tests had crashing cleanup parts, so I fixed them.
• Some test programs in NetBSD were leaving unkilled subprocesses behind. These subprocesses are daemons and thus fall out of the scope of what Kyua can detect and kill during the cleanup phase. I mistakenly tracked down the problem to rump, but Antti Kantee kindly found the real problem in the kernel (not in rump!).
• As a side effect of processes being left behind, I extended the functionality of pidfile(3) and implemented pid file support in bozohttpd.  This is to make the tests that spawn a bozohttpd in the background more robust, by giving them a way to forcibly kill the server during cleanup.
• These changes are still under review and not committed yet.

March 27, 2011 · Tags: atf, kyua, report
Continue reading (about 3 minutes)

Kyua: Weekly status report

These days, I find myself talking about Kyua to "many" people. In particular, whenever a new feature request for ATF comes in, I promise the requester that the feature will be addressed as part of Kyua. However, I can imagine that this behavior leaves the requester with mixed feelings: it is nice that the feature will be implemented but, at the same time, it is very hard to know when because the web site of Kyua does not provide many details about its current status.

In an attempt to give Kyua some more visibility, I will start posting weekly activity reports in this blog. These reports will also include any work done on the ATF front, as the two projects are highly related at this point. I write these reports regularly at work and I feel like it is a pretty good habit: every week, you have to spend some time thinking about what you did for the project and you feel guilty if the list of tasks is ~zero ;-) It also, as I said, gives more visibility to the work being done so that outsiders know that the project is not being ignored.

Before starting with what has happened this week, a bit of context. I have been traveling like crazy and hosting guests over for the last 2 months. This has given me virtually no time to work on Kyua but, finally, I have got a chance to do some work this past week.

So, what are the news?

• Implemented the --loglevel command line flag, which closes issue 14. Kyua now generates run-time logs of its internal activity to aid in postmortem debugging and this flag allows the user to control the verbosity of such logs.
• Antti Kantee hacked support for atf-run in the NetBSD source tree to dump a stack trace of any crashing test program. I have backported this code to the upstream ATF code and filed issue 15 to implement this same functionality in Kyua.
• Fixed a hang in atf-run that made it get stuck when a test case spawned a child processes and atf-run failed to terminate it. A quick test seems to indicate that Kyua is affected by a similar problem: it does not get stuck but it does not correctly kill the subprocesses. The problem will be addressed as part of issue 16.
• Oh, and by the way: Kyua will be presented at BSDCan 2011.

March 20, 2011 · Tags: atf, kyua, report
Continue reading (about 3 minutes)

Injecting C++ functions into Lua

The C++ interface to Lua implemented in Kyua exposes a lua::state class that wraps the lower-level lua_State* type. This class completely hides the internal C type of Lua to ensure that all calls that affect the state go through the lua::state class.

Things get a bit messy when we want to inject native functions into the Lua environment. These functions follow the prototype represented by the lua_CFunction type:

typedef int (*lua_CFunction)(lua_State*);

int
awesome_native_function(lua_State* state)
{
    // Uh, we have access to s, so we bypass the lua::state!
    ... do something nasty ...

    // Oh, and we can throw an exception here...
    //with bad consequences.
}

void
setup(...)
{
    lua::state state;
    state.push_c_function(awesome_native_function);
    state.set_global("myfunc");
    ... run some script ...
}

typedef int (*cxx_function)(lua::state&)

template< cxx_function Function >
int
wrap_cxx_function(lua_State* state)
{
    try {
        lua::state state_wrapper(state);
        return Function(state_wrapper);
    } catch (...) {
        luaL_error(state, "Geez, don't go into C's land!");
    }
}

int
awesome_native_function(lua::state& state)
{
    // See, we cannot access lua_State* now.

    ... do something ...
    throw std::runtime_error("And we can even do this!");
}

void
setup(...)
{
    lua::state state;
    state.push_c_function(
        wrap_cxx_function< awesome_native_function >);
    state.set_global("myfunc");
    ... run some script ...
}

January 17, 2011 · Tags: cxx, kyua, lua
Continue reading (about 2 minutes)

Error handling in Lua: the Kyua approach

About a week ago, I detailed the different approaches I encountered to deal with errors raised by the Lua C API. Later, I announced the new C++ interface for Lua implemented within Kyua. And today, I would like to talk about the specific mechanism I implemented in this library to deal with the Lua errors.

The first thing to keep in mind is that the whole purpose of Lua in the context of Kyua is to parse configuration files. This is an infrequent operation, so high performance does not matter: it is more valuable to me to be able to write robust algorithms fast than to have them run at optimal speed. The other key point to consider is that I want Kyua to be able to use prebuilt Lua libraries, which are built as C binaries.

The approach I took is to wrap every single unsafe Lua C API call in a "thin" (FSVO thin depending on the case) wrapper that gets called by lua_pcall. Anything that runs inside the wrapper is safe to Lua errors, as they are caught and safely reported to the caller.

Lets examine how this works by taking a look at an example: the wrapping of lua_getglobal. We have the following code (copy pasted from the utils/lua/wrap.cpp file but hand-edited for publishing here):

static int
protected_getglobal(lua_State* state)
{
    lua_getglobal(state, lua_tostring(state, -1));
    return 1;
}

void
lua::state::get_global(const std::string& name)
{
    lua_pushcfunction(_pimpl->lua_state, protected_getglobal);
    lua_pushstring(_pimpl->lua_state, name.c_str());
    if (lua_pcall(_pimpl->lua_state, 1, 1, 0) != 0)
        throw lua::api_error::from_stack(_pimpl->lua_state,
            "lua_getglobal");
}

January 14, 2011 · Tags: cxx, kyua, lua
Continue reading (about 3 minutes)

C++ interface to Lua for Kyua

Finally! After two weeks of holidays work, I have finally been able to submit Kyua's r39: a generic library that implements a C++ interface to Lua. The code is hosted in the utils/lua/ subdirectory.

From the revision description:

The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.

January 8, 2011 · Tags: cxx, kyua, lua
Continue reading (about 2 minutes)

Using RAII to clean up temporary values from a stack

For the last couple of days, I have been playing around with the Lua C API and have been writing a thin wrapper library for C++. The main purpose of this auxiliary library is to ensure that global interpreter resources such as the global state or the execution stack are kept consistent in the presence of exceptions — and, in particular, that none of these are leaked due to programming mistakes when handling error codes.

To illustrate this point, let's forget about Lua and consider a simpler case. Suppose we lost the ability to pass arguments and return values from functions in C++ and all we have is a stack that we pass around. With this in mind, we could implement a multiply function as follows:

void multiply(std::stack< int >& context) {
    const int arg1 = context.top();
    context.pop();
    const int arg2 = context.top();
    context.pop();
    context.push(arg1 * arg2);
}

std::stack< int > context;
context.push(5);
context.push(6);
multiply(context);
const int result = s.top();
s.pop();

void magic(std::stack< int >& context) {
    const int arg1 = context.top();
    context.pop();
    const int arg2 = context.top();
    context.pop();

    context.push(arg1 * arg2);
    context.push(arg1 / arg2);
    try {
        ... do something with the two values on top ...

        context.push(arg1 - arg2);
        try {
            ... do something with the three values on top ...
        } catch (...) {
            context.pop();  // arg1 - arg2
            throw;
        }
        context.pop();
    } catch (...) {
        context.pop();  // arg1 / arg2
        context.pop();  // arg1 * arg2
        throw;
    }
    context.pop();
    context.pop();
}

class temp_stack {
    std::stack< int >& _stack;
    int _pop_count;

public:
    temp_stack(std::stack< int >& stack_) :
        _stack(stack_), _pop_count(0) {}

    ~temp_stack(void)
    {
        while (_pop_count-- > 0)
            _stack.pop();
    }

    void push(int i)
    {
        _stack.push(i);
        _pop_count++;
    }
};

void magic(std::stack< int >& context) {
    const int arg1 = context.top();
    context.pop();
    const int arg2 = context.top();
    context.pop();

    temp_stack temp(context);

    temp_stack.push(arg1 * arg2);
    temp_stack.push(arg1 / arg2);
    ... do something with the two values on top ...

    temp_stack.push(arg1 - arg2);
    ... do something with the three values on top ...

    // Yes, we can return now.  No need to do manual pop()s!
}

December 27, 2010 · Tags: c, cxx, kyua, lua
Continue reading (about 6 minutes)

Kyua: Design of the configuration system

Over a week ago, I mostly finished the implementation of the runtime engine for test cases of Kyua and, along the way, realized that it is imperative to write a configuration system right now before the configuration code becomes messier than it already is.

To that end, I spent the last week working on a design document for the configuration system. Summarizing, the document describes what the requirements for the configuration files of Kyua are, what the possible alternatives to implement them are, and advocates the use of Lua — a tiny embedded programming language — to bring these configuration files to life.

It is your chance to get involved in the early stages of the development of Kyua! :-) Take a look at the email to kyua-discuss asking for comments and feel free to join the mailing list ;-)

December 22, 2010 · Tags: kyua, lua
Continue reading (about 1 minute)

Introducing Kyua

Wow. I have just realized that I have not blogged at all about the project that has kept me busy for the past two months! Not good, not good. "What is this project?", I hear. Well, this project is Kyua.

A bit of background first: the Automated Testing Framework, or ATF for short, is a project that I started during the Summer of Code of 2007. The major goal of ATF was, and still is, to provide a testing framework for the NetBSD operating system. The ATF framework is composed of a set of libraries to aid in the implementation of test cases in C, C++ and shell, and a set of tools to ease the execution of such test cases (atf-run) and to generate reports of the execution (atf-report).

At that point in time, I would say that the original design of ATF was nice. It made test programs intelligent enough to execute their test cases in a sandboxed environment. Such test programs could be executed on their own (without atf-run) and they exposed the same behavior as when they were run within the runtime monitor, atf-run. On paper this was nice, but in practice it has become a hassle. Additionally, some of these design decisions mean that particular features (in particular, parallel execution of tests) cannot be implemented at all. At the end of 2009 and beginning of 2010, I did some major refactorings to the code to make the test programs dumber and to move much of the common logic into atf-run, which helped a lot in fixing the major shortcomings encountered by the users... but the result is that, today, we have a huge mess.

Additionally, while ATF is composed of different modules conceptually separate from each other, there is some hard implementation couplings among them that impose severe restrictions during development. Tangentially, during the past 2 years of working at Google (and coding mainly in Python), I have been learning new neat programming techniques to make code more testable... and these are not followed at all by ATF. In fact, while the test suite of ATF seems very extensive, it definitely is not: there are many corner cases that are not tested and for which implementing tests would be very hard (which means that nasty bugs have easily sneaked in into releases).

Lastly, a very important point that affects directly the success of the project. Outsiders that want to contribute to ATF have a huge entry barrier: the source repository is managed by Monotone, the bug tracker is provided by Gnats (a truly user-unfriendly system), and the mailing lists are offered by majordomo. None of these tools is "standard" by today's common practices, and some of them are tied to NetBSD's hosting which puts some outsiders off.

For all the reasons above and as this year has been moving along, I have gotten fed up with the ATF code base. (OK, things are not that bad... but in my mind they do ;-) And here is where Kyua comes into the game.

Kyua is a project to address all the shortcomings listed above. First of all, the project uses off-the-shelf development tools that should make it much, much easier for external people to contribute. Secondly, the project intends to be much more modular, providing a clear separation between the different components and providing code that is easily testable. Lastly, Kyua intends to remain compatible with ATF so that there are no major disruptions for users. You can (and should) think of Kyua as ATF 2.0, not as a vastly different framework.

As of today, Kyua implements a runtime engine that is on par, feature-wise, to the one provided by atf-run. It is able to run test cases implemented with the ATF libraries and it is able to test itself. It currently contains 355 test cases that run in less than 20 seconds. (Compare that to the 536 test cases of ATF, which take over a minute to run, and Kyua is still really far from catching up with all the functionality of ATF.) Next actions involve implementing reports generation and configuration files.

Anyway. For more details on the project, I recommend you to read the original posting to atf-devel or the project's main page and wiki. And of course, you can also download the preliminary source code to take a look!

Enjoy :-)

December 16, 2010 · Tags: atf, kyua, netbsd
Continue reading (about 4 minutes)