Posts: Kyua

Showing 81 posts

Kyua graduates

After years of inactivity, the Kyua project has graduated as an open source citizen and has a new home under the FreeBSD umbrella! But uh… wait, what is Kyua and why is this exciting? To resolve confusion and celebrate this milestone, I’d like to revisit what Kyua is, how it came to be, why I stopped working on it for a while, why that was a problem for FreeBSD—and, indirectly, NetBSD—and how Kyua being free software has helped keep it alive.

August 2, 2024 · Tags: <a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about 14 minutes)

Offloading maintenance tasks to Travis CI

From day one, the Kyua source tree has had docstring annotations for all of its symbols. The goal of such docstrings is to document the code for the developers of Kyua: these docstrings were never intended to turn into pre-generated HTML documentation because Kyua does not offer an API once installed. As you might have noticed, Doxygen is an optional component of the build and it used to run on each make invocation. This changed “recently”. Nowadays, Doxygen is only run asynchronously on Travis CI to report docstring inconsistencies post-submission (see the DO=apidocs matrix entry if you are impatient). Combined with feature branches that are only merged into master when green, this is as good as the previous approach of running Doxygen along the build. Scratch that: this is even better because running Doxygen locally on each build took significant resources and penalized edit/build/test cycles.

May 21, 2015 · Tags: <a href="/tags/automation">automation</a>, <a href="/tags/ci">ci</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/travis">travis</a>
Continue reading (about 8 minutes)

Kyua turns parallel

After three months of intensive work on Kyua's executor Git branch, I am happy to announce that the new execution engine, whose crown feature is the ability to run test cases in parallel, has just landed in master and passes all self-tests! You can head over to the commit message for more details on the merge, read the NEWS entries, and skim throught the history of the executor branch to understand how this feature has been built.

February 28, 2015 · Tags: <a href="/tags/announce">announce</a>, <a href="/tags/kyua">kyua</a>
Continue reading (about 5 minutes)

Refocusing Kyua maybe?

The FreeBSD devsummit that just passed by gave me enough insight into Jenkins to question the long-term plans for Kyua. Uh, WHAT?! Let me explain. In the beginning... One of the original but unstated goals of Kyua was to fix the "mess" that is the NetBSD releng test run logs site: if you pay close attention, you will notice that various individuals have reinvented the wheel over and over again in an attempt to automate release builds and test suite runs. In other words: different parties have implemented independent continuous integration systems several times with more or less success.

May 23, 2014 · Tags: <a href="/tags/freebsd">freebsd</a>, <a href="/tags/jenkins">jenkins</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about 6 minutes)

How to merge multiple Git repositories into one

Are you looking for a method to merge multiple Git repositories into a single one? If so, you have reached the right tutorial! Please bear with me for a second while I provide you with background information and introduce the subject of our experiments. We’ll get to the actual procedure soon and you will be able to apply it to any repository of your choice. In the Kyua project, and with the introduction of the kyua-atf-compat component in the Summer of 2012, I decided to create independent Git repositories for each component. The rationale was that, because each component would be shipped as a standalone distfile, they ought to live in their own repositories.

February 15, 2014 · Tags: <a href="/tags/featured">featured</a>, <a href="/tags/git">git</a>, <a href="/tags/kyua">kyua</a>
Continue reading (about 4 minutes)

FreeBSD Test Suite tutorial at AsiaBSDCon 2014

I am pleased to announce that the tutorial on the FreeBSD Test Suite that I proposed for AsiaBSDCon 2014 has been accepted! The conference website will soon include more details, but allow me to spoil your wait: Goals: Learn how to use the test suite, how it is internally organized and how new tests can be written. Stretch goal: Get attendees to contribute one or more tests to the project. Audience: Mostly developers of FreeBSD that want to learn how the new test suite plugs into the system. That said, and because a major part of the tutorial will revolve around using the test suite for one's own benefit, everyone is welcome really. In particular, system administrators may get a useful tool out of this. The main surprise in the acceptance confirmation email is that materials are due by January 20th... which is around the corner! Time to rush in getting things ready. In the meantime, you can find more details on the tutorial by reading the proposal itself.

January 8, 2014 · Tags: <a href="/tags/conference">conference</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/testing">testing</a>
Continue reading (about 1 minute)

FreeBSD Test Suite goals and planning

I was really impressed and delighted to see how popular my previous (and first) post on FreeBSD testing, titled Introducing the FreeBSD Test Suite, was. Looks like this project may be of great interest to developers and users out there (not unsurprisingly) so I'll try to keep you all up-to-date with any key developments. A first question that arises from the announcement is: where are the test suite and infrastructure headed? After all, the continuous testing machines for amd64 are already up and running, so what else is there to do?

January 5, 2014 · Tags: <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/testing">testing</a>
Continue reading (about 2 minutes)

Introducing the FreeBSD Test Suite

I joined the FreeBSD committer ranks a couple of months ago with the intention to equip FreeBSD with an out-of-the-box test suite and with a testing infrastructure. The time until now has been quite fruitful and I have been rushing to get something ready for you before the year end. With that, I am very pleased to announce that the first mockup of the FreeBSD testing cluster is up and running! Point your browser at:

December 31, 2013 · Tags: <a href="/tags/featured">featured</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/testing">testing</a>
Continue reading (about 4 minutes)

lutok-0.3, kyua-testers-0.2 and kyua-cli-0.8

Yesterday was release day: I pushed out Lutok 0.3, Kyua Testers 0.2 and Kyua CLI 0.8. There are not a lot of changes in these new releases. The reason I cut them was to publish the new TAP-compliant tester and make it available for use in FreeBSD as soon as possible. I will be using this new feature as part of the FreeBSD Test Suite in order to hook existing test programs without having to rewrite them to use the ATF libraries (or at least not as a first step).

December 8, 2013 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/lutok">lutok</a>, <a href="/tags/release">release</a>
Continue reading (about 2 minutes)

Three productive days on the Kyua front

This being Thanksgiving week in the U.S. and Google giving us Thursday and Friday off, I decided to take Monday to Wednesday off as well to spend some time hacking on Kyua — yes, finally, after months of being inactive. And what a three productive days! Here comes a little briefing on the three fronts in which I made progress. (This puts on hold the header files series until next Monday... but most of you are probably away anyway. Enjoy the holidays if they apply to you!)

November 28, 2013 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/fedora">fedora</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 6 minutes)

EuroBSDCon 2013 takeaways

EuroBSDCon 2013 is done. If you have been following my daily posts over the last 4 days (day 1, day 2, day 3 and day 4) as well as #EuroBSDCon updates in Twitter, you may already have a pretty good idea of what went on here. However, with the conference over, it is now a good time to recap the whole event and present the takeaways of these four days which, overall, were quite interesting and productive.

September 30, 2013 · Tags: <a href="/tags/eurobsdcon">eurobsdcon</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about 6 minutes)

Live from EuroBSDCon 2013, day 4

Live from Malta today attending the EuroBSDCon 2013 conference. The conference is over; today was the second and last day and it has just finished. Hardware and virtualization One of the three tracks today included a lot of talks on hardware, porting of BSDs to new hardware and virtualization techniques. Of all these, the few talks I attended covered the topics in great detail and proved to be very interesting.

September 29, 2013 · Tags: <a href="/tags/conference">conference</a>, <a href="/tags/eurobsdcon">eurobsdcon</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/virtualization">virtualization</a>
Continue reading (about 4 minutes)

Live from EuroBSDCon 2013, day 3

Live from Malta today attending the EuroBSDCon 2013 conference. Today is the first day of the conference itself. Many more people have shown up as expected and there have been tons of very interesting talks all the time. It is both good and bad that there are several tracks: you can select the topic you are most interested in, but sometimes great talks overlap! Keynote Today's opening session was led by Theo de Raadt, the founder of OpenBSD. His keynote focused on explaining how there is no real research happening on operating systems any more and how new, risky technological changes can be tested in a real-world system like OpenBSD.

September 28, 2013 · Tags: <a href="/tags/conference">conference</a>, <a href="/tags/eurobsdcon">eurobsdcon</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about 4 minutes)

Setting up my old Mac Mini G4 as a development machine

I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below. I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it... which is great, but when thinking about it objectively, I am reluctant to unnecessarily "punish" all NetBSD users by increasing the build time of the system significantly. Don't get me wrong: Kyua itself runs efficiently on old machines, but building the code — particularly the hundreds of tests that come with it — takes just too long. This slowdown is not too noticeable on a relatively-modern machine, but it's unacceptable on not-so-old machines. Of course, I could proceed with the import right now (with the code disabled by default) and make it leaner later, but this would cause quite a first bad impression on our users. So it's better to delay the import a little bit until, at least, I have had a chance to simplify some core parts of the code. Mind you, this simplification work is already in progress and quite advanced: it consists on modularizing (as separate processes) some critical parts of the code and writing these with a much simpler style and in plain C. But back to the original point of my post. The first thing to mention about this experience is that with some effort and long waits, I've got a pretty decent setup for development even on this old machine. From time to time, I miss having a real Unix desktop at hand for development (no OS X, you are not one of those). The GUI behaves relatively well with a 1920x1200 display, running Fluxbox, traditional xterms, Mutt for GMail access and a bunch of other applications. Unfortunately, too many things feel really sluggish. A few specific examples: Firefox 16 is barely usable. I'm not sure there are many alternatives to decent web browsing for such an old non-Intel platform. Dillo is blazing fast and allows me to access online documentation and mailing list archives (just enough for development), but it's pretty much useless for any other purpose given the "Web 2.0".Any operation that involves pkgsrc takes ages. Even when building the simplest packages, one can notice the system crawl through all of the pkgsrc infrastructure. Sometimes this is the fault of a bad algorithm; other times it's just sh(1) being the wrong tool for something as complex as pkgsrc internals.Things like basic code editing in Emacs 24 are slow at responding to typing. Disabling font lock mode makes it feel fast again, but it's just surprising to see that even color coding is slow.I still remember my old and trusty machine from 10 years ago (a Pentium II 233 MHz): with a similar setup, it was significantly snappier. Yes, software has evolved and these packages now have many more features... but really, does editing a text file have to be sluggish? Leaving aside sluggishness, there is also the problem of instability. NetBSD/macppc is a tier 2 port, and things don't work as well as one would like. I personally would enjoy bringing this port to tier 1... but I currently lack the time (and basic knowledge of the architecture) to do so :-/ Anyway, the result of this exercise: the new code I'm writing to modularize Kyua builds damn fast in this machine, and keeping it this way is the whole point of having such an old machine as a build environment. So I'll try to keep using it to develop these new components of Kyua.

October 22, 2012 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/lab-notes">lab-notes</a>, <a href="/tags/mac">mac</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about 3 minutes)

Kyua gets its own blog

For the last couple of weeks, I have been pondering the creation of a Kyua-specific blog. And, after a lot of consideration, I have finally taken the plunge. Say hello to Engineering Kyua! From now on, all Kyua-related posts (as well as ATF posts) will go to the new blog. I recommend you to subscribe to Engineering Kyua's Atom feed right now to not miss a beat!  If you care enough about Kyua, that is... I may still post Kyua-related stuff in here once in a while, but you should assume that all news and, in particular, weekly status reports will be sent to the new blog. "Why?" Well, The Julipedia is supposed to be (and always has) my personal blog. Looking back at all the recent posts, they almost univocally are about Kyua and there is no personal content in them. In respect for the readers of this blog (who may not care about Kyua at all) and in order to attempt to give Kyua a more definite identity, it makes sense to move the posts to their own blog. Also, by having a blog dedicated to Kyua, I will not feel uncomfortable about publishing weekly status reports again. I previously felt that they were adding too much noise to this blog, and is the main reason behind why I stopped posting them at some point. Weekly reports have their value, mostly to keep myself focused and to allow outsiders to know what the project is up to (particularly in a world of DVCSs, where code changes may be kept private for weeks at a time). And you may wonder: "will you continue to post content here?" Sure I will, but I need ideas (suggestions welcome)! Today's social ecosystem makes it difficult for me to decide whether a post belongs in a blog, in Google+, in Twitter... and updating them all at once to provide the same content is pointless. Here is my take: for most of the irrelevant stuff that one may want to share at a personal level (photos, videos, arbitrary thoughts), social networks seem to provide a better platform. The blog seems a place more suited for short essays that should be indexable and be accessible by users across the web; for example, these include how-tos, technical explanations for a particular concept, or opinion articles. And, finally, Twitter seems like the place to throw pointers to longer articles elsewhere and very short opinion comments. I think this summarizes pretty well what my current "practices" around these systems follow. And, as you can deduce, this also explains (as you have experienced) why the blog gets fewer content than ever because most things are better suited for a social network.

June 4, 2012 · Tags: <a href="/tags/blog">blog</a>, <a href="/tags/kyua">kyua</a>
Continue reading (about 3 minutes)

Exposing a configuration tree through Lua

In the previous post, I discussed the type-safe tree data structure that is now in the Kyua codebase, aimed at representing the configuration of the program. In this post, we'll see how this data structure ties to the parsing of the configuration file. One goal in the design of the configuration file was to make its contents a simple key/value association (i.e. assigning values to predetermined configuration variables). Of course, the fact that the configuration file is just a Lua script means that additional constructions (conditionals, functions, etc.) can be used to compute these values before assignment, but in the end all we want to have is a collection of values for known keys. The tree data structure does exactly the latter: maintain the mapping of keys to values, and ensuring that only a set of "valid" keys can be set. But, as a data structure, it does not contain any of the "logic" involved in computing those values: that is the job of the script. Now, consider that we have the possible following syntaxes in the configuration file: simple_variable = "the value" complex.nested.variable = "some other value" These assignments map, exactly, to a tree::set() function call: the name of the key is passed as the first argument to tree::set() and the value is passed as the second argument. (Let's omit types for simplicity.) What we want to do is modify the Lua environment so that these assignments are possible, and that when such assignments happen, the internal tree object gets updated with the new values. In order to achieve this, the configuration library modifies the Lua environment as follows: The newindex metatable method of _G is overridden so that an assignment causes a direct call to the set method of the referenced key. The key name is readily available in the newindex arguments, so no further magic is needed. This handles the case of "a = b" (top-level variables).The index metatable method of _G is overridden so that, if the indexed element is not found, a new table is generated and injected into _G. This new table has a metatable of its own that performs the same operations as the newindex and index herein described. This handles the case of "a.b = c", as this trick causes the intermediate tables (in this case "a") to be transparently created.Each of the tables created by index has a "key" metatable field that contains the fully qualified key of the node the table corresponds to. This is necessary to be able to construct the full key to pass to the set method.There is further magic to ensure that values pre-populated in the tree (aka default values) can be queried from within Lua, and that variables can be set more than once. These details are uninteresting though.At the moment, we deny setting variables that have not been pre-defined in the tree structure, which means that if the user wants to define auxiliary variables or functions, these must be declared local to prevent calling into the _G hooks. This is quite nice, but we may need to change this later on if we want to export the standard Lua modules to the configuration files.

June 2, 2012 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 3 minutes)

Type-safe, dynamic tree data type

The core component of the new configuration library in Kyua is the utils::config::tree class: a type-safe, dynamic tree data type. This class provides a mapping of string keys to arbitrary types: all the nodes of the tree have a textual name, and they can either be inner nodes (no value attached to them), or leaf nodes (an arbitrary type attached as a value to them). The keys represent traversals through such tree, and do this by separating the node names with dots (things of the form root.inner1.innerN.leaf). The tree class is the in-memory representation of a configuration file, and is the data structure passed around methods and algorithms to tune their behavior. It replaces the previous config static structure. The following highlights describe the tree class: Keys are (and thus the tree layout is) pre-registered. One side-effect of moving away from a static C++ structure as the representation of the configuration to a dynamic structure such as a tree is that the compiler cannot longer validate the name of the configuration settings when they are queried. In the past, doing something like config.architecture would only compile if architecture was a valid structure defined... but now, code like config["architecture"] cannot be validated during the build. In order to overcome this limitation, trees must have their keys pre-defined. Pre-defining the keys declares their type within the tree.  Accesses to unknown keys results in an error right away, and accesses to pre-defined keys must always happen with their pre-recorded types. Note that pre-defined nodes can, or cannot, hold a value. The concept of "being set" is different than "being defined".Some nodes can be dynamic. Sometimes we do not know what particular keys are valid within a context. For example, the test_suites subtree of the configuration can contain arbitrary test suite names and properties within it, and there is no way for Kyua (at the moment) to know what keys are valid or not. As a result, the tree class allows defining a particular node as "dynamic", at which point accesses to any undefined keys below that node result in the creation of the node.Type safety. Every node has a type attached to it. The base configuration library provides common types such as bool_node, int_node and string_node, but the consumer can define its own node types to hold any other kind of data type. (It'd be possible, for example, to define a map_node to hold a full map as a tree leaf.) The "tricky" (and cool) part of type safety in this context is to avoid exposing type casts to the caller: the caller always knows what type corresponds to every key (because, remember, the caller had to predefine them!), so it knows what type to expect from every node. The tree class achieves this by using template methods, which just query the generic internal nodes and cast them out (after validation) to the requested type.Plain string representations. The end user has to be able to provide overrides to configuration properties through the command line... and the command line is untyped: everything is a string. The tree library, therefore, needs a mechanism to internalize strings (after validation) and convert them to the particular node types. Similarly, it is interesting to have a way to export the contents of a tree to strings so that they can be shown to the user.With that said, let's see a couple of examples. First, a simple one. Let's create a tree with a couple of fictitious nodes (one a string, one an integer), set some values and then query such values: config::tree tree; // Predefine the valid keys. tree.define< config::string_node >("kyua.architecture"); tree.define< config::int_node >("kyua.timeout"); // Populate the tree with some sample values. tree.set< config::string_node >("kyua.architecture", "powerpc"); tree.set< config::int_node >("kyua.timeout", 300); // Query the sample values. const std::string architecture = tree.lookup< config::string_node >("kyua.architecture"); const int timeout = tree.lookup< config::int_node >("kyua.timeout"); Yep, that's it. Note how the code just knows about keys and their types, but does not have to mess around with type casts nor tree nodes. And, if there is any typo in the property names or if there is a type mismatch between the property and its requested node type, the code will fail early. This, coupled with extensive unit tests, ensures that configuration keys are always queried consistently. Note that we'd also have set the keys above as follows: tree.set_string("kyua.architecture", "powerpc"); tree.set_string("kyua.timeout", "300"); ... which would result in the validation of "300" as a proper integer, conversion of it to a native integer, and storing the resulting number as the integer node it corresponds to. This is useful, again, when reading configuration overrides from the command line as types are not known in that context yet we want to store their values in the same data structure as the values read from the configuration file. Let's now see another very simple example showcasing dynamic nodes (which is a real-life example from the current Kyua configuration file): config::tree tree; // Predefine a subtree as dynamic. tree.define_dynamic("test_suites"); // Populate the subtree with fictitious values. tree.set< config::string_node >("test_suites.NetBSD.ffs", "ext2fs"); tree.set< config::int_node >("test_suites.NetBSD.iterations", 5); // And the querying would happen exactly as above with lookup(). Indeed, it'd be very cool if this tree type followed more standard STL conventions (iterators, for example). But I didn't really think about this when I started writing this class and, to be honest, I don't need this functionality. Now, if you paid close attention to the above, you can start smelling the relation of this structure to the syntax of configuration files. I'll tell you how this ties together with Lua in a later post. (Which may also explain why I chose this particular representation.)

May 29, 2012 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>
Continue reading (about 5 minutes)

Rethinking Kyua's configuration system

In the previous blog post, I described the problems that the implementation of the Kyua configuration file parsing and in-memory representation posed. I also hinted that some new code was coming and, after weeks of work, I'm happy to say that it has just landed in the tree! I really want to get to explaining the nitty-gritty details of the implementation, but I'll keep these for later. Let's focus first on what the goals for the new configuration module were, as these drove a lot of the implementation details:Key/value pairs representation: The previous configuration system did this already, and it is a pretty good form for a configuration file because it is a simple, understandable and widespread format. Note that I have not said anything yet about the types of the values.Tree-like representation: The previous configuration schema grouped test-suite specific properties under a "test_suites" map while it left internal run-time properties in the global namespace. The former is perfect and the latter was done just for simplicity. I want to move towards a tree of properties to give context to each of them so that they can be grouped semantically (e.g. kyua.report.*, kyua.runtime.*, etc.). The new code has not changed the structure of the properties yet (to remain compatible with previous files), but it adds very simple support to change this in the shortcoming future.Single-place parsing and validation: A configuration file is an external representation of a set of properties. This data is read (parsed) once and converted into an in-memory representation. All validation of the values of the properties must happen at this stage, and not when the properties are queried. The reason is that validation of external values must be consistent and has to happen in a controlled location (so that errors can all be reported at the same time). I have seen code in other projects where the configuration file is stored in memory as a set of key/value string pairs and parsing to other types (such as integers, etc.) is delayed until the values are used. The result is that, if a property is queried more than once, the validation will be implemented in different forms, each with its own bugs, which will result in dangerous inconsistencies.Type safety: This is probably the trickiest bit. Every configuration node must be stored in the type that makes most sense for its value. For example: a timeout in seconds is an integer, so the in-memory representation must be an integer. Or another example: the type describing the "unprivileged user" is a data structure that maps to a system user, yet the configuration file just specifies either a username or a UID. Keeping strict type validation in the code is interesting because it helps to ensure that parsing and validation happen in just a single place: whenever the configuration file is read, every property will have to be converted to its in-memory type, and this means that the validation can only happen at that particular time. Once the data is in memory, we can and have to assume that it is valid. Additionally, strict types ensure that the code querying such properties uses the values as intended, without having to do additional magic to map them to other types.Extensibility: Parsing a configuration file is a very generic concept, yet the previous code made the mistake of tying this logic with the specific details of Kyua configuration files. A goal of the new code has been to write a library that parses configuration files, and allows the Kyua-specific code to define the schema of the configuration file separately. (No, the library is not shipped separately at this point; it's placed in its own utils::config module.)With all this code in place, there are a bunch of things that can now be easily implemented. Consider the following:Properties to define the timeout of test cases depending on their size (long-standing issue 5).Properties to tune the UI behavior: width of the screen, whether to use color or not (no, there is no color support yet), etc.Properties to configure how reports look like "by default": if you generate reports of any form frequently, it is very likely that you will want them to look the same every time and hence you will want to define the report settings once in the configuration file.Hooks: one of the reasons for using Lua-based configuration files was to allow providing extra customization abilities to the user. Kyua could theoretically call back into Lua code to perform particular actions, and such actions could be explicitly stated by the user in the form of Lua functions. Neither the current configuration code nor Kyua has support for hooks, but the new implementation makes it rather easy to add them.And that's all for today. Now that you know what the current code is trying to achieve and why, we will be able to look at how the implementation does all this in the next posts.

May 28, 2012 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 4 minutes)

Kyua's configuration system showing its age

A couple of years ago, when Kyua was still a newborn, I wrote a very ad-hoc solution for the parsing and representation of its configuration files. The requirements for the configuration were minimal, as there were very few parameters to be exposed to the user. The implementation was quick and simple to allow further progress on other more-important parts of the project. (Yep, quick is an euphemism for dirty: the implementation of the "configuration class" has to special-case properties everywhere to deal with their particular types... just as the Lua script has to do too.) As I just mentioned in the previous paragraph, the set of parameters exposed through the configuration file were minimal. Let's recap what these are:Run-time variables: architecture and platform, which are two strings identifying the system; and unprivileged_user, which (if defined) is the name of the user under which to run unprivileged tests as. It is important to mention that the unprivileged_user is internally represented by a data type that includes several properties about a system user, and that it ensures that the data it contains is valid at all times. The fact that every property holds a specific type is an important design requirement.Test suite variables: every test suite can accept arbitrary configuration variables. Actually, these are defined by the test programs themselves. All of these properties are strings (and cannot be anything else because ATF test programs have no way of indicating the type of the configuration variables they accept/expect).Because of the reduced set of configurable properties, I opted to implement the configuration of the program as a simple data structure with one field per property, and a map of properties to represent the arbitrary test suite variables. The "parser" to populate this structure consists on a Lua module that loads these properties from a Lua script. The module hooks into the Lua metatables to permit things like "test_suites.NetBSD.timeout=20" to work without having to predeclare the intermediate tables. Unfortunately, as I keep adding more and more functionality to Kyua, I encounter additional places where a tunable would be appreciated by the end user (e.g. "disallow automatic line wrapping"). Exposing such tunable through a command-line flag would be a possibility, but some of these need to be permanent in order to be useful. It is clear that these properties have to be placed in the configuration file, and attempting to add them to the current codebase shows that the current abstractions in Kyua are not flexible enough. So, why am I saying all this? Well: during the last few weeks, I have been working on a new configuration module for Kyua. The goals have been simple:Have a generic configuration module that parses configuration files only, without any semantics about Kyua (e.g. what variables are valid or not). This ensures that the implementation is extensible and at the right level of abstraction.Be able to get rid of the ad-hoc parsing of configuration files.Allow defining properties in a strictly-typed tree structure. Think about being able to group properties by function, e.g. "kyua.host.architecture"; this is more or less what we have today for test-suite properties but the implementation is a special-case again and cannot be applied to other tunables.And... I am pleased to say that this code is about to get merged into the tree just in time for Kyua 0.4. In the next few posts, I will explain what the particular design constraints of this new configuration system were and outline a little bit its implementation. I think it's a pretty cool hack that mixes C++ data structures and Lua scripts in a "transparent" manner, albeit you may think it's too  complex. The key part is that, as this new configuration module is not specific to Kyua, you might want to borrow the code/ideas for your own use!

May 26, 2012 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>
Continue reading (about 3 minutes)

Kyua gets nicer console messages

For the last couple of weeks, particularly during a bunch of long flights, I have been improving the command-line user interface of Kyua by implementing controlled line wrappings on screen boundaries: messages that are too long to fit on the screen are preprocessed and split into multiple lines at word boundaries. This affects informational messages, error messages and, specially, the output of the built-in help command. I originally got this idea from Monotone and later implemented it into ATF but, when writing Kyua's code, I decided to postpone its implementation until a later stage. Reusing the code from ATF was not "nice" because the API of the formatting code was quite nasty, and reimplementing this feature during the initial stages of Kyua felt like a waste of time. However, because controlled line wrapping is crucial to having readable built-in help messages, I have had to do this eventually and the time finally came. The ATF approach Why did I say that the ATF code for line wrapping was quite nasty? The main reason is that the printing of messages was incredibly tied to their wrapping. All the code in ATF that prints a message to the screen has to deal with the line wrapping itself, which involves dealing with too many presentation details. For example, consider the help routine that prints the table of options and their descriptions. This routine has to calculate the width of the longest option first and then, for every option, output its name, output some padding, and output the description properly refilled so that subsequent lines are properly arranged with respect to the previous one. While this may not sound too bad in writing, it actually is in code. Furthermore, because all this formatting logic is spread out throughout the code, there is no way to perform decent unit testing. The unit testing did some basic tests on input text, but could not validate that more complex constructions were working right. The Kyua approach In the Kyua codebase, I decided to take a more declarative and functional approach. Smaller, pure building blocks that can be combined to achieve more complex constructions and that can be easily tested for correctness individually or in combination. I started by implementing a simple function that takes a paragraph and reformats it to any given length. This simple function alone gives full flexibility to the caller to decide how to later merge this reformatted text with other text: e.g. place a line prefix or bundle such text inside a table cell. The next step was to implement tables. Code wishing to print, e.g. the collection of options/commands along their descriptions only cares about declaring a table of two columns and N rows; why should it bother about properly lining up the two columns and printing them? It doesn't, hence the table approach. With tables, the caller can just decide which particular column needs to be wrapped if the table does not fit on the screen, and allow the formatting code to do this. Plus, having this higher level constructs means that we can eventually print the textual reports in a nicer, tabulated way (not done yet). And the last step was to mix all these higher level constructs into the console frontend class. This class (the ui) knows how to query the width of the terminal and knows how to fit certain kinds of text and/or tables within such width. For example, error messages are not tables: they are messages prefixed with the command name; only the message has to be reformatted if it does not fit while the rest of the text has to flow after the command name. Or, for tables, the maximum width of the terminal determines how wide the table can be and thus how much one of its columns has to be refilled. Getting this whole thing right working has proven to be extremely tricky and I'm sure there are still quite a few rough edges to be discovered. That said, it has been fun enough :-)  But, after this experience, I certainly don't want to imagine the pain that the writers of HTML/CSS renderers have endured... this text-based table-rendering is trivial compared to what web browsers do!

April 2, 2012 · Tags: <a href="/tags/kyua">kyua</a>
Continue reading (about 4 minutes)

Kyua generates its first public HTML report

Lately, three long trips (5 hours in a bus, and 6 and 10 hours in two planes) have allowed me to work on the long-promised HTML reporting feature of Kyua. The result of these three trips is, effectively, the ability to generate HTML reports for specific test actions! The current results are extremely rudimentary (they lack tons of would-be-useful information) and not that aesthetically pleasing. However, the database already records enough information to make these reports more useful and pretty, so "all that is left" is coming up with the necessary code to extract such information in an efficient way and spending time creating a visually-nicer appearance. None of these are as trivial as they sound, but I prefer to work one step at a time (i.e. coming up first with a very rough draft and improve it later) rather than keeping the feature private until it is "perfect". Without further ado, you can take a look at the report of the execution of the NetBSD test suite. This output comes from my NetBSD-current virtual machine, and I've set up a couple of cron jobs to keep it up to date. (If the "action X" in the title does not increase periodically, you will know that something is broken on my side. I found that the VM already crashed, so I don't know for how long it will run now after a restart...)

March 13, 2012 · Tags: <a href="/tags/kyua">kyua</a>
Continue reading (about 2 minutes)

Projects migrated to Git

I finally took the plunge. Yesterday night, I migrated the Kyua and Lutok repositories from Subversion to Git. And this morning I migrated ATF from Monotone and custom hosting to Git and Google Code; oh, and this took way longer than expected. Migration of Kyua and Lutok Migrating these two projects was straightforward. After preparing a fresh local Git repository following the instructions posted yesterday, pushing to Google Code is a simple matter: $ git remote add googlecode https://code.google.com/p/your-project $ git push googlecode --all $ git push googlecode --tags One of the nice things I discovered while doing this is that a Google Code project supports multiple repositories when the VCS system is set to Git or Mercurial. By default, the system creates the default and wiki repositories, but you can add more at will. This is understandable given that, in Subversion, you have the ability to check out individual directories of a project whereas you cannot do that in the other supported VCSs: you actually need different repositories to group different parts of the project. I performed the full migration under a Linux box so that I could avail of the most recent Git version along with a proven binary package. The migration went alright, but I encountered a little problem when attempting a fresh checkout from NetBSD: git under NetBSD will not work correctly against SSL servers because it lacks the necessary CA certificates. The solution is to install the security/mozilla-rootcerts package and follow the instructions printed during installation; why this does not happen automatically escapes my mind. Migration of ATF I had been having doubts about migrating ATF itself, although if Kyua was moved to Git, it was a prerequisite to move ATF as well.  Certainly I could convert the repository to Git, but where could I host it afterwards?  Creating a new Google Code project just for this seemed too much of a hassle. My illumination came when I found out, as above, that Google Code supports an arbitrary amount of repositories in a particular project when converting it to Git. So, for ATF, I just ran mtn git_export with appropriate flags, created a new atf repository on the Kyua site, and pushed the contents there. Along the way, I also decided to kill the home-grown ATF web site and replace it by a single page containing all the relevant information. At this point, ATF and Kyua are supposed to work together quite tightly (in the sense that ATF is just a "subcomponent" of Kyua), so coupling the two projects on the same site makes sense. Now, let's drink the kool aid.

February 26, 2012 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/git">git</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lutok">lutok</a>
Continue reading (about 3 minutes)

Kyua 0.3 released!

Dear readers, I am pleased to announce that Kyua 0.3 is available! The major feature in this release is the introduction of the "test results store"; i.e. a SQLite3-based database that records all test execution activity so that reports can be gathered in a central machine and reports generated out of it. This is still very experimental and the generated reports are quite rudimentary, but is a first step towards this direction. The Kyua 0.3 release page provides links to the download as well as the list of major changes in this new release. It has been a really long time since the 0.2 release. I wanted to postpone 0.3 until HTML reports were ready, but I have not had the time to implement this for a while already. Instead of postponing the release even further, I have decided to create it right now and then take the time to create nice reports separately. Additionally, I am planning on doing some repository changes and wanted to do them without an imminent release along the way. The package in pkgsrc has been updated and now I'm working on bringing binary packages to Fedora. Enjoy!

February 24, 2012 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/release">release</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

A couple of things happened this week: Spent quite a few time researching the idea of moving away from Monotone and Subversion to Git. I haven't made a decision yet, but I'm pretty convinced this is the right way to go. It will simplify development significantly, it will allow me to code offline (have a bunch of really long flights coming), and it will lower the entry barrier to Kyua by making all components use the same, mainstream VCS.Implemented filtering by result type of the test case results in the textual reports.I think it is time to prepare a 0.3 release. I wanted to wait until we had HTML reports in place, but this will require significant effort and I have been postponing the implementation for too long already. As it is now, the current codebase provides major changes since the ancient 0.2 release and it is worth a release. Then, we can create packages for NetBSD and Fedora instead of continuing to add new features, which should be a good step in giving further visibility to the project. Finally, we can reserve HTML reporting as the major feature for 0.4 :-)

February 13, 2012 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Switching projects to Git

The purpose of this post is to tell you the story of the Version Control System (VCS) choices I have made while maintaining my open source projects ATF, Kyua and Lutok. It also details where my thoughts are headed to these days. This is not a description of centralized vs. distributed VCSs, and it does not intend to be one. This does not intend to compare Monotone to Git either, although you'll probably feel like it while reading the text. Note that I have fully known the advantages of DVCSs over centralized systems for many years, but for some reason or another I have been "forced" to use centralized systems on and off. The Subversion hiccup explained below is... well... regrettable, but it's all part of the story! Hope you enjoy the read. Looking back at Monotone (and ATF) I still remember the moment I discovered Monotone in 2004: simply put, it blew my mind. It was clear to me that Distributed Version Control Systems (DVCSs) were going to be the future, and I eagerly adopted Monotone for my own projects. A year later, Git appeared and it took all the praise for DVCSs: developers all around started migrating en masse to Git, leaving behind other (D)VCSs. Many of these developers then went on to make Git usable (it certainly wasn't at first) and well-documented. (Note: I really dislike Git's origins... but I won't get into details; it has been many years since that happened.) One of the projects in which I chose to use Monotone was ATF. That might have been a good choice at the time despite being very biased, but it has caused problems over time. These have been: Difficulty to get Monotone installed: While most Linux distributions come with a Monotone binary package these days, it was not the case years ago. But even nowadays if all Linux distributions have binary packages, the main consumers of ATF are NetBSD users, and their only choice is to build their own binaries. This generates discomfort because there is a lot of FUD surrounding C++ and Boost.High entry barrier to potential contributors: It is a fact that Monotone is not popular, which means that nobody is familiar with it. Monotone's CLI is very similar to CVS, and I'd say the knowledge transition for basic usage is trivial, but the process of cloning a remote project was really convoluted until "recently". The lack of binary packages, combined with complex instructions on just how to fetch the sources of a project only help in scaring people away.Missing features: Despite years have passed, Monotone still lacks some important features that impact its usability. For example, to my knowledge, it's still not possible to do work-directory merges and, while the interactive merges offered by the tool seem like a cool idea, they are not really practical as you get no chance to validate the merge. It is also not possible, for example, to reference the parent commit of any given commit without looking at the parent's ID. (Yeah, yeah, in a DAG there may be more than one parent, but that's not the common case.) Or know what a push/pull operation is going to change on both sides of the connection. And key management and trust has been broken since day one and is still not fixed. Etc, etc, etc.No hosting: None of the major project hosting sites support Monotone. While there are some playground hosting sites, they are toys. I have also maintained my own servers sometimes, but it's certainly inconvenient and annoying.No tools support: Pretty much no major development tools support Monotone as a VCS backend. Consider Ohloh, your favorite bug tracking system or your editor/IDE. (I attempted to install Trac with some alpha plugin to add Monotone support and it was a huge mess.)No more active development: This is the drop that spills the cup. The developers of Monotone that created the foundations of the project left years ago. While the rest of the developers did a good job in coming up with a 1.0 release by March 2011, nothing else has happened since then. To me, it looks like a dead project at this point :-(Despite all this, I have been maintaining ATF in its Monotone repository, but I have felt the pain points above for years. Furthermore, the few times some end user has approached ATF to offer some contribution, he has had tons of trouble getting a fresh checkout of the repository and given up. So staying with Monotone hurts the project more than it helps. The adoption of Subversion (in Kyua) To fix this mess, when I created the Kyua project two years ago, I decided to use Subversion instead of a DVCS. I knew upfront that it was a clear regression from a functionality point of view, but I was going to live with it. The rationale for this decision was to make the entry barrier to Kyua much lower by using off-the-shelf project hosting. And, because NetBSD developers use CVS (shrugh), choosing Subversion was a reasonable choice because of the workflow similarities to CVS and thus, supposedly, the low entry barrier. Sincerely, the choice of Subversion has not fixed anything, and it has introduced its own trouble. Let's see why: ATF continues to be hosted in a Monotone repository, and Kyua depends on ATF. You can spot the problem, can't you? It's a nightmare to check out all the dependencies of Kyua, using different tools, just to get the thing working.As of today, Git is as popular, if not more, than Subversion. All the major operating systems have binary packages for Git and/or bundle Git in their base installation (hello, OS X!). Installing Git on NetBSD is arguably easier (at least faster!) than Subversion. Developers are used to Git. Or let me fix that: developers love Git.Subversion gets on the way more than it helps; it really does once you have experienced what other VCSs have to offer. I currently maintain independent checkouts of the repository (appropriately named 1, 2 and 3) so that I can develop different patches on each before committing the changes. This gets old really quickly. Not to mention when I have to fly for hours, as being stuck without an internet connection and plain-old Subversion... is suboptimal. Disconnected operation is key.The fact that Subversion is slowing down development, and the fact that it really does not help in getting new contributors more than Git would, make me feel it is time to say Subversion goodbye. The migration to Git At this point, I am seriously considering switching all of ATF, Lutok and Kyua to Git. No Mercurial, no Bazaar, no Fossil, no anything else. Git. I am still not decided, and at this point all I am doing is toying around the migration process of the existing Monotone and Subversion repositories to Git while preserving as much of the history as possible. (It's not that hard, but there are a couple of details I want to sort out first.) But why Git?First and foremost, because it is the most popular DVCS. I really want to have the advantages of disconnected development back. (I have tried git-svn and svk and they don't make the cut.)At work, I have been using Git for a while to cope with the "deficiencies" of the centralized VCS of choice. We use the squashing functionality intensively, and I find this invaluable to constantly and shamelessly commit incomplete/broken pieces of code that no-one will ever see. Not everything deserves being in the recorded history!Related to the above, I've grown accustomed to keeping unnamed, private branches in my local copy of the repository. These branches needn't match the public repository. In Monotone, you had this functionality in the form of "multiple heads for a given branch", but this approach is not as flexible as named private branches.Monotone is able to export a repository to Git, so the transition is easy for ATF. I have actually been doing this periodically so that Ohloh can gather stats for ATF.Lutok and ATF are hosted in Google Code, and this hosting platform now supports Git out of the box.No Mercurial? Mercurial looks a lot like Monotone, and it is indeed very tempting. However, the dependency on Python is not that appropriate in the NetBSD context. Git, without its documentation, builds very quickly and is lightweight enough. Plus, if I have to change my habits, I would rather go with Git given that the other open source projects I am interested in use Git.No Bazaar? No, not that popular. And the fact that this is based on GNU arch makes me cringe.No Fossil? This tool looks awesome and provides much more than DVCS functionality: think about distributed wiki and bug tracking; cool, huh? It also appears to be a strong contender in the current discussions of what system should NetBSD choose to replace CVS. However, it is a one-man effort, much like Monotone was. And few people are familiar with it, so Fossil wouldn't solve the issue of lowering the entry barrier. Choosing Fossil would mean repeating the same mistake as choosing Monotone.So, while Git has its own deficiencies — e.g. I still don't like the fact that it is unable to record file moves (heuristics are not the same) — it seems like a very good choice. The truth is, it will ease development by a factor of a million (OK, maybe not that much) and, because the only person (right?) that currently cares about the upstream sources for any of these projects is me, nobody should be affected by the change. The decision may seem a bit arbitrary given that the points above don't provide too much rationale to compare Git against the other alternatives. But if I want to migrate, I have to make a choice and this is the one that seems most reasonable. Comments? Encouragements? Criticisms?

February 11, 2012 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/git">git</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lutok">lutok</a>, <a href="/tags/monotone">monotone</a>, <a href="/tags/vcs">vcs</a>
Continue reading (about 8 minutes)

Kyua: Weekly status report

Created an RPM package for Lutok for inclusion in Fedora.Created a preliminary RPM spec for ATF for Fedora. Now in discussions with the FPC to figure out how to install the tests on a Fedora system, as /usr/tests may not be appropriate.No activity on Kyua itself though, unfortunately.

February 7, 2012 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lutok">lutok</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

This comes two days late... but anyway, I felt like posting it now instead of waiting until next Sunday/Monday. The activity past week focused mostly on implementing support for the require.memory test-case metadata property recently introduced into ATF. This was non-trivial due to the need to write some tricky Autoconf code to make this "slightly portable". Seriously: It's scary to see how hard it is to perform, in a portable manner, an operation as simple as "query the amount of physical memory"... but oh well, such are the native Unix APIs... I later spent the weekend on Lutok, preparing and publishing the project's first release and writing my first RPM library spec for it. This was a prerequisite for the upcoming 0.3 release of Kyua and thus deserved special attention!

February 1, 2012 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/lutok">lutok</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Released ATF 0.15 and imported it into NetBSD.Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them. While this sounds like a cool idea (yay, code reuse!), it resulted in a ugly, nasty and horrible mess. In the end, I just ended up implementing custom parsing of the formatters, which was way easier and "good enough" for Kyua's needs.Started work on backporting ATF's new require.memory property into Kyua. This needs having a way to parse and format byte quantities in user-friendly forms (e.g. 1k, 2m, etc.)... hence the previous work on utils::format!Set up a Google+ Page for Kyua. I have no idea what to use it for yet. Maybe the status reports should go in there. Ideas?

January 23, 2012 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Finally some progress! Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.Backported local pkgsrc patches to ATF into the upstream code.Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua. Not very successful yet unfortunately.Nothing on the Kyua front, but getting a new release of ATF out of the door has higher priority now!

January 15, 2012 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time!

January 10, 2012 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Happy new year! No activity past week as I was completely away from the computer.

January 3, 2012 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, not much activity this week due to travel reasons. Anyway, some work went in: Preliminary code to generate HTML reports from kyua report. This is easy peasy but boring. The current code was written as a proof of concept and is awful, hence why it was not committed. I'm now working in cleaning it up.Backported test program and test case timestamping into ATF based on a patch from Paul Goyette. This is a very useful feature to have, and it will have to be added to Kyua later. (It has always been planned to be added, but have not had the time yet.)

December 19, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Some significant improvements this week: Finally submitted the code to store and load full test case definitions. This is quite tricky (and currently very, very ugly) but it works and it will allow the reports to include all kinds of information from the test cases.Removed the Atffiles from the tree; yay! For a long time, I had been using atf-run to run broken tests because atf-run allowed me to watch the output of the test case being debugged. However, this has been unnecessary since the introduction of the debug command in late August. I now feel confident that these files can go. (And debug is much more powerful than atf-run because you can target a single test case instead of a whole test program.)Some crazy work attempting to hide the name of SQLite types from the sqlite::statement interface. I've been only able to do so somewhat decently for bind, but all my attempts at doing the same with column result in horrible code so far. So no, such changes have not been submitted.As of a few minutes ago, kyua test now records the output of the test cases (stdout and stderr) into the database. These will be invaluable for debugging of test cases, particularly when the reports are posted online.Some preliminary work at implementing HTML reports. This, however, has not received much progress due to the previous item requiring completion.I'm quite excited at this point. HTML reports are a few weeks away at most. Once that happens, it will be time to start considering replacing the atf-run / atf-report duo for good, particularly within NetBSD. This will certainly not be easy... but all the work that has gone into Kyua so far has this sole goal!

December 11, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

Some more work towards allowing storing and loading of full test case definitions (which in turn will allow us to provide detailed HTML reports). I have local changes that do this, but they are gated by the lack of some additional tests and probably some optimizations, because they slow down kyua report significantly. Regarding commits, I have only submitted some related cleanup-changes.I got distracted by invalid Doxygen comments and traced down why they were not being correctly validated (which is the whole point of running Doxygen during the build). It turns out I had to enable some extra EXTRACT_* settings in the Doxyfile. After doing so, I realized there are many, many documentation problems in the code... and I have been fixing them since. It's a tough operation, but I'm more than half-way through already. To give you an idea, the current diffstat shows about 650 new comment lines (!).

December 5, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes. Due to this, I ended up doing some cleanups. First, I reimplemented the way in which test programs that fail to list their test cases are represented. And second, I got rid of the useless test_case_id class, which exposed even further problems in the data types that represent test cases. It's now time to sit and think if the current representations of test programs and test cases make sense and, if not, how to better redo them. Not going to be easy, but I hope to have some time for this cleanup during this upcoming week.

November 28, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori. At the moment, the report is just some disorganized plain-text.  For example: $ kyua test[... wait for the tests to run ...]Committed action 82$ kyua report ===> Skipped tests [... too long to show ...] ===> Expected failures integration/cmd_report_test:output__console__change_file  ->     expected_failure: --output not implemented yet:     atf-check failed; see the output of the test for details ===> Failed tests store/transaction_test:put_test_case__ok  ->     failed: Line 663: stmt.step() not met ===> Summary Action: 82 Test cases: 934 total, 15 skipped, 1 expected failures,0 broken, 1 failed I'm now working on changing the database schema to be able to really store all the data about test cases, because at the moment I'm only storing their names. Once all the original data is stored, the report command will have lots more information to work with, and then will be the time to start improving the format of the reports. As mentioned earlier, having interactive HTML dashboards is high in the priority list, and a very important goal of Kyua altogether. Stay tuned :-)

November 20, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept. I have now started work on loading the data of test case results for a particular action, and once that is done, the report command will start yielding really useful data: i.e. it will actually tell you what happened during a particular execution of a test suite. The way I'm approaching the work these days is by building the skeleton code to implement the basic functionality first (which actually involves writing a lot of nasty code), with the goal of adding missing pieces later bit by bit. For example, at this moment I'm only targeting text-based outputs with a limited set of data. However, when that is done, adding extra data or different formats will be relatively easy. Generating HTML dashboards (without going through XML, as was the case of atf-report!) is definitely highly prioritized. By the way: I just realized it has already been one year since Kyua saw life. Wow, time flies. And only now we are approaching a point where killing the atf-run / atf-report pair is doable. I'm excited.

November 14, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database. "Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results. It's interesting to realize that most of these features are just one SQL query away. If you install Kyua, you can already run a few tests and then use kyua db-exec to issue arbitrary SQL queries against the database; the schema (see store/schema.sql) might look a bit convoluted, but a bunch of NATURAL JOINs will yield the desired output. The feature requests that have the highest priority at this point are the ability to generate a report of the last tests run both as a text file and as an HTML dashboard, because having these features means we can finally kill the atf-run and atf-report pair. At this point I'm, once again, "stuck" while figuring out how to best organize the code to make all these things possible while still keeping a nice separation across the existing layers (cli, engine and store)... all without introducing much unnecessary complexity. But exciting times lie ahead!

November 7, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

Submitted a placeholder implementation of the new persistence layer (store top-level directory). This only supports opening a database and ensuring its metadata is valid. See r253.Added a db-exec CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet. See r255.Miscellaneous fixes and improvements to utils::env and utils::sqlite.Preliminary code to support putting objects (like actions and contexts) into the database. I've been thinking about this for a while and finally came up with a design that completely decouples the persistence needs from the higher-level classes in the engine layer. I haven't submitted the code yet though, as it lacks tests. (Still thinking how to write the loading of objects though.)

October 31, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.After this refactoring, the code left in the CLI modules is purely presentation-related, and the code in the engine implements all the logic.The goal of these changes is to be able to hide the interactions with the database in these controllers. The CLI layer has no business in dealing with the database connection (other than allowing the user to specify which database to talk to, of course).Implemented a very simple RAII model for SQLite transactions.Some additions to the utils::sqlite bindings to simplify some common calling patterns (e.g. binding statement parameters by name).Preliminary prototypes at database initialization. This involves creating new databases and populating them with the initial schema, plus dealing with database metadata to, e.g. detect if we are dealing with the correct schema version.The code for this is still too crappy to be submitted, so don't look for it in the repository just yet!The design document details many things that should be part of the schema (e.g. "sessions"0, but I've decided that I'll start easy with a simplified schema and later build on top of it. Otherwise there will be too many clunky moving parts to deal with while the fundamental ideas are not yet completely clear.Fixes to let the code build and run again in NetBSD (macppc at least).I've now been stuck for a few days trying to figure out what the best way to implement the conversion of (new) in-memory objects to database objects is, and how to later recover these objects. E.g. what the correct abstractions are to take test case results and put them in the database, and how to retrieve these results to generate reports later on. I now start to have a clear mental picture on how this should look like, but I have yet to see how it will scale.

October 23, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

Added support to prepare and execute statements to utils::sqlite.Added the UTILS_PURE definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.Made a quick attempt at getting kyua test record test results in a simple database; success! This was just a proof of concept, so it is not submitted yet.Started to refactor the code to move many logic constructions from the cli to the engine. With the need to store test results in the database, it's clear that these semantics do not belong in the CLI, but the current code structure do not easily allow this. Need to add some "controller" classes in the engine to hide all the interaction among internal components.

October 16, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Some more reading on SQLite. The Using SQLite book is close to awesome; very easy to read and full of insightful tips.Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep procrastinating. So, ladies and gentlemen, welcome the new utils::sqlite module in Kyua. At the moment, this just provides the barebones to open and close a database. I need to start playing around with preparing statements before I can model these in my wrapper classes. (Yes, the API structure is extremely similar to that in Lutok.)Update (23:40): I sent this too early, thinking I would not have anything nice to report this week either. But as it turns out, I finally had some spare time this late evening and got a chance to submit the extremely-incomplete SQLite C++ bindings I've been toying around with. See the update above! :-)

October 9, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, not much time this week either :-(I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases.  However, I've not had the to time to finish these edits and publish them.

October 3, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, no activity this week other than some brainstorming on the database design. Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!

September 26, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report (db designdoc edition!)

Moved the code of utils::lua to a new project, Lutok.Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.Been chewing through the first chapters of the "Using SQLite" book to refresh my SQL skills. And, most importantly, wrote a preliminary design document for the database store of Kyua and the reporting features. Comments certainly welcome! Be aware that this is how atf-report will be replaced, so once this is done we should be able to finally kill atf-run and atf-report altogether :-)

September 18, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/lutok">lutok</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Introducing Lutok: A lightweight C++ API for Lua

It has finally happened. Lutok is the result of what was promised in the "Splitting utils::lua from Kyua" web post. Quoting the project web page: Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a small subset of miscellaneous utility functions built on top of the wrappers. Lutok focuses on providing a clean and safe C++ interface; the drawback is that it is not suitable for performance-critical environments. In order to implement error-safe C++ wrappers on top of a Lua C binary library, Lutok adds several layers or abstraction and error checking that go against the original spirit of the Lua C API and thus degrade performance. Lutok was originally developed within Kyua but was later split into its own project to make it available to general developers.Coming up with a name for this project was quite an odyssey, and is what has delayed is release more than I wanted. My original candidate was "luawrap" which, although not very original, was to-the-point and easy to understand. Unfortunately, that name did not clear with the legal department and I had to propose several other names, some of which were not acceptable either. Eventually, I settled with "Lutok", which comes from "LUa TOolKit". At this point, the source tree of Lutok provides pretty much the same code as the utils::lua module of Kyua. While it may be enough to get you started, I'm pretty sure you will lack some functions in the state class. If that is the case, don't hesitate to file a bug report to let me know what is missing. In case you missed the link above, the project page is here: Lutok in Google Code.

September 15, 2011 · Tags: <a href="/tags/announce">announce</a>, <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>, <a href="/tags/lutok">lutok</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(On another order of things, I have started writing a design document for the database that will collect test case results and other information. Will share it as soon as it is readable and more or less complete.

September 12, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Slow week.  We had some team-related events at work and I have friends over from Dublin, so it has been hard to find some time to do work on Kyua.  Regardless, here comes the weekly report: Split utils::lua into its own package, per some user's request.  I'm still setting up the separate project and have to do lots of cleanup on the code, so nothing is available yet.Started experimenting on the long promised "tests results database".  So far, I have started writing a small utils::sqlite3 C++, RAII-based wrapper for SQLite 3 and a SQL schema for the database.  My rusty SQL skills don't help :-PHowever, all this work is local so there have been no commits to the repository this week; sorry!

September 4, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Splitting utils::lua from Kyua

If you remember a post from January titled C++ interface to Lua for Kyua (wow, time flies), the Kyua codebase includes a small library to wrap the native Lua C library into a more natural C++ interface. You can take a look at the current code as of r129. Quoting the previous post: The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.While the RAII wrappers and other C++-specific constructions are a very nice thing to have, this library has to jump through a lot of hoops to interact with binary Lua versions built for C. This makes utils::lua not usable for performance-critical environments. Things would be way easier if utils::lua linked to a Lua binary built for C++, but unfortunately that is not viable in most, if not all, systems with binary packaging systems (read: most Linux distributions, BSD systems, etc.). That said, I've had requests from a bunch of people to provide utils::lua separately from Kyua regardless of any performance shortcomings it may have, and this is what I have started doing this weekend. So far, I already have a pretty clumsy standalone package (I'll keep the name to myself for now ;-) that provides this library on its own with the traditional Automake, Autoconf and Libtool support. Once this is a bit better quality, and once I modify Kyua to link against this external library and assess that things work fine, I'll make the decision on how to publish this (but most likely it should be a separate project in Google Code). Splitting the code doesn't come with its own issues though: maintaining a separate package will involve more work and hopefully/supposedly, dealing with quite a few feature requests to add missing functionality! Also, it means that utils::lua cannot use any of the other Kyua libraries (utils::sanity for example), so I lose a bit of consistency across the Kyua codebase. I am also not sure about how to share non-library code (in particular, the m4 macros for Autoconf) across the two packages. So, my question is: are you interested in utils::lua being shipped separately? :-)  Do you have any cool use cases for it that you can share here? Thanks!

September 3, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 3 minutes)

Kyua: Weekly status report

Not a very active week: I've been on-call four days and they have been quite intense. Plus I have had to go through a "hurricane" in NYC. That said, I had some time to do a bit of work on Kyua and the results have been nice :-) Made calls to getopt_long(3) work with GNU Getopt by using the correct value of optind to reset option processing.Improved the configure script to error out in a clearer way when missing dependencies (pkg.m4 and Lua) are not found.Did some portability fixes.And released Kyua 0.2! (along with a pkgsrc package) At this point, I have to start thinking how to implement test suite reporting within Kyua (i.e. how to replace atf-report). This probably means learning SQLite and refreshing my incredibly rusty SQL skills. Also, it's time to (probably) split the utils::lua library in a separate package because there is several people interested in this.

August 28, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua 0.2 released

Dear readers, I am very proud to announce the second formal release of Kyua, 0.2.  This release comes with lots of internal changes and a few user-visible changes.  Instead of listing all the changes and news here, I'll just recommend you to visit the 0.2 release page and read all the notes. This release has been tested under NetBSD-current, Mac OS X Snow Leopard, Debian sid and Ubuntu 10.04.1 LTS. Please report any problems that you encounter. Have fun!

August 24, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/release">release</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Implemented the "debug" command. Still very rudimentary, this command allows the user to run a test case without capturing its stdout nor stderr to aid in debugging of failed test cases. In the future, this command will also allow things like keeping the work directory for manual inspection, or spawning a shell or a debugger in the work directory after a test case is executed. Many build fixes under different platforms in preparation for a 0.2 release. In particular, Kyua now builds under Ubuntu 10.04.1 LTS but some tests fail. Had to disable the execution of the bootstrap test suite within Kyua because it stalls in systems where the default shell is not bash.  I presume this is a bug in GNU Autotest, so I filed a report.

August 21, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Changed the --config and --variable options to be program-wide instead of command-specific. The configuration file should be able to hold properties that tune the behavior of Kyua, not just the execution of tests, so this makes sense. Added the config subcommand, which provides a way to inspect the configuration as read by Kyua. Got rid of the test_suites_var function from configuration files and replaced it by simple assignments to variables in the test_suites global table. Enabled detection of unused parameters by the compiler and fixed all warnings. Changed developer mode to only control whether warnings are enforced or not (not to enable the warnings themselves) and made developer mode be disabled on formal releases. Barring release testing, Kyua 0.2 should be ready soon :-)

August 15, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Added the ability to explicitly define timeouts for plain test programs. This just completes the work from past week that made running plain test programs at all but had an ugly TODO in it to implement this missing feature. The bootstrap test suite now runs as a single test case within the whole Kyua test suite. Demonstrates the plain test programs interface functionality :-) Started reshuffling code to make the <tt>--config</tt> and related flags program-wide. The goal is to allow the configuration file to tune the behavior of all of Kyua, so these flags must be made generic. They were previously specific to the <tt>test</tt> subcommand only.

August 8, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Implemented the "plain" test interface. This allows plugging "foreign test programs" into a Kyua-based test suite. (A foreign test program is a program that does not use any testing framework: it reports success and failure by means of an exit code.)Generalized code between the atf and plain interfaces and did some cleanups.Attempted to fix the ATF_REQUIRE_EQ macros in ATF to evaluate their arguments only once. This has proven to be tricky and therefore it is not done yet. At the moment, the macros reevaluate the arguments on a failure condition, which is not too big of a deal.

August 1, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Finished splitting the atf-specific code from the generic structures in the engine. The engine now supports the addition of extra test interfaces with minimal effort.Started implementing a "plain" test interface for test programs that do not use any test framework. This is to allow muxing non-atf tests into atf-based test cases, which is required in the NetBSD tree.

July 25, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Slow week. I've been busy moving to NYC! Kept working on the splitting of ATF-specific code from the core test abstractions. The work is now focused on refactoring the results-reporting pieces of the code, which are non-trivial.

July 18, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

One of the major features I want in place for Kyua 0.2 is the ability to run "foreign" test programs as part of a test suite: i.e. to be able to plug non-ATF test programs into a Kyuafile. The rationale for this is to lower the entry barrier of newcomers to Kyua and, also, to allow running some foreign test suites that exist in the NetBSD source tree but that are currently not run. The work this week has gone in the direction outlined above among other things: Created an abstract base class to represent test programs and provided an implementation for ATF.Did the same thing for test cases.Moved the kyua-cli package from pkgsrc-wip into pkgsrc head. Installing Kyua is now a breeze under NetBSD (and possibly under other platforms supported by pkgsrc!) The next steps are to generalize the test case results, clearly separate the ATF-specific code from the general abstractions, and add an implementation to run simple test programs.

July 10, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Belated update:Created a pkgsrc package for kyua-cli. Still in pkgsrc-wip though because pkgsrc is in a feature freeze.Wrote a little tutorial on how to run NetBSD tests using Kyua.Started work on 0.2 by doing a minor UI fix in the about command.I've now started to look at how to split the engine into different "runners" to add support for test programs written without ATF. Not that I plan to use this feature... but having it in place will ensure that the internal interfaces are clean and may help in adoption of Kyua.

July 5, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

This has been the big week: Wrote some user documentation for the kyua binary.Fixed some distcheck problems.Released Kyua 0.1!The next immediate thing to do is to write a short tutorial on how to run the NetBSD tests with Kyua and get some people to actually try it. After that, there are many things to improve and features to add :-)

June 26, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua 0.1 released!

Dear readers, I'm very proud to announce that the very first release of Kyua, obviously numbered 0.1, has been published! The release page for kyua-cli-0.1 contains more details about what this release offers, and don't hesitate to join the kyua-discuss mailing list and ask if you have questions. Kyua was started sometime in past October and it has taken over six months to get the first public version. (It is true that my free time has heavily fluctuated during this period though, so it does not mean that six months of intensive coding have gone into the release ;-) Kyua 0.1 hovers at almost 40K lines of code according to Ohloh and comes with 800 test cases, yet it only implements a replacement for the atf-run component of ATF. I have a package ready for pkgsrc, but unfortunately can't submit it because pkgsrc is in a freeze in preparation for the next stable branch. No matter what, expect detailed instructions on how to install Kyua and how to run the NetBSD-current test suite with it very soon. Enjoy, and keep the bug reports coming!

June 23, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/release">release</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

A couple of things have happened: Released ATF 0.14. This release was followed by an import into NetBSD and fixing of subsequent fallout.Some performance improvements to atf-sh. After killing a bunch of complex shell constructions and removing lots of obsolete functions, the performance results are significant. There is still room for improvement of course, and I still need to quantify how these optimizations behave in single-core machines.I certainly expected more progress this past week... but in case you don't know: I am moving countries very soon now, and as the move date approaches, there is more and more stuff to be done at home so less and less time for hacking.

June 19, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Added support for recursion from the top-level Kyuafile. This Kyuafile should not reference any directories explicitly because the directories at the top level are supposed to be created by the installation of packages. Closed issue 9.Improved error messages when the test programs are bogus. Closed issue 13.Backported format-printf attribute improvements from NetBSD head to ATF.Miscellaneous build and run fixes for both Kyua and ATF in NetBSD and OS X.Cut a release candidate for atf-0.14 and started testing on NetBSD.The kyua-cli codebase is now feature complete. Blocking the 0.1 release are the need to polish the release documents and the requirement of releasing atf-0.14 beforehand. Should happen soon :-)

June 13, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Some long-standing bug fixes / improvements have gone in this week: Improvements to the cleanup routine, which is used to destroy the work directory of a test case after the test case has terminated:Heavy refactoring to be tolerant to failures. These failures may arise when a child of the test case does not exit immediately and holds temporary files in the work directory open for longer than expected.Any file systems that the test case leaves mounted within the work directory will now be unmounted, just as the ATF test interface mandates. I realize that this adds a lot of complexity to the runtime engine for very little gain. If/when we revise the tests interface, it will be worth to reconsider this and maybe leave the cleanup of mounted file systems to the test case altogether.As a result, issue 17 has been fixed!Kyua now captures common termination signals (such as SIGINT) and exits in a controlled manner. What this means is that Kyua will now kill any active test programs and clean up any existing work directories before exiting. What this also means is that issue 4 is fixed.To increase amusements, a little FYI: the above points have never worked correctly in ATF, and the codebase of ATF makes it extremely hard to implement them right. I have to confess that it has been tricky to implement the above in Kyua as well, but I feel much more confident in that the implementation works well. Of course, there may be some corner cases left... but, all in all, it's more robust and easier to manage. The list of pending tasks for 0.1 shortens!

June 5, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

Some cool stuff this week, albeit not large-scale: Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.Added support for several environment variables to allow users (and tests) to override built-in paths. I can't imagine right now any legitimate use for these variables, but hardcoded values are bad in general, atf-run provided these same variables, and these variables are very handy for testing purposes.Added support for the new require.files test-case metadata property to both ATF and Kyua. This new property allows tests to specify a set of files that they require in order to run, and is useful for those tests that can't run before make install is executed. The functionality planned for the 0.1 release is now pretty much complete. There is still a few rough edges to clean, some documentation to write, and some little features to implement/fix. See the open bugs for 0.1 to get an idea of the remaining tasks.

May 29, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

This week:Cleaned up the internal code of the "list" command and added a few unit tests.Added integration tests for the "test" command that focus mostly on the behavior of the "test" command itself. There is still a need for full integration tests that validate the execution of the test cases themselves and their cleanup, and these will be tricky to write.Changed atf-c, atf-c++ and atf-sh to show a warning when a test program is run by hand. Users should really be using atf-run to execute the tests, or otherwise things like isolation or timeouts will not work (and they'll conclude that atf is broken!).

May 22, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report, BSDCan 2011 edition

I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers. Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them. (Video coming soon, I hope.) The talk was later followed by several questions and off-session conversations about testing in general in BSDs. Among these, I gathered a few random ideas / feelings: The POSIX 1003.3 standard defines the particular results a test can emit (see the corresponding DejaGnu documentation). Both ATF and Kyua already implement all the results defined in the standard, but they use different names and extend the standard with many extra results. Given that the standard does not define useful concepts like "expected failures", an idea that came up is to provide a flag to force POSIX compliance at the cost of being less useful. Why? Just for the sake of saying that Kyua conforms to this standard.The audience seemed to like the idea of a "tests results store" quite a bit, and the sound of SQLite for the implementation was not bullied. This is something I'm eager to work on, but not before I publish a 0.1 release.I highlighted the possibility of allowing Kyua to run "foreign" test programs so that we could integrate the results into the database. This could be useful to run tests for which we (*BSD) have no control (e.g. gcc) in an integrated manner. The idea was not bullied by anyone either.FreeBSD has already been looking at ATF / Kyua and they are open to collaboration.OpenBSD won't import any new C++ code, and adding C-based tests to the tree while relegating the C++ runtime to the ports is not an option. Somehow I expected this.Junos (the FreeBSD-based operating system from Jupiter Networks) recently imported ATF and they are happy with it so far. Yay!Would be nice to have a feature to run tests remotely after, maybe, deploying a single particular test and its dependencies. This is gonna be tricky and not in my current immediate plans.Other than that, I had little time to do some coding:Fixed a problem in which both ATF and Kyua were not correctly resetting the timezone of the executed tests. I only found this because, after arriving in Canada, some Kyua tests would start to fail. (Yes, the fix is in both code bases!)Added some support to capture deadly signals that terminate Kyua so that Kyua can print an informational message stating that something went wrong and which log file contains more information. See r121.That's it folks! Thanks to those attending the conference and, in particular, to those that came to my talk :-)

May 16, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/conference">conference</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 3 minutes)

Kyua: Weekly status report

Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow.

May 8, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, I have had no time for coding this week. The only things I could do were:Fixed a few build problems on NetBSD introduced during past week's changes.Built Kyua and ran a few tests on NetBSD/macppc (just for the joy of it).Coding has been eclipsed by the preparation of my presentation for BSDCan 2011; at this point, this has priority over any code changes. I'd argue that preparing the presentation is also part of the project, so some time has been invested ;-) More next week, hopefully, but I don't expect being able to do any big code improvements until after BSDCan.

May 1, 2011 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

Ouch; I'm exhausted. I just finished a multi-hour hacking session to get the implementation of the list subcommand in control. It is now in a very nice user-facing shape, although its code deserves a little bit of house cleaning (coming soon). Anyway, this week's progress: Added the kyuaify.sh script. This little tool takes a test suite (say, NetBSD's /usr/tests directory) and converts all its Atffiles into Kyuafiles. The tool is no sophisticated at all; in fact, it is a pretty simple script that I haven't tested with any other test suites so far. See the announcement of kyuaify for some extra details.Added logging support for the Lua code and changed the Lua modules to spit some logging information while processing Kyuafiles and configuration files.Added a mechanism in the user interface module to consistently print informational, warning and error messages.Implemented proper test filtering (after several iterations). What does proper mean? Well, for starters, test filters are always relative to the test suite's root (although we already saw this in last week's report). But the most important thing is that the filters are now validated: nice, user-friendly errors will be reported when the collection of tests is non-disjoint, when it includes duplicate names or when any of the provided filters does not match any test case. I really need to document the rationale of these in the manual, but for now the r118 commit message includes a few details.Drafted some notes for the BSDCan 2011 conference. I am quite tempted to reuse parts of the presentation from NYCBSDCon 2010, but I really want to give more emphasis on Kyua this time. In case you don't know, Kyua was first announced in NYCBSDCon 2010 and it was still a very immature project. The project has changed a lot since then.The wishful plan for next week is to clean up the internals of the list command (by refactoring and adding unit tests) and implement preliminary integration tests for the test subcommand. The latter scares me quite a bit. But... hmm... I guess preparing the presentation for BSDCan 2011 has priority.

April 24, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

This week started easy:Added integration tests for the about and help subcommands. These were pretty easy to do.Added integration tests for the list subcommand. I initially added these tests as expected failures to reason about the appearance and behavior of this command from the point of view of the user before actually working on the code... and I am still writing such code to make these tests pass!This is where things got a bit awry. Polishing the behavior of the list command so that its interface is consistent among all flag and argument combinations is tricky and non-trivial. I ended up having to change code deep down in the source tree to implement missing features and to change existing bits and pieces:Implemented support to print all test case properties, not only the user-specific ones. The properties recognized by the runtime engine are stored as individual arguments of a structure, so these required some externalization code.Implemented "test case filtering". This allows users to select what tests to run on the command line at the test case granularity. For example, foo/bar selects all tests in a subdirectory if bar is a directory, or all the tests in the bar test program if it is a binary. But what is new (read: not found in ATF) is that you can even do foo/bar:test-1 where test-1 is the name of a test case within the foo/bar test program. I've been silently wishing for this feature to be available in ATF because it shortens the build/test/edit cycle, but it was not easy to add.Changed the internal representation of test suites to ensure all test program names are relative to the root of the test suite. The root of the test suite is considered to be the directory in which the top-level Kyuafile lives (/usr/tests/ in NetBSD). The whole point of doing this is to provide some consistency to the filters when the user decides to execute tests that are not in the current directory. For example, the following are now equivalent: $ cd /usr/tests && kyua list fs/tmpfs $ kyua list -k /usr/tests/Kyuafile fs/tmpfsThe plans for the upcoming week are to finish with the clean up of the list command (which involves adding proper error reporting, refactoring of the command module and addition of unit tests) and start cleaning up the test command. Also, remember that BSDCan 2011 is now around the corner and that I will be talking about ATF, Kyua and NetBSD in it!  My plan was to have a kyua-cli-0.1 release by the time of the conference, although this will be tricky to achieve...

April 17, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 3 minutes)

Kyua: Weekly status report

Few things worth mentioning this week as reviewing Summer of Code student applications has taken priority. The good thing is that there are several strong applications for NetBSD; the bad thing is that none relate directly to testing. Anyway, the work this week: Added a pkg-config file for atf-sh as well as an Autoconf macro to detect its presence. This is needed by Kyua to easily find atf-sh. (Yes, I know: this is an abuse of pkg-config, but it works pretty well and is consistent with atf-c and atf-c++.)Implemented basic integration tests for Kyua in r98 using atf-sh. These tests are still very simple but provide a placeholder into which new tests will be plugged. Having good integration test coverage is key in preparation for a 0.1 release. Oh, and by the way, this revision has bumped the number of tests to 601, crossing the 600 barrier :-)That's pretty much it. Now, back to attempting to fix my home server as a fresh installation of NetBSD/macppc has decided to not boot any more.  (Yes, this has blocked most of my weekend...)

April 10, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 1 minute)

Kyua: Weekly status report

This week's work has been quite active on the ATF front but not so much in the Kyua one. I keep being incredibly busy on the weekends (read: traveling!) so it's hard to get any serious development work done. What has happened? Finally tracked down and fixed some random atf-run crashes that had been hunting the NetBSD test suite for months (see PR bin/44176). The fix is in reality an ugly workaround for the fact that a work directory cannot be considered "stable" even after the test case terminates. There may be dying processes around that touch the work directory contents, and the cleanup code in atf-run was not coping well with those. As it turns out, this problem also exists in Kyua (even though it's not as pronounced because arbitrary failures when running a test case do not crash the runtime engine) so I filed issue 17 to address it.Released ATF 0.13 and imported it both to NetBSD-current and pkgsrc. As a side note, Kyua requires the new features in this release, so putting it out there is a requirement to release Kyua 0.1. This new release does not have a big effect on NetBSD though, because the copy of ATF in NetBSD has been constantly receiving cherry-picks of the upstream fixes.Replaced several TODO items in the Kyua code with proper calls to the logging subsystem. These TODO items were referring to conditions in the code that should not happen, but for which we cannot do any proper recovery (like errors in a destructor). Sure, these could be better signaled as an assertion... but these code paths can be triggered in extremely-tricky conditions and having Kyua crash because of them is not nice (particularly when the side-effects of executing that code paths are non-critical).So, in retrospect, I have fulfilled the goal set past week of releasing ATF 0.13, but I haven't got to the addition of integration tests. Oh well... let's see if this upcoming week provides more spare time.

April 3, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 2 minutes)

Kyua: Weekly status report

This has been a slow week. In the previous report, I set the goal of getting Kyua to run the NetBSD test suite accurately (i.e. to report the same results as atf-run), and this has been accomplished. Actually, the changes required in Kyua to make this happen were minimal, but I got side-tracked fixing issues in NetBSD itself (both in the test suite and in the kernel!). So, the things done: Fixed Kyua to correctly kill any dangling subprocesses of a test case and thus match the behavior of atf-run. This was the only change required to make issue 16 happen: i.e. to get Kyua to report the same results as atf-run for the NetBSD test suite.Based on a suggestion from Antti Kantee, an alternative way to handle this would be to not kill any processes and just report the test case as broken if it fails to clean itself up. The rationale being that the runtime engine can kill dangling subprocesses in 99% of the occasions, but not always. The exception are those subprocesses that change their process group. It'd be better to make all cleanups explicit instead of hiding this corner case, as it can lead to confusion. Addressing this will have to wait though, as it is a pretty invasive change.Before closing issue 16, I want to implement some integration tests for Kyua to ensure that the whole system behaves as we expect (which is what the NetBSD test suite is currently doing implicitly).Kyua is pickier than atf-run: if a cleanup routine of a test case fails or crashes, Kyua will (correctly) report the test case as broken while atf-run will silently ignore this situation. Some NetBSD tests had crashing cleanup parts, so I fixed them.Some test programs in NetBSD were leaving unkilled subprocesses behind. These subprocesses are daemons and thus fall out of the scope of what Kyua can detect and kill during the cleanup phase. I mistakenly tracked down the problem to rump, but Antti Kantee kindly found the real problem in the kernel (not in rump!).As a side effect of processes being left behind, I extended the functionality of pidfile(3) and implemented pid file support in bozohttpd.  This is to make the tests that spawn a bozohttpd in the background more robust, by giving them a way to forcibly kill the server during cleanup.These changes are still under review and not committed yet.For the upcoming week, I plan to add some basic integration tests to Kyua and release ATF 0.13. I've been running a NetBSD system with the latest ATF code integrated for a while (because Kyua requires it) and things have been working well.

March 27, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 3 minutes)

Kyua: Weekly status report

These days, I find myself talking about Kyua to "many" people. In particular, whenever a new feature request for ATF comes in, I promise the requester that the feature will be addressed as part of Kyua. However, I can imagine that this behavior leaves the requester with mixed feelings: it is nice that the feature will be implemented but, at the same time, it is very hard to know when because the web site of Kyua does not provide many details about its current status. In an attempt to give Kyua some more visibility, I will start posting weekly activity reports in this blog. These reports will also include any work done on the ATF front, as the two projects are highly related at this point. I write these reports regularly at work and I feel like it is a pretty good habit: every week, you have to spend some time thinking about what you did for the project and you feel guilty if the list of tasks is ~zero ;-) It also, as I said, gives more visibility to the work being done so that outsiders know that the project is not being ignored. Before starting with what has happened this week, a bit of context. I have been traveling like crazy and hosting guests over for the last 2 months. This has given me virtually no time to work on Kyua but, finally, I have got a chance to do some work this past week. So, what are the news?Implemented the --loglevel command line flag, which closes issue 14. Kyua now generates run-time logs of its internal activity to aid in postmortem debugging and this flag allows the user to control the verbosity of such logs.Antti Kantee hacked support for atf-run in the NetBSD source tree to dump a stack trace of any crashing test program. I have backported this code to the upstream ATF code and filed issue 15 to implement this same functionality in Kyua.Fixed a hang in atf-run that made it get stuck when a test case spawned a child processes and atf-run failed to terminate it. A quick test seems to indicate that Kyua is affected by a similar problem: it does not get stuck but it does not correctly kill the subprocesses. The problem will be addressed as part of issue 16.Oh, and by the way: Kyua will be presented at BSDCan 2011.My plans for this week are to make Kyua run the full NetBSD test suite without regressions when compared to ATF. Basically, the results of a test run with Kyua should be exactly the same as those of a test run with ATF. No dangling processes should be left behind. Lastly, if you are interested in these reports and other Kyua news, you can subscribe to the kyua label feed and, if you want to stay up to date with any changes performed to the code, subscribe to the kyua-log mailing list.

March 20, 2011 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/report">report</a>
Continue reading (about 3 minutes)

Injecting C++ functions into Lua

The C++ interface to Lua implemented in Kyua exposes a lua::state class that wraps the lower-level lua_State* type. This class completely hides the internal C type of Lua to ensure that all calls that affect the state go through the lua::state class. Things get a bit messy when we want to inject native functions into the Lua environment. These functions follow the prototype represented by the lua_CFunction type:typedef int (*lua_CFunction)(lua_State*);Now, let's consider this code:int awesome_native_function(lua_State* state) { // Uh, we have access to s, so we bypass the lua::state! ... do something nasty ... // Oh, and we can throw an exception here... //with bad consequences. } void setup(...) { lua::state state; state.push_c_function(awesome_native_function); state.set_global("myfunc"); ... run some script ... }The fact that we must pass a lua_CFunction prototype to the lua_pushcfunction object means that such function must have access to the raw lua_State* pointer... which we want to avoid. What we really want is the caller code to define a function such as:typedef int (*cxx_function)(lua::state&)In an ideal world, the lua::state class would implement a push_cxx_function that took a cxx_function, generated a thin C wrapper and injected such generated wrapper into Lua. Unfortunately, we are not in an ideal world: C++ does not have high-order functions and thus the "generate a wrapper function" part of the previous proposal does not really work. What we can do instead, though, is to make the creation of C wrappers for these C++ functions trivial. And this is what r42 did. The approach I took is similar to this overly-simplified (and broken) example:template< cxx_function Function > int wrap_cxx_function(lua_State* state) { try { lua::state state_wrapper(state); return Function(state_wrapper); } catch (...) { luaL_error(state, "Geez, don't go into C's land!"); } }This template wrapper takes a cxx_function object and generates a corresponding C function at compile time. This wrapper function ensures that C++ state does not propagate into the C world, as that often has catastrophical consequences. (Due to language limitations, the input function must have external linkage. So no, it cannot be static.) As a result, we can rewrite our original snippet as:int awesome_native_function(lua::state& state) { // See, we cannot access lua_State* now. ... do something ... throw std::runtime_error("And we can even do this!"); } void setup(...) { lua::state state; state.push_c_function( wrap_cxx_function< awesome_native_function >); state.set_global("myfunc"); ... run some script ... }Neat? I think so, but maybe not so much. I'm pretty sure there are cooler ways of achieving the above purpose in a cleaner way, but this one works nicely and has few overhead.

January 17, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 2 minutes)

Error handling in Lua: the Kyua approach

About a week ago, I detailed the different approaches I encountered to deal with errors raised by the Lua C API. Later, I announced the new C++ interface for Lua implemented within Kyua. And today, I would like to talk about the specific mechanism I implemented in this library to deal with the Lua errors. The first thing to keep in mind is that the whole purpose of Lua in the context of Kyua is to parse configuration files. This is an infrequent operation, so high performance does not matter: it is more valuable to me to be able to write robust algorithms fast than to have them run at optimal speed. The other key point to consider is that I want Kyua to be able to use prebuilt Lua libraries, which are built as C binaries. The approach I took is to wrap every single unsafe Lua C API call in a "thin" (FSVO thin depending on the case) wrapper that gets called by lua_pcall. Anything that runs inside the wrapper is safe to Lua errors, as they are caught and safely reported to the caller. Lets examine how this works by taking a look at an example: the wrapping of lua_getglobal. We have the following code (copy pasted from the utils/lua/wrap.cpp file but hand-edited for publishing here):static int protected_getglobal(lua_State* state) { lua_getglobal(state, lua_tostring(state, -1)); return 1; } void lua::state::get_global(const std::string& name) { lua_pushcfunction(_pimpl->lua_state, protected_getglobal); lua_pushstring(_pimpl->lua_state, name.c_str()); if (lua_pcall(_pimpl->lua_state, 1, 1, 0) != 0) throw lua::api_error::from_stack(_pimpl->lua_state, "lua_getglobal"); }The state::get_global method is my public wrapper for the lua_getglobal Lua C API call. This wrapper first prepares the Lua stack by pushing the address of the C function to call and its parameters and then issues a lua_pcall call that executes the C function in a Lua protected environment. In this case, the argument preparation for protected_getglobal is trivial because the lua_getglobal call does not require access to any preexisting values on the Lua stack. Things get much trickier when that happens as in the case of the lua_getglobal wrapper. I'll leave understanding how to do this as an exercise to the reader (but you can cheat by looking at line 154). Anyway. The above looks all very nice and safe and the tests for the state::get_global function, even the ones that intentionally cause a failure, all work fine. So we are good, right? Nope! Unfortunately, the code above is not fully safe to Lua errors. In order to prepare the lua_pcall execution, the code must push values on the stack. As it turns out, both lua_pushcfunction and lua_pushstring can fail if they run out of memory (OOM). Such failure would of course be captured inside a protected environment... but we have a little chicken'n'egg problem here. That said, OOM failures are rare so I'm going to leverage this fact and not worry about it. (Note to self: install a lua_atpanic handler to complain loudly if that ever happens.) Addendum: Bundling Lua within my program and building it as a C++ binary with exception reporting enabled in luaconf.h would magically solve all my issues. I know. But I don't fancy the idea of bundling the library into my source tree for a variety of reasons.

January 14, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 3 minutes)

C++ interface to Lua for Kyua

Finally! After two weeks of holidays work, I have finally been able to submit Kyua's r39: a generic library that implements a C++ interface to Lua. The code is hosted in the utils/lua/ subdirectory. From the revision description:The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors. The library also provides a place (the operations module) to add miscellaneous utility functions built on top of the wrappers.In other words: this code aims to decouple all details of the interaction with the Lua C API from the main code of Kyua so that the high level algorithms do not have to worry about Lua C API idiosyncrasies. Further changes to Kyua to implement the new configuration system will follow soon as all the basic code to talk to Lua has been ironed out. Also expect some extra posts regarding the design decisions that went on this helper code and, in particular, about error reporting as mentioned in the previous post. (Yep, Lua and Kyua sound similar. But that was never intended; promise!)

January 8, 2011 · Tags: <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 2 minutes)

Using RAII to clean up temporary values from a stack

For the last couple of days, I have been playing around with the Lua C API and have been writing a thin wrapper library for C++. The main purpose of this auxiliary library is to ensure that global interpreter resources such as the global state or the execution stack are kept consistent in the presence of exceptions — and, in particular, that none of these are leaked due to programming mistakes when handling error codes. To illustrate this point, let's forget about Lua and consider a simpler case. Suppose we lost the ability to pass arguments and return values from functions in C++ and all we have is a stack that we pass around. With this in mind, we could implement a multiply function as follows:void multiply(std::stack< int >& context) { const int arg1 = context.top(); context.pop(); const int arg2 = context.top(); context.pop(); context.push(arg1 * arg2); }And we could call our function as this:std::stack< int > context; context.push(5); context.push(6); multiply(context); const int result = s.top(); s.pop();In fact, my friends, this is more-or-less what your C/C++ compiler is internally doing when converting code to assembly language. The way the stack is organized to perform calls is known as the calling conventions of an ABI (language/platform combination). Anyway, back to our point. One important property of such a stack-based system is that any function that deals with the stack must leave it in a consistent state: if the function pushes temporary values (read: local variables) into the stack, such temporary values must be gone upon return no matter how the function terminates. Otherwise, the caller will not find the stack as it expects, which will surely cause trouble at a later stage. The above example works just fine because our function is extremely simple and does not put anything on the stack. But things get messier when our functions can fail halfway through, and, in particular, if such failures are signaled by exceptions. In these cases, the function will abort abruptly and the function must take care to clean up any values that may still be left on the stack. Let's consider another example:void magic(std::stack< int >& context) { const int arg1 = context.top(); context.pop(); const int arg2 = context.top(); context.pop(); context.push(arg1 * arg2); context.push(arg1 / arg2); try { ... do something with the two values on top ... context.push(arg1 - arg2); try { ... do something with the three values on top ... } catch (...) { context.pop(); // arg1 - arg2 throw; } context.pop(); } catch (...) { context.pop(); // arg1 / arg2 context.pop(); // arg1 * arg2 throw; } context.pop(); context.pop(); }The above is a completely fictitious and useless function, but serves to illustrate the point. magic() starts by pushing two values on the stack and then performs some computation that reads these two values. It later pushes an additional value and does some more computations on the three temporary values that are on the top of the stack. The "problem" is that the computation code can throw an exception. If it does, we must sanitize the stack to remove the two or three values we have already pushed. Otherwise, the caller will receive the exception, it will assume nothing has happened, and will leak values on the stack (bad thing). To prevent this, we have added a couple of try/catch clauses to capture these possible exceptions and to clean up the already-pushed values before exiting the function. Unfortunately, this gets old very quickly: having to add try/catch statements surrounding every call is boring, ugly, and hard to read (remember that, potentially, any statement can throw an exception). You can see this in the example above with the two nested try/catch blocks. To mitigate this situation, we can apply a RAII-like technique to make popping elements on errors completely transparent and automated. If we can make it transparent, writing the code is easier and reading it is trivial; if we can make it automated, we can be certain that our error paths (rarely tested!) correctly clean up any global state. In C++, destructors are deterministically executed whenever a variable goes out of scope, so we can use this to our advantage to clean up temporary values. Let's consider this class:class temp_stack { std::stack< int >& _stack; int _pop_count; public: temp_stack(std::stack< int >& stack_) : _stack(stack_), _pop_count(0) {} ~temp_stack(void) { while (_pop_count-- > 0) _stack.pop(); } void push(int i) { _stack.push(i); _pop_count++; } };With this, we can rewrite our function as:void magic(std::stack< int >& context) { const int arg1 = context.top(); context.pop(); const int arg2 = context.top(); context.pop(); temp_stack temp(context); temp_stack.push(arg1 * arg2); temp_stack.push(arg1 / arg2); ... do something with the two values on top ... temp_stack.push(arg1 - arg2); ... do something with the three values on top ... // Yes, we can return now. No need to do manual pop()s! }Simple, huh? Our temp_stack function keeps track of how many elements have been pushed on the stack. Whenever the function terminates, be it due to reaching the end of the body or due to an exception thrown anywhere, the temp_stack destructor will remove all elements previously registered from the stack. This ensures that the function leaves the global state (the stack) as it was on entry — modulo the function parameters consumed as part of the calling conventions. So how does all this play together with Lua? Well, Lua maintains a stack to communicate parameters and return values between C and Lua. Such stack can be managed in a similar way with a RAII class, which makes it very easy to write native functions that deal with the stack and clean it up correctly in all cases. I would like to show you some non-fictitious code right now, but it's not ready yet ;-) But when it is, it will be part of Kyua. Stay tuned! And, to conclude: to make C++ code robust, wrap objects that need manual clean up (pointers, file descriptors, etc.) with small wrapper classes that perform such clean up on destruction. These classes are typically fully inlined and contain a single member field, so they do not impose any performance penalty. But, on the contrary, your code can avoid the need of many try/catch blocks, which are tricky to get right and hard to validate. (Unfortunately, this technique cannot be applied in, e.g. Java or Python, because the execution of the class destructors is completely non-deterministic and not guaranteed to happen whatsoever!)

December 27, 2010 · Tags: <a href="/tags/c">c</a>, <a href="/tags/cxx">cxx</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 6 minutes)

Kyua: Design of the configuration system

Over a week ago, I mostly finished the implementation of the runtime engine for test cases of Kyua and, along the way, realized that it is imperative to write a configuration system right now before the configuration code becomes messier than it already is. To that end, I spent the last week working on a design document for the configuration system. Summarizing, the document describes what the requirements for the configuration files of Kyua are, what the possible alternatives to implement them are, and advocates the use of Lua — a tiny embedded programming language — to bring these configuration files to life. It is your chance to get involved in the early stages of the development of Kyua! :-) Take a look at the email to kyua-discuss asking for comments and feel free to join the mailing list ;-)

December 22, 2010 · Tags: <a href="/tags/kyua">kyua</a>, <a href="/tags/lua">lua</a>
Continue reading (about 1 minute)

Introducing Kyua

Wow. I have just realized that I have not blogged at all about the project that has kept me busy for the past two months! Not good, not good. "What is this project?", I hear. Well, this project is Kyua. A bit of background first: the Automated Testing Framework, or ATF for short, is a project that I started during the Summer of Code of 2007. The major goal of ATF was, and still is, to provide a testing framework for the NetBSD operating system. The ATF framework is composed of a set of libraries to aid in the implementation of test cases in C, C++ and shell, and a set of tools to ease the execution of such test cases (atf-run) and to generate reports of the execution (atf-report). At that point in time, I would say that the original design of ATF was nice. It made test programs intelligent enough to execute their test cases in a sandboxed environment. Such test programs could be executed on their own (without atf-run) and they exposed the same behavior as when they were run within the runtime monitor, atf-run. On paper this was nice, but in practice it has become a hassle. Additionally, some of these design decisions mean that particular features (in particular, parallel execution of tests) cannot be implemented at all. At the end of 2009 and beginning of 2010, I did some major refactorings to the code to make the test programs dumber and to move much of the common logic into atf-run, which helped a lot in fixing the major shortcomings encountered by the users... but the result is that, today, we have a huge mess. Additionally, while ATF is composed of different modules conceptually separate from each other, there is some hard implementation couplings among them that impose severe restrictions during development. Tangentially, during the past 2 years of working at Google (and coding mainly in Python), I have been learning new neat programming techniques to make code more testable... and these are not followed at all by ATF. In fact, while the test suite of ATF seems very extensive, it definitely is not: there are many corner cases that are not tested and for which implementing tests would be very hard (which means that nasty bugs have easily sneaked in into releases). Lastly, a very important point that affects directly the success of the project. Outsiders that want to contribute to ATF have a huge entry barrier: the source repository is managed by Monotone, the bug tracker is provided by Gnats (a truly user-unfriendly system), and the mailing lists are offered by majordomo. None of these tools is "standard" by today's common practices, and some of them are tied to NetBSD's hosting which puts some outsiders off. For all the reasons above and as this year has been moving along, I have gotten fed up with the ATF code base. (OK, things are not that bad... but in my mind they do ;-) And here is where Kyua comes into the game. Kyua is a project to address all the shortcomings listed above. First of all, the project uses off-the-shelf development tools that should make it much, much easier for external people to contribute. Secondly, the project intends to be much more modular, providing a clear separation between the different components and providing code that is easily testable. Lastly, Kyua intends to remain compatible with ATF so that there are no major disruptions for users. You can (and should) think of Kyua as ATF 2.0, not as a vastly different framework. As of today, Kyua implements a runtime engine that is on par, feature-wise, to the one provided by atf-run. It is able to run test cases implemented with the ATF libraries and it is able to test itself. It currently contains 355 test cases that run in less than 20 seconds. (Compare that to the 536 test cases of ATF, which take over a minute to run, and Kyua is still really far from catching up with all the functionality of ATF.) Next actions involve implementing reports generation and configuration files. Anyway. For more details on the project, I recommend you to read the original posting to atf-devel or the project's main page and wiki. And of course, you can also download the preliminary source code to take a look! Enjoy :-)

December 16, 2010 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/netbsd">netbsd</a>
Continue reading (about 4 minutes)


Julio Merino
A blog on operating systems, programming languages, testing, build systems, my own software projects and even personal productivity. Specifics include FreeBSD, Linux, Rust, Bazel and EndBASIC.

0 subscribers

Follow @jmmv on Mastodon Follow @jmmv on Twitter RSS feed

Featured software

Featured posts