Posts: Report

Kyua: Weekly status report

A couple of things happened this week:

• Spent quite a few time researching the idea of moving away from Monotone and Subversion to Git. I haven't made a decision yet, but I'm pretty convinced this is the right way to go. It will simplify development significantly, it will allow me to code offline (have a bunch of really long flights coming), and it will lower the entry barrier to Kyua by making all components use the same, mainstream VCS.
• Implemented filtering by result type of the test case results in the textual reports.

February 13, 2012 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Created an RPM package for Lutok for inclusion in Fedora.
• Created a preliminary RPM spec for ATF for Fedora. Now in discussions with the FPC to figure out how to install the tests on a Fedora system, as /usr/tests may not be appropriate.
• No activity on Kyua itself though, unfortunately.

February 7, 2012 · Tags: atf, kyua, lutok, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This comes two days late... but anyway, I felt like posting it now instead of waiting until next Sunday/Monday.

The activity past week focused mostly on implementing support for the require.memory test-case metadata property recently introduced into ATF. This was non-trivial due to the need to write some tricky Autoconf code to make this "slightly portable". Seriously: It's scary to see how hard it is to perform, in a portable manner, an operation as simple as "query the amount of physical memory"... but oh well, such are the native Unix APIs...

I later spent the weekend on Lutok, preparing and publishing the project's first release and writing my first RPM library spec for it. This was a prerequisite for the upcoming 0.3 release of Kyua and thus deserved special attention!

February 1, 2012 · Tags: kyua, lutok, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Released ATF 0.15 and imported it into NetBSD.
• Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.
• I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them. While this sounds like a cool idea (yay, code reuse!), it resulted in a ugly, nasty and horrible mess. In the end, I just ended up implementing custom parsing of the formatters, which was way easier and "good enough" for Kyua's needs.
• Started work on backporting ATF's new require.memory property into Kyua. This needs having a way to parse and format byte quantities in user-friendly forms (e.g. 1k, 2m, etc.)... hence the previous work on utils::format!
• Set up a Google+ Page for Kyua. I have no idea what to use it for yet. Maybe the status reports should go in there. Ideas?

January 23, 2012 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Finally some progress!

• Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.
• Backported local pkgsrc patches to ATF into the upstream code.
• Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.
• Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua. Not very successful yet unfortunately.

January 15, 2012 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time!

January 10, 2012 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Happy new year!

No activity past week as I was completely away from the computer.

January 3, 2012 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

The post title should mention ATF instead of Kyua... but I'm keeping the usual one for consistency:

• Integrated timestamps into the XML and HTML reports generated by atf-report. These should soon show up in the continuous tests of NetBSD.
• Work on integrating the use of POSIX timers into atf-run after Christos Zoulas performed these changes in the NetBSD tree. The result is quite awful because I need to keep compatibility with systems that do not provide the "new" interface...

December 26, 2011 · Tags: atf, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, not much activity this week due to travel reasons. Anyway, some work went in:

• Preliminary code to generate HTML reports from kyua report. This is easy peasy but boring. The current code was written as a proof of concept and is awful, hence why it was not committed. I'm now working in cleaning it up.
• Backported test program and test case timestamping into ATF based on a patch from Paul Goyette. This is a very useful feature to have, and it will have to be added to Kyua later. (It has always been planned to be added, but have not had the time yet.)

December 19, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Some significant improvements this week:

• Finally submitted the code to store and load full test case definitions. This is quite tricky (and currently very, very ugly) but it works and it will allow the reports to include all kinds of information from the test cases.
• Removed the Atffiles from the tree; yay! For a long time, I had been using atf-run to run broken tests because atf-run allowed me to watch the output of the test case being debugged. However, this has been unnecessary since the introduction of the debug command in late August. I now feel confident that these files can go. (And debug is much more powerful than atf-run because you can target a single test case instead of a whole test program.)
• Some crazy work attempting to hide the name of SQLite types from the sqlite::statement interface. I've been only able to do so somewhat decently for bind, but all my attempts at doing the same with column result in horrible code so far. So no, such changes have not been submitted.
• As of a few minutes ago, kyua test now records the output of the test cases (stdout and stderr) into the database. These will be invaluable for debugging of test cases, particularly when the reports are posted online.
• Some preliminary work at implementing HTML reports. This, however, has not received much progress due to the previous item requiring completion.

December 11, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Some more work towards allowing storing and loading of full test case definitions (which in turn will allow us to provide detailed HTML reports). I have local changes that do this, but they are gated by the lack of some additional tests and probably some optimizations, because they slow down kyua report significantly. Regarding commits, I have only submitted some related cleanup-changes.
• I got distracted by invalid Doxygen comments and traced down why they were not being correctly validated (which is the whole point of running Doxygen during the build). It turns out I had to enable some extra EXTRACT_* settings in the Doxyfile. After doing so, I realized there are many, many documentation problems in the code... and I have been fixing them since. It's a tough operation, but I'm more than half-way through already. To give you an idea, the current diffstat shows about 650 new comment lines (!).

December 5, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes.

Due to this, I ended up doing some cleanups. First, I reimplemented the way in which test programs that fail to list their test cases are represented. And second, I got rid of the useless test_case_id class, which exposed even further problems in the data types that represent test cases.

It's now time to sit and think if the current representations of test programs and test cases make sense and, if not, how to better redo them. Not going to be easy, but I hope to have some time for this cleanup during this upcoming week.

November 28, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori.

$ kyua test

[... wait for the tests to run ...]

Committed action 82

$ kyua report
===> Skipped tests
[... too long to show ...]
===> Expected failures
integration/cmd_report_test:output__console__change_file  ->
    expected_failure: --output not implemented yet:
    atf-check failed; see the output of the test for details
===> Failed tests
store/transaction_test:put_test_case__ok  ->
    failed: Line 663: stmt.step() not met
===> Summary
Action: 82
Test cases: 934 total, 15 skipped, 1 expected failures,

0 broken, 1 failed

November 20, 2011 · Tags: kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept.

November 14, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database.

"Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results. It's interesting to realize that most of these features are just one SQL query away. If you install Kyua, you can already run a few tests and then use kyua db-exec to issue arbitrary SQL queries against the database; the schema (see store/schema.sql) might look a bit convoluted, but a bunch of NATURAL JOINs will yield the desired output.

The feature requests that have the highest priority at this point are the ability to generate a report of the last tests run both as a text file and as an HTML dashboard, because having these features means we can finally kill the atf-run and atf-report pair. At this point I'm, once again, "stuck" while figuring out how to best organize the code to make all these things possible while still keeping a nice separation across the existing layers (cli, engine and store)... all without introducing much unnecessary complexity. But exciting times lie ahead!

November 7, 2011 · Tags: kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Submitted a placeholder implementation of the new persistence layer (store top-level directory). This only supports opening a database and ensuring its metadata is valid. See r253.
• Added a db-exec CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet. See r255.
• Miscellaneous fixes and improvements to utils::env and utils::sqlite.
• Preliminary code to support putting objects (like actions and contexts) into the database. I've been thinking about this for a while and finally came up with a design that completely decouples the persistence needs from the higher-level classes in the engine layer. I haven't submitted the code yet though, as it lacks tests. (Still thinking how to write the loading of objects though.)

October 31, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.
• Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.
• After this refactoring, the code left in the CLI modules is purely presentation-related, and the code in the engine implements all the logic.
• The goal of these changes is to be able to hide the interactions with the database in these controllers. The CLI layer has no business in dealing with the database connection (other than allowing the user to specify which database to talk to, of course).
• Implemented a very simple RAII model for SQLite transactions.
• Some additions to the utils::sqlite bindings to simplify some common calling patterns (e.g. binding statement parameters by name).
• Preliminary prototypes at database initialization. This involves creating new databases and populating them with the initial schema, plus dealing with database metadata to, e.g. detect if we are dealing with the correct schema version.
• The code for this is still too crappy to be submitted, so don't look for it in the repository just yet!
• The design document details many things that should be part of the schema (e.g. "sessions"0, but I've decided that I'll start easy with a simplified schema and later build on top of it. Otherwise there will be too many clunky moving parts to deal with while the fundamental ideas are not yet completely clear.
• Fixes to let the code build and run again in NetBSD (macppc at least).

October 23, 2011 · Tags: kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Added support to prepare and execute statements to utils::sqlite.
• Added the UTILS_PURE definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.
• Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.
• Made a quick attempt at getting kyua test record test results in a simple database; success! This was just a proof of concept, so it is not submitted yet.
• Started to refactor the code to move many logic constructions from the cli to the engine. With the need to store test results in the database, it's clear that these semantics do not belong in the CLI, but the current code structure do not easily allow this. Need to add some "controller" classes in the engine to hide all the interaction among internal components.

October 16, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Some more reading on SQLite. The Using SQLite book is close to awesome; very easy to read and full of insightful tips.
• Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep procrastinating. So, ladies and gentlemen, welcome the new utils::sqlite module in Kyua. At the moment, this just provides the barebones to open and close a database. I need to start playing around with preparing statements before I can model these in my wrapper classes. (Yes, the API structure is extremely similar to that in Lutok.)

October 9, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Unfortunately, not much time this week either :-(
• I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases.  However, I've not had the to time to finish these edits and publish them.

October 3, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, no activity this week other than some brainstorming on the database design.

Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!

September 26, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report (db designdoc edition!)

• Moved the code of utils::lua to a new project, Lutok.
• Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.
• Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.
• Been chewing through the first chapters of the "Using SQLite" book to refresh my SQL skills.
• And, most importantly, wrote a preliminary design document for the database store of Kyua and the reporting features. Comments certainly welcome! Be aware that this is how atf-report will be replaced, so once this is done we should be able to finally kill atf-run and atf-report altogether :-)

September 18, 2011 · Tags: kyua, lutok, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(
• On another order of things, I have started writing a design document for the database that will collect test case results and other information. Will share it as soon as it is readable and more or less complete.

September 12, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Slow week.  We had some team-related events at work and I have friends over from Dublin, so it has been hard to find some time to do work on Kyua.  Regardless, here comes the weekly report:

• Split utils::lua into its own package, per some user's request.  I'm still setting up the separate project and have to do lots of cleanup on the code, so nothing is available yet.
• Started experimenting on the long promised "tests results database".  So far, I have started writing a small utils::sqlite3 C++, RAII-based wrapper for SQLite 3 and a SQL schema for the database.  My rusty SQL skills don't help :-P

September 4, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Not a very active week: I've been on-call four days and they have been quite intense. Plus I have had to go through a "hurricane" in NYC. That said, I had some time to do a bit of work on Kyua and the results have been nice :-)

• Made calls to getopt_long(3) work with GNU Getopt by using the correct value of optind to reset option processing.
• Improved the configure script to error out in a clearer way when missing dependencies (pkg.m4 and Lua) are not found.
• Did some portability fixes.
• And released Kyua 0.2! (along with a pkgsrc package)
  

August 28, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Implemented the "debug" command. Still very rudimentary, this command allows the user to run a test case without capturing its stdout nor stderr to aid in debugging of failed test cases. In the future, this command will also allow things like keeping the work directory for manual inspection, or spawning a shell or a debugger in the work directory after a test case is executed.
• Many build fixes under different platforms in preparation for a 0.2 release. In particular, Kyua now builds under Ubuntu 10.04.1 LTS but some tests fail.
• Had to disable the execution of the bootstrap test suite within Kyua because it stalls in systems where the default shell is not bash.  I presume this is a bug in GNU Autotest, so I filed a report.

August 21, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Changed the --config and --variable options to be program-wide instead of command-specific. The configuration file should be able to hold properties that tune the behavior of Kyua, not just the execution of tests, so this makes sense.
• Added the config subcommand, which provides a way to inspect the configuration as read by Kyua.
• Got rid of the test_suites_var function from configuration files and replaced it by simple assignments to variables in the test_suites global table.
• Enabled detection of unused parameters by the compiler and fixed all warnings.
• Changed developer mode to only control whether warnings are enforced or not (not to enable the warnings themselves) and made developer mode be disabled on formal releases.

August 15, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Added the ability to explicitly define timeouts for plain test programs. This just completes the work from past week that made running plain test programs at all but had an ugly TODO in it to implement this missing feature.
• The bootstrap test suite now runs as a single test case within the whole Kyua test suite. Demonstrates the plain test programs interface functionality :-)
• Started reshuffling code to make the <tt>--config</tt> and related flags program-wide. The goal is to allow the configuration file to tune the behavior of all of Kyua, so these flags must be made generic. They were previously specific to the <tt>test</tt> subcommand only.

August 8, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Implemented the "plain" test interface. This allows plugging "foreign test programs" into a Kyua-based test suite. (A foreign test program is a program that does not use any testing framework: it reports success and failure by means of an exit code.)
• Generalized code between the atf and plain interfaces and did some cleanups.
• Attempted to fix the ATF_REQUIRE_EQ macros in ATF to evaluate their arguments only once. This has proven to be tricky and therefore it is not done yet. At the moment, the macros reevaluate the arguments on a failure condition, which is not too big of a deal.

August 1, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

• Finished splitting the atf-specific code from the generic structures in the engine. The engine now supports the addition of extra test interfaces with minimal effort.
• Started implementing a "plain" test interface for test programs that do not use any test framework. This is to allow muxing non-atf tests into atf-based test cases, which is required in the NetBSD tree.

July 25, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Slow week. I've been busy moving to NYC!

• Kept working on the splitting of ATF-specific code from the core test abstractions. The work is now focused on refactoring the results-reporting pieces of the code, which are non-trivial.

July 18, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

One of the major features I want in place for Kyua 0.2 is the ability to run "foreign" test programs as part of a test suite: i.e. to be able to plug non-ATF test programs into a Kyuafile. The rationale for this is to lower the entry barrier of newcomers to Kyua and, also, to allow running some foreign test suites that exist in the NetBSD source tree but that are currently not run.

The work this week has gone in the direction outlined above among other things:

• Created an abstract base class to represent test programs and provided an implementation for ATF.
• Did the same thing for test cases.
• Moved the kyua-cli package from pkgsrc-wip into pkgsrc head. Installing Kyua is now a breeze under NetBSD (and possibly under other platforms supported by pkgsrc!)

July 10, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Belated update:

July 5, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This has been the big week:

• Wrote some user documentation for the kyua binary.
• Fixed some distcheck problems.
• Released Kyua 0.1!

June 26, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

A couple of things have happened:

• Released ATF 0.14. This release was followed by an import into NetBSD and fixing of subsequent fallout.
• Some performance improvements to atf-sh. After killing a bunch of complex shell constructions and removing lots of obsolete functions, the performance results are significant. There is still room for improvement of course, and I still need to quantify how these optimizations behave in single-core machines.

June 19, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

June 13, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Some long-standing bug fixes / improvements have gone in this week:

• Improvements to the cleanup routine, which is used to destroy the work directory of a test case after the test case has terminated:
• Heavy refactoring to be tolerant to failures. These failures may arise when a child of the test case does not exit immediately and holds temporary files in the work directory open for longer than expected.
• Any file systems that the test case leaves mounted within the work directory will now be unmounted, just as the ATF test interface mandates. I realize that this adds a lot of complexity to the runtime engine for very little gain. If/when we revise the tests interface, it will be worth to reconsider this and maybe leave the cleanup of mounted file systems to the test case altogether.
• As a result, issue 17 has been fixed!
• Kyua now captures common termination signals (such as SIGINT) and exits in a controlled manner. What this means is that Kyua will now kill any active test programs and clean up any existing work directories before exiting. What this also means is that issue 4 is fixed.

June 5, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

Some cool stuff this week, albeit not large-scale:

• Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.
• Added support for several environment variables to allow users (and tests) to override built-in paths. I can't imagine right now any legitimate use for these variables, but hardcoded values are bad in general, atf-run provided these same variables, and these variables are very handy for testing purposes.
• Added support for the new require.files test-case metadata property to both ATF and Kyua. This new property allows tests to specify a set of files that they require in order to run, and is useful for those tests that can't run before make install is executed.
  

May 29, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This week:

May 22, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report, BSDCan 2011 edition

I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers.

Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them. (Video coming soon, I hope.)

The talk was later followed by several questions and off-session conversations about testing in general in BSDs. Among these, I gathered a few random ideas / feelings:

• The POSIX 1003.3 standard defines the particular results a test can emit (see the corresponding DejaGnu documentation). Both ATF and Kyua already implement all the results defined in the standard, but they use different names and extend the standard with many extra results. Given that the standard does not define useful concepts like "expected failures", an idea that came up is to provide a flag to force POSIX compliance at the cost of being less useful. Why? Just for the sake of saying that Kyua conforms to this standard.
• The audience seemed to like the idea of a "tests results store" quite a bit, and the sound of SQLite for the implementation was not bullied. This is something I'm eager to work on, but not before I publish a 0.1 release.
• I highlighted the possibility of allowing Kyua to run "foreign" test programs so that we could integrate the results into the database. This could be useful to run tests for which we (*BSD) have no control (e.g. gcc) in an integrated manner. The idea was not bullied by anyone either.
• FreeBSD has already been looking at ATF / Kyua and they are open to collaboration.
• OpenBSD won't import any new C++ code, and adding C-based tests to the tree while relegating the C++ runtime to the ports is not an option. Somehow I expected this.
• Junos (the FreeBSD-based operating system from Jupiter Networks) recently imported ATF and they are happy with it so far. Yay!
• Would be nice to have a feature to run tests remotely after, maybe, deploying a single particular test and its dependencies. This is gonna be tricky and not in my current immediate plans.

May 16, 2011 · Tags: atf, conference, kyua, report
Continue reading (about 3 minutes)

Kyua: Weekly status report

Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow.

May 8, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Unfortunately, I have had no time for coding this week. The only things I could do were:

May 1, 2011 · Tags: kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

Ouch; I'm exhausted. I just finished a multi-hour hacking session to get the implementation of the list subcommand in control. It is now in a very nice user-facing shape, although its code deserves a little bit of house cleaning (coming soon).

Anyway, this week's progress:

• Added the kyuaify.sh script. This little tool takes a test suite (say, NetBSD's /usr/tests directory) and converts all its Atffiles into Kyuafiles. The tool is no sophisticated at all; in fact, it is a pretty simple script that I haven't tested with any other test suites so far. See the announcement of kyuaify for some extra details.
• Added logging support for the Lua code and changed the Lua modules to spit some logging information while processing Kyuafiles and configuration files.
• Added a mechanism in the user interface module to consistently print informational, warning and error messages.
• Implemented proper test filtering (after several iterations). What does proper mean? Well, for starters, test filters are always relative to the test suite's root (although we already saw this in last week's report). But the most important thing is that the filters are now validated: nice, user-friendly errors will be reported when the collection of tests is non-disjoint, when it includes duplicate names or when any of the provided filters does not match any test case. I really need to document the rationale of these in the manual, but for now the r118 commit message includes a few details.
• Drafted some notes for the BSDCan 2011 conference. I am quite tempted to reuse parts of the presentation from NYCBSDCon 2010, but I really want to give more emphasis on Kyua this time. In case you don't know, Kyua was first announced in NYCBSDCon 2010 and it was still a very immature project. The project has changed a lot since then.

April 24, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

• Added integration tests for the about and help subcommands. These were pretty easy to do.
• Added integration tests for the list subcommand. I initially added these tests as expected failures to reason about the appearance and behavior of this command from the point of view of the user before actually working on the code... and I am still writing such code to make these tests pass!

April 17, 2011 · Tags: atf, kyua, report
Continue reading (about 3 minutes)

Kyua: Weekly status report

Few things worth mentioning this week as reviewing Summer of Code student applications has taken priority. The good thing is that there are several strong applications for NetBSD; the bad thing is that none relate directly to testing. Anyway, the work this week:

• Added a pkg-config file for atf-sh as well as an Autoconf macro to detect its presence. This is needed by Kyua to easily find atf-sh. (Yes, I know: this is an abuse of pkg-config, but it works pretty well and is consistent with atf-c and atf-c++.)
• Implemented basic integration tests for Kyua in r98 using atf-sh. These tests are still very simple but provide a placeholder into which new tests will be plugged. Having good integration test coverage is key in preparation for a 0.1 release. Oh, and by the way, this revision has bumped the number of tests to 601, crossing the 600 barrier :-)

April 10, 2011 · Tags: atf, kyua, report
Continue reading (about 1 minute)

Kyua: Weekly status report

This week's work has been quite active on the ATF front but not so much in the Kyua one. I keep being incredibly busy on the weekends (read: traveling!) so it's hard to get any serious development work done.

What has happened?

• Finally tracked down and fixed some random atf-run crashes that had been hunting the NetBSD test suite for months (see PR bin/44176). The fix is in reality an ugly workaround for the fact that a work directory cannot be considered "stable" even after the test case terminates. There may be dying processes around that touch the work directory contents, and the cleanup code in atf-run was not coping well with those. As it turns out, this problem also exists in Kyua (even though it's not as pronounced because arbitrary failures when running a test case do not crash the runtime engine) so I filed issue 17 to address it.
• Released ATF 0.13 and imported it both to NetBSD-current and pkgsrc. As a side note, Kyua requires the new features in this release, so putting it out there is a requirement to release Kyua 0.1. This new release does not have a big effect on NetBSD though, because the copy of ATF in NetBSD has been constantly receiving cherry-picks of the upstream fixes.
• Replaced several TODO items in the Kyua code with proper calls to the logging subsystem. These TODO items were referring to conditions in the code that should not happen, but for which we cannot do any proper recovery (like errors in a destructor). Sure, these could be better signaled as an assertion... but these code paths can be triggered in extremely-tricky conditions and having Kyua crash because of them is not nice (particularly when the side-effects of executing that code paths are non-critical).

April 3, 2011 · Tags: atf, kyua, report
Continue reading (about 2 minutes)

Kyua: Weekly status report

This has been a slow week. In the previous report, I set the goal of getting Kyua to run the NetBSD test suite accurately (i.e. to report the same results as atf-run), and this has been accomplished. Actually, the changes required in Kyua to make this happen were minimal, but I got side-tracked fixing issues in NetBSD itself (both in the test suite and in the kernel!). So, the things done:

• Fixed Kyua to correctly kill any dangling subprocesses of a test case and thus match the behavior of atf-run. This was the only change required to make issue 16 happen: i.e. to get Kyua to report the same results as atf-run for the NetBSD test suite.
• Based on a suggestion from Antti Kantee, an alternative way to handle this would be to not kill any processes and just report the test case as broken if it fails to clean itself up. The rationale being that the runtime engine can kill dangling subprocesses in 99% of the occasions, but not always. The exception are those subprocesses that change their process group. It'd be better to make all cleanups explicit instead of hiding this corner case, as it can lead to confusion. Addressing this will have to wait though, as it is a pretty invasive change.
• Before closing issue 16, I want to implement some integration tests for Kyua to ensure that the whole system behaves as we expect (which is what the NetBSD test suite is currently doing implicitly).
• Kyua is pickier than atf-run: if a cleanup routine of a test case fails or crashes, Kyua will (correctly) report the test case as broken while atf-run will silently ignore this situation. Some NetBSD tests had crashing cleanup parts, so I fixed them.
• Some test programs in NetBSD were leaving unkilled subprocesses behind. These subprocesses are daemons and thus fall out of the scope of what Kyua can detect and kill during the cleanup phase. I mistakenly tracked down the problem to rump, but Antti Kantee kindly found the real problem in the kernel (not in rump!).
• As a side effect of processes being left behind, I extended the functionality of pidfile(3) and implemented pid file support in bozohttpd.  This is to make the tests that spawn a bozohttpd in the background more robust, by giving them a way to forcibly kill the server during cleanup.
• These changes are still under review and not committed yet.

March 27, 2011 · Tags: atf, kyua, report
Continue reading (about 3 minutes)

Kyua: Weekly status report

These days, I find myself talking about Kyua to "many" people. In particular, whenever a new feature request for ATF comes in, I promise the requester that the feature will be addressed as part of Kyua. However, I can imagine that this behavior leaves the requester with mixed feelings: it is nice that the feature will be implemented but, at the same time, it is very hard to know when because the web site of Kyua does not provide many details about its current status.

In an attempt to give Kyua some more visibility, I will start posting weekly activity reports in this blog. These reports will also include any work done on the ATF front, as the two projects are highly related at this point. I write these reports regularly at work and I feel like it is a pretty good habit: every week, you have to spend some time thinking about what you did for the project and you feel guilty if the list of tasks is ~zero ;-) It also, as I said, gives more visibility to the work being done so that outsiders know that the project is not being ignored.

Before starting with what has happened this week, a bit of context. I have been traveling like crazy and hosting guests over for the last 2 months. This has given me virtually no time to work on Kyua but, finally, I have got a chance to do some work this past week.

So, what are the news?

• Implemented the --loglevel command line flag, which closes issue 14. Kyua now generates run-time logs of its internal activity to aid in postmortem debugging and this flag allows the user to control the verbosity of such logs.
• Antti Kantee hacked support for atf-run in the NetBSD source tree to dump a stack trace of any crashing test program. I have backported this code to the upstream ATF code and filed issue 15 to implement this same functionality in Kyua.
• Fixed a hang in atf-run that made it get stuck when a test case spawned a child processes and atf-run failed to terminate it. A quick test seems to indicate that Kyua is affected by a similar problem: it does not get stuck but it does not correctly kill the subprocesses. The problem will be addressed as part of issue 16.
• Oh, and by the way: Kyua will be presented at BSDCan 2011.

March 20, 2011 · Tags: atf, kyua, report
Continue reading (about 3 minutes)