Offloading maintenance tasks to Travis CI
From day one, the Kyua source tree has had docstring annotations for all of its symbols. The goal of such docstrings is to document the code for the developers of Kyua: these docstrings were never intended to turn into pre-generated HTML documentation because Kyua does not offer an API once installed. As you might have noticed, Doxygen is an optional component of the build and it used to run on each make invocation.
May 21, 2015
·
Tags:
automation, ci, kyua, travis
Continue reading (about
8 minutes)
Kyua turns parallel
After three months of intensive work on Kyua's executor Git branch, I am happy to announce that the new execution engine, whose crown feature is the ability to run test cases in parallel, has just landed in master and passes all self-tests! You can head over to the commit message for more details on the merge, read the NEWS entries, and skim throught the history of the executor branch to understand how this feature has been built.
February 28, 2015
·
Tags:
announce, kyua
Continue reading (about
5 minutes)
Refocusing Kyua maybe?
The FreeBSD devsummit that just passed by gave me enough insight into Jenkins to question the long-term plans for Kyua. Uh, WHAT?! Let me explain. In the beginning... One of the original but unstated goals of Kyua was to fix the "mess" that is the NetBSD releng test run logs site: if you pay close attention, you will notice that various individuals have reinvented the wheel over and over again in an attempt to automate release builds and test suite runs.
May 23, 2014
·
Tags:
freebsd, jenkins, kyua, netbsd
Continue reading (about
6 minutes)
How to merge multiple Git repositories into one
Are you looking for a method to merge multiple Git repositories into a single one? If so, you have reached the right tutorial! Please bear with me for a second while I provide you with background information and introduce the subject of our experiments. We’ll get to the actual procedure soon and you will be able to apply it to any repository of your choice. In the Kyua project, and with the introduction of the kyua-atf-compat component in the Summer of 2012, I decided to create independent Git repositories for each component.
February 15, 2014
·
Tags:
featured, git, kyua
Continue reading (about
4 minutes)
FreeBSD Test Suite tutorial at AsiaBSDCon 2014
I am pleased to announce that the tutorial on the FreeBSD Test Suite that I proposed for AsiaBSDCon 2014 has been accepted! The conference website will soon include more details, but allow me to spoil your wait: Goals: Learn how to use the test suite, how it is internally organized and how new tests can be written. Stretch goal: Get attendees to contribute one or more tests to the project. Audience: Mostly developers of FreeBSD that want to learn how the new test suite plugs into the system.
January 8, 2014
·
Tags:
conference, freebsd, kyua, testing
Continue reading (about
1 minute)
FreeBSD Test Suite goals and planning
I was really impressed and delighted to see how popular my previous (and first) post on FreeBSD testing, titled Introducing the FreeBSD Test Suite, was. Looks like this project may be of great interest to developers and users out there (not unsurprisingly) so I'll try to keep you all up-to-date with any key developments. A first question that arises from the announcement is: where are the test suite and infrastructure headed?
January 5, 2014
·
Tags:
freebsd, kyua, testing
Continue reading (about
2 minutes)
Introducing the FreeBSD Test Suite
I joined the FreeBSD committer ranks a couple of months ago with the intention to equip FreeBSD with an out-of-the-box test suite and with a testing infrastructure. The time until now has been quite fruitful and I have been rushing to get something ready for you before the year end. With that, I am very pleased to announce that the first mockup of the FreeBSD testing cluster is up and running!
December 31, 2013
·
Tags:
featured, freebsd, kyua, testing
Continue reading (about
4 minutes)
lutok-0.3, kyua-testers-0.2 and kyua-cli-0.8
Yesterday was release day: I pushed out Lutok 0.3, Kyua Testers 0.2 and Kyua CLI 0.8. There are not a lot of changes in these new releases. The reason I cut them was to publish the new TAP-compliant tester and make it available for use in FreeBSD as soon as possible. I will be using this new feature as part of the FreeBSD Test Suite in order to hook existing test programs without having to rewrite them to use the ATF libraries (or at least not as a first step).
December 8, 2013
·
Tags:
kyua, lutok, release
Continue reading (about
2 minutes)
Three productive days on the Kyua front
This being Thanksgiving week in the U.S. and Google giving us Thursday and Friday off, I decided to take Monday to Wednesday off as well to spend some time hacking on Kyua — yes, finally, after months of being inactive. And what a three productive days! Here comes a little briefing on the three fronts in which I made progress. (This puts on hold the header files series until next Monday.
November 28, 2013
·
Tags:
atf, fedora, freebsd, kyua, lua
Continue reading (about
6 minutes)
EuroBSDCon 2013 takeaways
EuroBSDCon 2013 is done. If you have been following my daily posts over the last 4 days (day 1, day 2, day 3 and day 4) as well as #EuroBSDCon updates in Twitter, you may already have a pretty good idea of what went on here. However, with the conference over, it is now a good time to recap the whole event and present the takeaways of these four days which, overall, were quite interesting and productive.
September 30, 2013
·
Tags:
eurobsdcon, freebsd, kyua, netbsd
Continue reading (about
6 minutes)
Live from EuroBSDCon 2013, day 4
Live from Malta today attending the EuroBSDCon 2013 conference. The conference is over; today was the second and last day and it has just finished. Hardware and virtualization One of the three tracks today included a lot of talks on hardware, porting of BSDs to new hardware and virtualization techniques. Of all these, the few talks I attended covered the topics in great detail and proved to be very interesting.
September 29, 2013
·
Tags:
conference, eurobsdcon, kyua, virtualization
Continue reading (about
4 minutes)
Live from EuroBSDCon 2013, day 3
Live from Malta today attending the EuroBSDCon 2013 conference. Today is the first day of the conference itself. Many more people have shown up as expected and there have been tons of very interesting talks all the time. It is both good and bad that there are several tracks: you can select the topic you are most interested in, but sometimes great talks overlap! Keynote Today's opening session was led by Theo de Raadt, the founder of OpenBSD.
September 28, 2013
·
Tags:
conference, eurobsdcon, kyua, netbsd
Continue reading (about
4 minutes)
Setting up my old Mac Mini G4 as a development machine
I've spent quite a few time last week setting up my old Mac Mini G4 — a PPC 1.2GHz with 1GB of RAM running NetBSD/macppc current — as a "workstation" for the development of Kyua and other tools for NetBSD. Yes, this machine is very slow, but that's the whole point of the exercise I'm going to narrate below. I recently got approval from the NetBSD core team to import Kyua into the NetBSD source tree and replace ATF with it.
October 22, 2012
·
Tags:
kyua, lab-notes, mac, netbsd
Continue reading (about
3 minutes)
Kyua gets its own blog
For the last couple of weeks, I have been pondering the creation of a Kyua-specific blog. And, after a lot of consideration, I have finally taken the plunge. Say hello to Engineering Kyua! From now on, all Kyua-related posts (as well as ATF posts) will go to the new blog. I recommend you to subscribe to Engineering Kyua's Atom feed right now to not miss a beat! If you care enough about Kyua, that is.
June 4, 2012
·
Tags:
blog, kyua
Continue reading (about
3 minutes)
Exposing a configuration tree through Lua
In the previous post, I discussed the type-safe tree data structure that is now in the Kyua codebase, aimed at representing the configuration of the program. In this post, we'll see how this data structure ties to the parsing of the configuration file. One goal in the design of the configuration file was to make its contents a simple key/value association (i.e. assigning values to predetermined configuration variables). Of course, the fact that the configuration file is just a Lua script means that additional constructions (conditionals, functions, etc.
June 2, 2012
·
Tags:
cxx, kyua, lua
Continue reading (about
3 minutes)
Type-safe, dynamic tree data type
The core component of the new configuration library in Kyua is the utils::config::tree class: a type-safe, dynamic tree data type. This class provides a mapping of string keys to arbitrary types: all the nodes of the tree have a textual name, and they can either be inner nodes (no value attached to them), or leaf nodes (an arbitrary type attached as a value to them). The keys represent traversals through such tree, and do this by separating the node names with dots (things of the form root.
May 29, 2012
·
Tags:
cxx, kyua
Continue reading (about
5 minutes)
Rethinking Kyua's configuration system
In the previous blog post, I described the problems that the implementation of the Kyua configuration file parsing and in-memory representation posed. I also hinted that some new code was coming and, after weeks of work, I'm happy to say that it has just landed in the tree! I really want to get to explaining the nitty-gritty details of the implementation, but I'll keep these for later. Let's focus first on what the goals for the new configuration module were, as these drove a lot of the implementation details:Key/value pairs representation: The previous configuration system did this already, and it is a pretty good form for a configuration file because it is a simple, understandable and widespread format.
May 28, 2012
·
Tags:
cxx, kyua, lua
Continue reading (about
4 minutes)
Kyua's configuration system showing its age
A couple of years ago, when Kyua was still a newborn, I wrote a very ad-hoc solution for the parsing and representation of its configuration files. The requirements for the configuration were minimal, as there were very few parameters to be exposed to the user. The implementation was quick and simple to allow further progress on other more-important parts of the project. (Yep, quick is an euphemism for dirty: the implementation of the "
May 26, 2012
·
Tags:
cxx, kyua
Continue reading (about
3 minutes)
Kyua gets nicer console messages
For the last couple of weeks, particularly during a bunch of long flights, I have been improving the command-line user interface of Kyua by implementing controlled line wrappings on screen boundaries: messages that are too long to fit on the screen are preprocessed and split into multiple lines at word boundaries. This affects informational messages, error messages and, specially, the output of the built-in help command. I originally got this idea from Monotone and later implemented it into ATF but, when writing Kyua's code, I decided to postpone its implementation until a later stage.
April 2, 2012
·
Tags:
kyua
Continue reading (about
4 minutes)
Kyua generates its first public HTML report
Lately, three long trips (5 hours in a bus, and 6 and 10 hours in two planes) have allowed me to work on the long-promised HTML reporting feature of Kyua. The result of these three trips is, effectively, the ability to generate HTML reports for specific test actions! The current results are extremely rudimentary (they lack tons of would-be-useful information) and not that aesthetically pleasing. However, the database already records enough information to make these reports more useful and pretty, so "
March 13, 2012
·
Tags:
kyua
Continue reading (about
2 minutes)
Projects migrated to Git
I finally took the plunge. Yesterday night, I migrated the Kyua and Lutok repositories from Subversion to Git. And this morning I migrated ATF from Monotone and custom hosting to Git and Google Code; oh, and this took way longer than expected. Migration of Kyua and Lutok Migrating these two projects was straightforward. After preparing a fresh local Git repository following the instructions posted yesterday, pushing to Google Code is a simple matter:
February 26, 2012
·
Tags:
atf, git, kyua, lutok
Continue reading (about
3 minutes)
Kyua 0.3 released!
Dear readers, I am pleased to announce that Kyua 0.3 is available! The major feature in this release is the introduction of the "test results store"; i.e. a SQLite3-based database that records all test execution activity so that reports can be gathered in a central machine and reports generated out of it. This is still very experimental and the generated reports are quite rudimentary, but is a first step towards this direction.
February 24, 2012
·
Tags:
kyua, release
Continue reading (about
1 minute)
Kyua: Weekly status report
A couple of things happened this week: Spent quite a few time researching the idea of moving away from Monotone and Subversion to Git. I haven't made a decision yet, but I'm pretty convinced this is the right way to go. It will simplify development significantly, it will allow me to code offline (have a bunch of really long flights coming), and it will lower the entry barrier to Kyua by making all components use the same, mainstream VCS.
February 13, 2012
·
Tags:
kyua, report
Continue reading (about
1 minute)
Switching projects to Git
The purpose of this post is to tell you the story of the Version Control System (VCS) choices I have made while maintaining my open source projects ATF, Kyua and Lutok. It also details where my thoughts are headed to these days. This is not a description of centralized vs. distributed VCSs, and it does not intend to be one. This does not intend to compare Monotone to Git either, although you'll probably feel like it while reading the text.
February 11, 2012
·
Tags:
atf, git, kyua, lutok, monotone, vcs
Continue reading (about
8 minutes)
Kyua: Weekly status report
Created an RPM package for Lutok for inclusion in Fedora.Created a preliminary RPM spec for ATF for Fedora. Now in discussions with the FPC to figure out how to install the tests on a Fedora system, as /usr/tests may not be appropriate.No activity on Kyua itself though, unfortunately.
February 7, 2012
·
Tags:
atf, kyua, lutok, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This comes two days late... but anyway, I felt like posting it now instead of waiting until next Sunday/Monday. The activity past week focused mostly on implementing support for the require.memory test-case metadata property recently introduced into ATF. This was non-trivial due to the need to write some tricky Autoconf code to make this "slightly portable". Seriously: It's scary to see how hard it is to perform, in a portable manner, an operation as simple as "
February 1, 2012
·
Tags:
kyua, lutok, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Released ATF 0.15 and imported it into NetBSD.Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them.
January 23, 2012
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Finally some progress! Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.Backported local pkgsrc patches to ATF into the upstream code.Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua.
January 15, 2012
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time!
January 10, 2012
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Happy new year! No activity past week as I was completely away from the computer.
January 3, 2012
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, not much activity this week due to travel reasons. Anyway, some work went in: Preliminary code to generate HTML reports from kyua report. This is easy peasy but boring. The current code was written as a proof of concept and is awful, hence why it was not committed. I'm now working in cleaning it up.Backported test program and test case timestamping into ATF based on a patch from Paul Goyette.
December 19, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Some significant improvements this week: Finally submitted the code to store and load full test case definitions. This is quite tricky (and currently very, very ugly) but it works and it will allow the reports to include all kinds of information from the test cases.Removed the Atffiles from the tree; yay! For a long time, I had been using atf-run to run broken tests because atf-run allowed me to watch the output of the test case being debugged.
December 11, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Some more work towards allowing storing and loading of full test case definitions (which in turn will allow us to provide detailed HTML reports). I have local changes that do this, but they are gated by the lack of some additional tests and probably some optimizations, because they slow down kyua report significantly. Regarding commits, I have only submitted some related cleanup-changes.I got distracted by invalid Doxygen comments and traced down why they were not being correctly validated (which is the whole point of running Doxygen during the build).
December 5, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes.
November 28, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori. At the moment, the report is just some disorganized plain-text. For example:
November 20, 2011
·
Tags:
kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept.
November 14, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database. "Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results.
November 7, 2011
·
Tags:
kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Submitted a placeholder implementation of the new persistence layer (store top-level directory). This only supports opening a database and ensuring its metadata is valid. See r253.Added a db-exec CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet.
October 31, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.
October 23, 2011
·
Tags:
kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Added support to prepare and execute statements to utils::sqlite.Added the UTILS_PURE definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.Made a quick attempt at getting kyua test record test results in a simple database; success!
October 16, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Some more reading on SQLite. The Using SQLite book is close to awesome; very easy to read and full of insightful tips.Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep procrastinating. So, ladies and gentlemen, welcome the new utils::sqlite module in Kyua. At the moment, this just provides the barebones to open and close a database.
October 9, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, not much time this week either :-(I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases. However, I've not had the to time to finish these edits and publish them.
October 3, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, no activity this week other than some brainstorming on the database design. Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!
September 26, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report (db designdoc edition!)
Moved the code of utils::lua to a new project, Lutok.Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.
September 18, 2011
·
Tags:
kyua, lutok, report
Continue reading (about
1 minute)
Introducing Lutok: A lightweight C++ API for Lua
It has finally happened. Lutok is the result of what was promised in the "Splitting utils::lua from Kyua" web post. Quoting the project web page: Lutok provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors.
September 15, 2011
·
Tags:
announce, cxx, kyua, lua, lutok
Continue reading (about
2 minutes)
Kyua: Weekly status report
My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(On another order of things, I have started writing a design document for the database that will collect test case results and other information.
September 12, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Slow week. We had some team-related events at work and I have friends over from Dublin, so it has been hard to find some time to do work on Kyua. Regardless, here comes the weekly report: Split utils::lua into its own package, per some user's request. I'm still setting up the separate project and have to do lots of cleanup on the code, so nothing is available yet.Started experimenting on the long promised "
September 4, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Splitting utils::lua from Kyua
If you remember a post from January titled C++ interface to Lua for Kyua (wow, time flies), the Kyua codebase includes a small library to wrap the native Lua C library into a more natural C++ interface. You can take a look at the current code as of r129. Quoting the previous post: The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua.
September 3, 2011
·
Tags:
cxx, kyua, lua
Continue reading (about
3 minutes)
Kyua: Weekly status report
Not a very active week: I've been on-call four days and they have been quite intense. Plus I have had to go through a "hurricane" in NYC. That said, I had some time to do a bit of work on Kyua and the results have been nice :-) Made calls to getopt_long(3) work with GNU Getopt by using the correct value of optind to reset option processing.Improved the configure script to error out in a clearer way when missing dependencies (pkg.
August 28, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua 0.2 released
Dear readers, I am very proud to announce the second formal release of Kyua, 0.2. This release comes with lots of internal changes and a few user-visible changes. Instead of listing all the changes and news here, I'll just recommend you to visit the 0.2 release page and read all the notes. This release has been tested under NetBSD-current, Mac OS X Snow Leopard, Debian sid and Ubuntu 10.04.1 LTS.
August 24, 2011
·
Tags:
kyua, release
Continue reading (about
1 minute)
Kyua: Weekly status report
Implemented the "debug" command. Still very rudimentary, this command allows the user to run a test case without capturing its stdout nor stderr to aid in debugging of failed test cases. In the future, this command will also allow things like keeping the work directory for manual inspection, or spawning a shell or a debugger in the work directory after a test case is executed. Many build fixes under different platforms in preparation for a 0.
August 21, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Changed the --config and --variable options to be program-wide instead of command-specific. The configuration file should be able to hold properties that tune the behavior of Kyua, not just the execution of tests, so this makes sense. Added the config subcommand, which provides a way to inspect the configuration as read by Kyua. Got rid of the test_suites_var function from configuration files and replaced it by simple assignments to variables in the test_suites global table.
August 15, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Added the ability to explicitly define timeouts for plain test programs. This just completes the work from past week that made running plain test programs at all but had an ugly TODO in it to implement this missing feature. The bootstrap test suite now runs as a single test case within the whole Kyua test suite. Demonstrates the plain test programs interface functionality :-) Started reshuffling code to make the <tt>--config</tt> and related flags program-wide.
August 8, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Implemented the "plain" test interface. This allows plugging "foreign test programs" into a Kyua-based test suite. (A foreign test program is a program that does not use any testing framework: it reports success and failure by means of an exit code.)Generalized code between the atf and plain interfaces and did some cleanups.Attempted to fix the ATF_REQUIRE_EQ macros in ATF to evaluate their arguments only once. This has proven to be tricky and therefore it is not done yet.
August 1, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Finished splitting the atf-specific code from the generic structures in the engine. The engine now supports the addition of extra test interfaces with minimal effort.Started implementing a "plain" test interface for test programs that do not use any test framework. This is to allow muxing non-atf tests into atf-based test cases, which is required in the NetBSD tree.
July 25, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Slow week. I've been busy moving to NYC! Kept working on the splitting of ATF-specific code from the core test abstractions. The work is now focused on refactoring the results-reporting pieces of the code, which are non-trivial.
July 18, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
One of the major features I want in place for Kyua 0.2 is the ability to run "foreign" test programs as part of a test suite: i.e. to be able to plug non-ATF test programs into a Kyuafile. The rationale for this is to lower the entry barrier of newcomers to Kyua and, also, to allow running some foreign test suites that exist in the NetBSD source tree but that are currently not run.
July 10, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Belated update:Created a pkgsrc package for kyua-cli. Still in pkgsrc-wip though because pkgsrc is in a feature freeze.Wrote a little tutorial on how to run NetBSD tests using Kyua.Started work on 0.2 by doing a minor UI fix in the about command.I've now started to look at how to split the engine into different "runners" to add support for test programs written without ATF. Not that I plan to use this feature.
July 5, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This has been the big week: Wrote some user documentation for the kyua binary.Fixed some distcheck problems.Released Kyua 0.1!The next immediate thing to do is to write a short tutorial on how to run the NetBSD tests with Kyua and get some people to actually try it. After that, there are many things to improve and features to add :-)
June 26, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua 0.1 released!
Dear readers, I'm very proud to announce that the very first release of Kyua, obviously numbered 0.1, has been published! The release page for kyua-cli-0.1 contains more details about what this release offers, and don't hesitate to join the kyua-discuss mailing list and ask if you have questions. Kyua was started sometime in past October and it has taken over six months to get the first public version. (It is true that my free time has heavily fluctuated during this period though, so it does not mean that six months of intensive coding have gone into the release ;-) Kyua 0.
June 23, 2011
·
Tags:
kyua, release
Continue reading (about
1 minute)
Kyua: Weekly status report
A couple of things have happened: Released ATF 0.14. This release was followed by an import into NetBSD and fixing of subsequent fallout.Some performance improvements to atf-sh. After killing a bunch of complex shell constructions and removing lots of obsolete functions, the performance results are significant. There is still room for improvement of course, and I still need to quantify how these optimizations behave in single-core machines.I certainly expected more progress this past week.
June 19, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Added support for recursion from the top-level Kyuafile. This Kyuafile should not reference any directories explicitly because the directories at the top level are supposed to be created by the installation of packages. Closed issue 9.Improved error messages when the test programs are bogus. Closed issue 13.Backported format-printf attribute improvements from NetBSD head to ATF.Miscellaneous build and run fixes for both Kyua and ATF in NetBSD and OS X.Cut a release candidate for atf-0.
June 13, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Some long-standing bug fixes / improvements have gone in this week: Improvements to the cleanup routine, which is used to destroy the work directory of a test case after the test case has terminated:Heavy refactoring to be tolerant to failures. These failures may arise when a child of the test case does not exit immediately and holds temporary files in the work directory open for longer than expected.Any file systems that the test case leaves mounted within the work directory will now be unmounted, just as the ATF test interface mandates.
June 5, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Some cool stuff this week, albeit not large-scale: Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.Added support for several environment variables to allow users (and tests) to override built-in paths.
May 29, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This week:Cleaned up the internal code of the "list" command and added a few unit tests.Added integration tests for the "test" command that focus mostly on the behavior of the "test" command itself. There is still a need for full integration tests that validate the execution of the test cases themselves and their cleanup, and these will be tricky to write.Changed atf-c, atf-c++ and atf-sh to show a warning when a test program is run by hand.
May 22, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report, BSDCan 2011 edition
I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers. Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them.
May 16, 2011
·
Tags:
atf, conference, kyua, report
Continue reading (about
3 minutes)
Kyua: Weekly status report
Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow.
May 8, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, I have had no time for coding this week. The only things I could do were:Fixed a few build problems on NetBSD introduced during past week's changes.Built Kyua and ran a few tests on NetBSD/macppc (just for the joy of it).Coding has been eclipsed by the preparation of my presentation for BSDCan 2011; at this point, this has priority over any code changes. I'd argue that preparing the presentation is also part of the project, so some time has been invested ;-)
May 1, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Ouch; I'm exhausted. I just finished a multi-hour hacking session to get the implementation of the list subcommand in control. It is now in a very nice user-facing shape, although its code deserves a little bit of house cleaning (coming soon). Anyway, this week's progress: Added the kyuaify.sh script. This little tool takes a test suite (say, NetBSD's /usr/tests directory) and converts all its Atffiles into Kyuafiles. The tool is no sophisticated at all; in fact, it is a pretty simple script that I haven't tested with any other test suites so far.
April 24, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
This week started easy:Added integration tests for the about and help subcommands. These were pretty easy to do.Added integration tests for the list subcommand. I initially added these tests as expected failures to reason about the appearance and behavior of this command from the point of view of the user before actually working on the code... and I am still writing such code to make these tests pass!This is where things got a bit awry.
April 17, 2011
·
Tags:
atf, kyua, report
Continue reading (about
3 minutes)
Kyua: Weekly status report
Few things worth mentioning this week as reviewing Summer of Code student applications has taken priority. The good thing is that there are several strong applications for NetBSD; the bad thing is that none relate directly to testing. Anyway, the work this week: Added a pkg-config file for atf-sh as well as an Autoconf macro to detect its presence. This is needed by Kyua to easily find atf-sh. (Yes, I know: this is an abuse of pkg-config, but it works pretty well and is consistent with atf-c and atf-c++.
April 10, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This week's work has been quite active on the ATF front but not so much in the Kyua one. I keep being incredibly busy on the weekends (read: traveling!) so it's hard to get any serious development work done. What has happened? Finally tracked down and fixed some random atf-run crashes that had been hunting the NetBSD test suite for months (see PR bin/44176). The fix is in reality an ugly workaround for the fact that a work directory cannot be considered "
April 3, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
This has been a slow week. In the previous report, I set the goal of getting Kyua to run the NetBSD test suite accurately (i.e. to report the same results as atf-run), and this has been accomplished. Actually, the changes required in Kyua to make this happen were minimal, but I got side-tracked fixing issues in NetBSD itself (both in the test suite and in the kernel!). So, the things done:
March 27, 2011
·
Tags:
atf, kyua, report
Continue reading (about
3 minutes)
Kyua: Weekly status report
These days, I find myself talking about Kyua to "many" people. In particular, whenever a new feature request for ATF comes in, I promise the requester that the feature will be addressed as part of Kyua. However, I can imagine that this behavior leaves the requester with mixed feelings: it is nice that the feature will be implemented but, at the same time, it is very hard to know when because the web site of Kyua does not provide many details about its current status.
March 20, 2011
·
Tags:
atf, kyua, report
Continue reading (about
3 minutes)
Injecting C++ functions into Lua
The C++ interface to Lua implemented in Kyua exposes a lua::state class that wraps the lower-level lua_State* type. This class completely hides the internal C type of Lua to ensure that all calls that affect the state go through the lua::state class. Things get a bit messy when we want to inject native functions into the Lua environment. These functions follow the prototype represented by the lua_CFunction type:typedef int (*lua_CFunction)(lua_State*);Now, let's consider this code:int
January 17, 2011
·
Tags:
cxx, kyua, lua
Continue reading (about
2 minutes)
Error handling in Lua: the Kyua approach
About a week ago, I detailed the different approaches I encountered to deal with errors raised by the Lua C API. Later, I announced the new C++ interface for Lua implemented within Kyua. And today, I would like to talk about the specific mechanism I implemented in this library to deal with the Lua errors. The first thing to keep in mind is that the whole purpose of Lua in the context of Kyua is to parse configuration files.
January 14, 2011
·
Tags:
cxx, kyua, lua
Continue reading (about
3 minutes)
C++ interface to Lua for Kyua
Finally! After two weeks of holidays work, I have finally been able to submit Kyua's r39: a generic library that implements a C++ interface to Lua. The code is hosted in the utils/lua/ subdirectory. From the revision description:The utils::lua library provides thin C++ wrappers around the Lua C API to ease the interaction between C++ and Lua. These wrappers make intensive use of RAII to prevent resource leakage, expose C++-friendly data types, report errors by means of exceptions and ensure that the Lua stack is always left untouched in the face of errors.
January 8, 2011
·
Tags:
cxx, kyua, lua
Continue reading (about
2 minutes)
Using RAII to clean up temporary values from a stack
For the last couple of days, I have been playing around with the Lua C API and have been writing a thin wrapper library for C++. The main purpose of this auxiliary library is to ensure that global interpreter resources such as the global state or the execution stack are kept consistent in the presence of exceptions — and, in particular, that none of these are leaked due to programming mistakes when handling error codes.
December 27, 2010
·
Tags:
c, cxx, kyua, lua
Continue reading (about
6 minutes)
Kyua: Design of the configuration system
Over a week ago, I mostly finished the implementation of the runtime engine for test cases of Kyua and, along the way, realized that it is imperative to write a configuration system right now before the configuration code becomes messier than it already is. To that end, I spent the last week working on a design document for the configuration system. Summarizing, the document describes what the requirements for the configuration files of Kyua are, what the possible alternatives to implement them are, and advocates the use of Lua — a tiny embedded programming language — to bring these configuration files to life.
December 22, 2010
·
Tags:
kyua, lua
Continue reading (about
1 minute)
Introducing Kyua
Wow. I have just realized that I have not blogged at all about the project that has kept me busy for the past two months! Not good, not good. "What is this project?", I hear. Well, this project is Kyua. A bit of background first: the Automated Testing Framework, or ATF for short, is a project that I started during the Summer of Code of 2007. The major goal of ATF was, and still is, to provide a testing framework for the NetBSD operating system.
December 16, 2010
·
Tags:
atf, kyua, netbsd
Continue reading (about
4 minutes)
Featured software
Featured posts
- Fast machines, slow machines
- EndBASIC 0.10: Core language, evolved
- Farewell, Microsoft; hello, Snowflake!
- Rust is hard, yes, but does it matter?
- Rust traits and dependency injection
- A year on Windows: Introduction
- Always be quitting
- How does Google keep build times low?
- How does Google avoid clean builds?
- Unit-testing a console app (a text editor)
- More...
