Kyua: Weekly status report
A couple of things happened this week: Spent quite a few time researching the idea of moving away from Monotone and Subversion to Git. I haven't made a decision yet, but I'm pretty convinced this is the right way to go. It will simplify development significantly, it will allow me to code offline (have a bunch of really long flights coming), and it will lower the entry barrier to Kyua by making all components use the same, mainstream VCS.
February 13, 2012
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Created an RPM package for Lutok for inclusion in Fedora.Created a preliminary RPM spec for ATF for Fedora. Now in discussions with the FPC to figure out how to install the tests on a Fedora system, as /usr/tests may not be appropriate.No activity on Kyua itself though, unfortunately.
February 7, 2012
·
Tags:
atf, kyua, lutok, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This comes two days late... but anyway, I felt like posting it now instead of waiting until next Sunday/Monday. The activity past week focused mostly on implementing support for the require.memory test-case metadata property recently introduced into ATF. This was non-trivial due to the need to write some tricky Autoconf code to make this "slightly portable". Seriously: It's scary to see how hard it is to perform, in a portable manner, an operation as simple as "
February 1, 2012
·
Tags:
kyua, lutok, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Released ATF 0.15 and imported it into NetBSD.Added support for integer/float printf-like modifiers to the utils::format module. These will be required to beautify size and time quantities in the reports and error messages.I spent way more time than I wanted on this. At first, I attempted to use std::snprintf to parse and process the format modifiers for integers and floats so that I could avoid implementing a custom parser for them.
January 23, 2012
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Finally some progress! Backported the require.memory changes in NetBSD to the ATF upstream code, and extended them to support OS X as well.Backported local pkgsrc patches to ATF into the upstream code.Started to prepare ATF 0.15 by doing test runs of NetBSD/i386 and NetBSD/amd64 and by building the code in various Linux distributions. Several build bugs fixed along the way.Spent a long while trying to figure out how the Fedora package maintainer procedure has changed since 3 years ago to create packages for ATF, Lutok and Kyua.
January 15, 2012
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
It's that time of the week again (although delayed by over a day). Unfortunately, no progress either during past week. Being on semi-vacations doesn't leave much "free time"... However, I traveled back home yesterday and getting back to my daily routine should give some spare time!
January 10, 2012
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Happy new year! No activity past week as I was completely away from the computer.
January 3, 2012
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
The post title should mention ATF instead of Kyua... but I'm keeping the usual one for consistency: Integrated timestamps into the XML and HTML reports generated by atf-report. These should soon show up in the continuous tests of NetBSD.Work on integrating the use of POSIX timers into atf-run after Christos Zoulas performed these changes in the NetBSD tree. The result is quite awful because I need to keep compatibility with systems that do not provide the "
December 26, 2011
·
Tags:
atf, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, not much activity this week due to travel reasons. Anyway, some work went in: Preliminary code to generate HTML reports from kyua report. This is easy peasy but boring. The current code was written as a proof of concept and is awful, hence why it was not committed. I'm now working in cleaning it up.Backported test program and test case timestamping into ATF based on a patch from Paul Goyette.
December 19, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Some significant improvements this week: Finally submitted the code to store and load full test case definitions. This is quite tricky (and currently very, very ugly) but it works and it will allow the reports to include all kinds of information from the test cases.Removed the Atffiles from the tree; yay! For a long time, I had been using atf-run to run broken tests because atf-run allowed me to watch the output of the test case being debugged.
December 11, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Some more work towards allowing storing and loading of full test case definitions (which in turn will allow us to provide detailed HTML reports). I have local changes that do this, but they are gated by the lack of some additional tests and probably some optimizations, because they slow down kyua report significantly. Regarding commits, I have only submitted some related cleanup-changes.I got distracted by invalid Doxygen comments and traced down why they were not being correctly validated (which is the whole point of running Doxygen during the build).
December 5, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
My goal for this past week was to change the database layer to be able to store full definitions of the test cases, and to later be able to load these while scanning an action. This is to allow the report command to provide all kinds of information about the executed tests, not just their names and their results. However, adding this functionality has proven to be more complex than I wished because the current types to represent test programs and test cases are kinda broken: that the abstractions chosen a while ago do not seem to be appropriate, and this is complicating further changes.
November 28, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
I only have one thing to report this week, but oh boy it's big: the report command finally reports the results of a run of a test suite! Yes, you heard well: Kyua is, finally, able to keep a record of all the previous executions of test suites and allows you to extract reports of any of them a posteriori. At the moment, the report is just some disorganized plain-text. For example:
November 20, 2011
·
Tags:
kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Kyua has finally gained a report subcommand, aimed at processing the output data of an action (stored in the database) and generating a user-friendly report in a variety of formats. This is still extremely incomplete, so don't get your hopes too high yet ;-) The current version of the report command takes an action and all it does is dump its runtime context (run directory, environment variables, etc.). Consider it just a proof of concept.
November 14, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Many things have happened this week, but they can all be summarized in one single sentence: kyua test now records the results of the execution of a test suite into the SQLite database. "Why is this important?", you ask. Well, that's a good question. Recording test results opens the gate to many long-awaited features that should be coming soon, such as the ability to inspect the history of a particular test, to query all available data of a test result and/or to generate a dashboard of test results.
November 7, 2011
·
Tags:
kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Submitted a placeholder implementation of the new persistence layer (store top-level directory). This only supports opening a database and ensuring its metadata is valid. See r253.Added a db-exec CLI command to execute arbitrary SQL commands onto the database. This is handy during development and testing, but may also help users to extract information out of the database in those cases where the CLI does not cover their needs just yet.
October 31, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Moved all the logic code of the "debug", "list" and "test" commands from the CLI layer to the engine layer.Up until now, the CLI modules implementing these commands contained all the logic to load Kyuafiles and iterating over them to find matching tests and applying the desired operation to them. This kind of code belongs in "driver" modules (aka controllers) of the engine layer, because there is nothing UI-related in them.
October 23, 2011
·
Tags:
kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Added support to prepare and execute statements to utils::sqlite.Added the UTILS_PURE definition to tag functions as pure. I should now sweep through old code to apply the attribute where possible.Created a pkgsrc package for Vera++. Investigating if I can use this tool for coding style validation, as the current code of Kyua is a bit messy in this regard.Made a quick attempt at getting kyua test record test results in a simple database; success!
October 16, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Some more reading on SQLite. The Using SQLite book is close to awesome; very easy to read and full of insightful tips.Cleaned up my preliminary SQLite wrapper code. Still very incomplete, but I've bitten the bullet and decided to commit the new library as is; otherwise I'll just keep procrastinating. So, ladies and gentlemen, welcome the new utils::sqlite module in Kyua. At the moment, this just provides the barebones to open and close a database.
October 9, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, not much time this week either :-(I am currently working on some adjustments to the design document of the database to describe new ideas; the previous design was incomplete in some areas and/or not accurate enough to support the described use cases. However, I've not had the to time to finish these edits and publish them.
October 3, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, no activity this week other than some brainstorming on the database design. Why? My shipment container from Dublin arrived and I spent most of the weekend organizing stuff, setting up my little NetBSD box and attending a friend's wedding!
September 26, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report (db designdoc edition!)
Moved the code of utils::lua to a new project, Lutok.Attempted to integrate a copy of Lutok into the Kyua source code to simplify installing Kyua. I have been playing with Subversion externals and Autoconf/Automake hacks to make this happen, but unfortunately haven't got a pleasant solution yet.Modified Lutok to not expose the Lua C API at all from header files and cleaned up the Kyua code to cope with the changes.
September 18, 2011
·
Tags:
kyua, lutok, report
Continue reading (about
1 minute)
Kyua: Weekly status report
My plan for this week was to release utils::lua as a separate module. Unfortunately, this has not been possible because I've been fighting with legal to clear the name for the project. I don't have an approved name yet, so this will have to wait a bit more :-(On another order of things, I have started writing a design document for the database that will collect test case results and other information.
September 12, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Slow week. We had some team-related events at work and I have friends over from Dublin, so it has been hard to find some time to do work on Kyua. Regardless, here comes the weekly report: Split utils::lua into its own package, per some user's request. I'm still setting up the separate project and have to do lots of cleanup on the code, so nothing is available yet.Started experimenting on the long promised "
September 4, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Not a very active week: I've been on-call four days and they have been quite intense. Plus I have had to go through a "hurricane" in NYC. That said, I had some time to do a bit of work on Kyua and the results have been nice :-) Made calls to getopt_long(3) work with GNU Getopt by using the correct value of optind to reset option processing.Improved the configure script to error out in a clearer way when missing dependencies (pkg.
August 28, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Implemented the "debug" command. Still very rudimentary, this command allows the user to run a test case without capturing its stdout nor stderr to aid in debugging of failed test cases. In the future, this command will also allow things like keeping the work directory for manual inspection, or spawning a shell or a debugger in the work directory after a test case is executed. Many build fixes under different platforms in preparation for a 0.
August 21, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Changed the --config and --variable options to be program-wide instead of command-specific. The configuration file should be able to hold properties that tune the behavior of Kyua, not just the execution of tests, so this makes sense. Added the config subcommand, which provides a way to inspect the configuration as read by Kyua. Got rid of the test_suites_var function from configuration files and replaced it by simple assignments to variables in the test_suites global table.
August 15, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Added the ability to explicitly define timeouts for plain test programs. This just completes the work from past week that made running plain test programs at all but had an ugly TODO in it to implement this missing feature. The bootstrap test suite now runs as a single test case within the whole Kyua test suite. Demonstrates the plain test programs interface functionality :-) Started reshuffling code to make the <tt>--config</tt> and related flags program-wide.
August 8, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Implemented the "plain" test interface. This allows plugging "foreign test programs" into a Kyua-based test suite. (A foreign test program is a program that does not use any testing framework: it reports success and failure by means of an exit code.)Generalized code between the atf and plain interfaces and did some cleanups.Attempted to fix the ATF_REQUIRE_EQ macros in ATF to evaluate their arguments only once. This has proven to be tricky and therefore it is not done yet.
August 1, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Finished splitting the atf-specific code from the generic structures in the engine. The engine now supports the addition of extra test interfaces with minimal effort.Started implementing a "plain" test interface for test programs that do not use any test framework. This is to allow muxing non-atf tests into atf-based test cases, which is required in the NetBSD tree.
July 25, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Slow week. I've been busy moving to NYC! Kept working on the splitting of ATF-specific code from the core test abstractions. The work is now focused on refactoring the results-reporting pieces of the code, which are non-trivial.
July 18, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
One of the major features I want in place for Kyua 0.2 is the ability to run "foreign" test programs as part of a test suite: i.e. to be able to plug non-ATF test programs into a Kyuafile. The rationale for this is to lower the entry barrier of newcomers to Kyua and, also, to allow running some foreign test suites that exist in the NetBSD source tree but that are currently not run.
July 10, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Belated update:Created a pkgsrc package for kyua-cli. Still in pkgsrc-wip though because pkgsrc is in a feature freeze.Wrote a little tutorial on how to run NetBSD tests using Kyua.Started work on 0.2 by doing a minor UI fix in the about command.I've now started to look at how to split the engine into different "runners" to add support for test programs written without ATF. Not that I plan to use this feature.
July 5, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This has been the big week: Wrote some user documentation for the kyua binary.Fixed some distcheck problems.Released Kyua 0.1!The next immediate thing to do is to write a short tutorial on how to run the NetBSD tests with Kyua and get some people to actually try it. After that, there are many things to improve and features to add :-)
June 26, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
A couple of things have happened: Released ATF 0.14. This release was followed by an import into NetBSD and fixing of subsequent fallout.Some performance improvements to atf-sh. After killing a bunch of complex shell constructions and removing lots of obsolete functions, the performance results are significant. There is still room for improvement of course, and I still need to quantify how these optimizations behave in single-core machines.I certainly expected more progress this past week.
June 19, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Added support for recursion from the top-level Kyuafile. This Kyuafile should not reference any directories explicitly because the directories at the top level are supposed to be created by the installation of packages. Closed issue 9.Improved error messages when the test programs are bogus. Closed issue 13.Backported format-printf attribute improvements from NetBSD head to ATF.Miscellaneous build and run fixes for both Kyua and ATF in NetBSD and OS X.Cut a release candidate for atf-0.
June 13, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Some long-standing bug fixes / improvements have gone in this week: Improvements to the cleanup routine, which is used to destroy the work directory of a test case after the test case has terminated:Heavy refactoring to be tolerant to failures. These failures may arise when a child of the test case does not exit immediately and holds temporary files in the work directory open for longer than expected.Any file systems that the test case leaves mounted within the work directory will now be unmounted, just as the ATF test interface mandates.
June 5, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
Some cool stuff this week, albeit not large-scale: Implemented the --variable flag in the test command. This flag, which can be specified multiple times, allows a user to override any configuration variable (be it a built-in or a test-suite variable) from the command line. This is actually the same as atf-run's -v flag, but with a clear separation between built-in configuration settings and test-suite specific settings.Added support for several environment variables to allow users (and tests) to override built-in paths.
May 29, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This week:Cleaned up the internal code of the "list" command and added a few unit tests.Added integration tests for the "test" command that focus mostly on the behavior of the "test" command itself. There is still a need for full integration tests that validate the execution of the test cases themselves and their cleanup, and these will be tricky to write.Changed atf-c, atf-c++ and atf-sh to show a warning when a test program is run by hand.
May 22, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report, BSDCan 2011 edition
I spent past week in Ottawa, Canada, attending the BSDCan 2011 conference. The conference was composed of lots of interesting content and hosted many influential and insightful BSD developers. While the NetBSD presence was very reduced, I could have some valuable talks with both NetBSD and FreeBSD developers. Anyway. As part of BSDCan 2011, I gave a talk titled "Automated testing in NetBSD: past, present and future". The talk focused on explaining what led to the development of ATF in the context of NetBSD, what related technologies exist in NetBSD (rump, anita and dashboards), what ATF's shortcomings are and how Kyua plans to resolve them.
May 16, 2011
·
Tags:
atf, conference, kyua, report
Continue reading (about
3 minutes)
Kyua: Weekly status report
Unfortunately, no progress whatsoever this week :-( Too busy at work and preparing my upcoming trips. Time to fly to BSDCan 2011 tomorrow.
May 8, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Unfortunately, I have had no time for coding this week. The only things I could do were:Fixed a few build problems on NetBSD introduced during past week's changes.Built Kyua and ran a few tests on NetBSD/macppc (just for the joy of it).Coding has been eclipsed by the preparation of my presentation for BSDCan 2011; at this point, this has priority over any code changes. I'd argue that preparing the presentation is also part of the project, so some time has been invested ;-)
May 1, 2011
·
Tags:
kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
Ouch; I'm exhausted. I just finished a multi-hour hacking session to get the implementation of the list subcommand in control. It is now in a very nice user-facing shape, although its code deserves a little bit of house cleaning (coming soon). Anyway, this week's progress: Added the kyuaify.sh script. This little tool takes a test suite (say, NetBSD's /usr/tests directory) and converts all its Atffiles into Kyuafiles. The tool is no sophisticated at all; in fact, it is a pretty simple script that I haven't tested with any other test suites so far.
April 24, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
This week started easy:Added integration tests for the about and help subcommands. These were pretty easy to do.Added integration tests for the list subcommand. I initially added these tests as expected failures to reason about the appearance and behavior of this command from the point of view of the user before actually working on the code... and I am still writing such code to make these tests pass!This is where things got a bit awry.
April 17, 2011
·
Tags:
atf, kyua, report
Continue reading (about
3 minutes)
Kyua: Weekly status report
Few things worth mentioning this week as reviewing Summer of Code student applications has taken priority. The good thing is that there are several strong applications for NetBSD; the bad thing is that none relate directly to testing. Anyway, the work this week: Added a pkg-config file for atf-sh as well as an Autoconf macro to detect its presence. This is needed by Kyua to easily find atf-sh. (Yes, I know: this is an abuse of pkg-config, but it works pretty well and is consistent with atf-c and atf-c++.
April 10, 2011
·
Tags:
atf, kyua, report
Continue reading (about
1 minute)
Kyua: Weekly status report
This week's work has been quite active on the ATF front but not so much in the Kyua one. I keep being incredibly busy on the weekends (read: traveling!) so it's hard to get any serious development work done. What has happened? Finally tracked down and fixed some random atf-run crashes that had been hunting the NetBSD test suite for months (see PR bin/44176). The fix is in reality an ugly workaround for the fact that a work directory cannot be considered "
April 3, 2011
·
Tags:
atf, kyua, report
Continue reading (about
2 minutes)
Kyua: Weekly status report
This has been a slow week. In the previous report, I set the goal of getting Kyua to run the NetBSD test suite accurately (i.e. to report the same results as atf-run), and this has been accomplished. Actually, the changes required in Kyua to make this happen were minimal, but I got side-tracked fixing issues in NetBSD itself (both in the test suite and in the kernel!). So, the things done:
March 27, 2011
·
Tags:
atf, kyua, report
Continue reading (about
3 minutes)
Kyua: Weekly status report
These days, I find myself talking about Kyua to "many" people. In particular, whenever a new feature request for ATF comes in, I promise the requester that the feature will be addressed as part of Kyua. However, I can imagine that this behavior leaves the requester with mixed feelings: it is nice that the feature will be implemented but, at the same time, it is very hard to know when because the web site of Kyua does not provide many details about its current status.
March 20, 2011
·
Tags:
atf, kyua, report
Continue reading (about
3 minutes)
Featured software
Featured posts
- Fast machines, slow machines
- EndBASIC 0.10: Core language, evolved
- Farewell, Microsoft; hello, Snowflake!
- Rust is hard, yes, but does it matter?
- Rust traits and dependency injection
- A year on Windows: Introduction
- Always be quitting
- How does Google keep build times low?
- How does Google avoid clean builds?
- Unit-testing a console app (a text editor)
- More...
