Three productive days on the Kyua front

This being Thanksgiving week in the U.S. and Google giving us Thursday and Friday off, I decided to take Monday to Wednesday off as well to spend some time hacking on Kyua — yes, finally, after months of being inactive. And what a three productive days!

Here comes a little briefing on the three fronts in which I made progress. (This puts on hold the header files series until next Monday... but most of you are probably away anyway. Enjoy the holidays if they apply to you!)

Updated Kyua packages for Fedora

The first thing I tackled, which was long overdue, was the update of the kyua-cli package in Fedora past 0.5, fixing along the way the old broken and unbuildable package. I could not do this update right away when I published 0.6 given that I had to create a package for kyua-testers first, and the review for the new spec file got stuck awaiting for a reviewer for quite a while. But finally, after it got a reviewer assigned, the process blocked on me for way too long.

After unblocking the review of kyua-testers, things went quite smoothly. I submitted the package, fixed the build of kyua-cli while upgrading it 0.7, and pushed new binaries for both rawhide and f20.

FreeBSD, Kyua and TAP

My next step, according to the FreeBSD Test Suite project plan, was to migrate a bunch of the existing tests in src/tools/regression/ to the new Kyua-based framework to demonstrate how easy this process would be. The goal was (is) to just move the tests without having to rewrite them using the ATF libraries. I thought this would be easily done by using the plain tester in Kyua... but I was wrong.

The truth is that test programs in src/tools/regression/ implement the Test Anything Protocol and, unfortunately, this protocol does not define the exit status of the test program. As a result, the plain tester in Kyua is unsuitable to run such tests because it may always report them as passing even when they fail.

Seeing how simple the protocol is, I decided to take a stab at implementing a TAP-compliant backend for Kyua. And, after some work, the prototype morphed into actual usable code able to run (some of) the existing FreeBSD tests. The code landed in the main Git repository of Kyua and will be available in the next release (due by sometime next week probably). You can read some more on this topic on the ongoing thread in freebsd-testing.

But this is not all on this front. The converted bin/sh/ tests were getting stuck mysteriously half-way through while the originals weren't. This was strange given that I had not touched their code and thus the problem had to lie somewhere in Kyua. The problem was exposed by an attempt to run a login shell within a test, which resulted in the shell trying to handle the controlling session and waiting for it indefinitely given that Kyua was in the foreground (or something like that; I don't know all the details).

The solution was simpler than it seemed: just changed the test isolation code to put tests in their own session instead of only in their own process group. New sessions don't have a controlling terminal by default, so attempts to get access to it from within a test will fail immediately instead of just blocking. The fix worked just fine (although I'm not fully convinced it's really the right fix yet after reading more about ptys and ttys than I wanted).

And that's still not all. As a result of this, I figured that I could plug the bootstrap tests into the test suite again after two years of first attempting this and failing. I believe the issue experienced there was the same one I just fixed. Enabling the tests was easy, but getting make distcheck to pass was incredibly annoying and something that made me waste what seemed hours.

Lua binding for ATF

Much to my dismay, Kyua has slowly morphed into a complex beast and C++ has not helped in preventing it. Actually, it is not that the code is that complex, but the build times are unbearably slow to the point of being a major hurdle for development.

The idea of rewriting parts of Kyua using Lua had been floating my mind for quite a while given that Lua is already a dependency of Kyua and, really, most of the code would be better written in a higher-level language. In fact, this is one of the reasons that motivated the split of the complex testing code into kyua-testers a year ago. Other than that, I was not sure how to make any progress on this idea... until yesterday morning, when it occurred to me that scripting report generation would be the solution to allow customizable user reports.

But before attempting to rewrite any production code, a prerequisite for it is to have an ATF-compliant testing library for Lua that I can use and trust. And that's the last thing I've been working on.

Coming up with a working prototype wasn't particularly trivial given that my Lua knowledge was close to zero... but, after a few hours with book in hand and much head-scratching, I got the prototype to a functional state. To be honest, it was a nice mental exercise given that I had not learned any new languages for a while! Pretty exciting.

With the foundations for this Lua binding in place, moving it forward to a workable product is now relatively easy. I have no idea yet how I will ship it though. The simple choice is to make it one more binding in the atf package; however: 1) the API is significantly different to the other ATF APIs and 2) ATF should go away eventually... so adding more shiny-new code to this package doesn't seem like a brilliant idea. The alternative is to ship it separately, but then it's very tempting to dump many utility functions into such package as well. (Read: I'm really enjoying Lua's simplicity but the base libraries are just too limited by default. Yes, yes, I know why.)

Anyway. A sneak peek on where things stand at the moment (n.b. code not yet pushed out so highly volatile):

local atf = require("atf")

...

tc = atf.TestFixture:new("repr")

function tc:nil_body()
    self:assert_equal("nil", string.repr(nil))
end

function tc:number_body()
    self:assert_equal("3", string.repr(3))
    self:assert_equal("-18", string.repr(-18))
    self:assert_equal("5.8", string.repr(5.8))
end

...

atf.main{...}

Smells like Python's unittest, doesn't it? Yes, it does. After years of using that interface at work, I wish ATF's APIs were similar.

Stay tuned...

and enjoy the holidays! But, before leaving: do you have any suggestions on any of the above?