Redefining Kyua's mission

A couple of months ago, I posted some initial thoughts on refocusing Kyua's goals. Basically, the underlying thought was to drop any Continuous Integration ambitions I had from Kyua's plans and delegate such functionality to better suited systems (e.g. Jenkins or Travis CI).

Kyua must focus on delivering one and only one thing, and make sure that that thing really shines. What is that thing? Keep reading.

Taking action on those radical thoughts right away was not a great strategy: any major shift in plans and priorities needs to be properly assessed. And that's why I am writing about this today: over the last couple of months, I have given a lot of thought to those initial ideas and prototyped one of the simplifications that they would enable. I am now decided to clearly narrow and properly define the scope of the Kyua project.

Rest assured, though, that any cuts will not affect anything you do today with Kyua (modulo some small-ish UI changes). The cuts come from reducing the scope of the project and in focusing any future plans. So "how does that make a difference, then?", you say. For one, peace of mind. Knowing where the project is not going lets me truly focus on where you the project really needs to go. (Nothing new actually, but I had to bring this guideline into practice.) For another, the ability to actually simplify the project's architecture will increase the agility with which I can make changes (as, arguably, the current architecture is just too complex.

With this in mind, there are several things that can and will change. I am not going to provide details on any of them; only a brief description to give you an idea of where I am going with these thoughts.

Focus on the actual top consumers

Let's face it: Kyua's main consumer today is FreeBSD (and maybe NetBSD). Understanding this and internalizing this fact is an important step towards simplification.

While I certainly do not want to tie Kyua to the specifics of any operating system, it's true that by assuming that Kyua is for the BSD systems and their environment, we can shape various aspects of Kyua.

In fact, this has always been the case and is the reason why Kyua is written mostly in C++ and not in any other (better) higher-level language. The code has to work within the constraints of a BSD system and thus the set of dependencies one can rely on is limited.

Another clear example of this is deciding to use the CI systems that people have already built around the test suites of FreeBSD and NetBSD, be them Jenkins deployments or more ad-hoc setups.

Removal of the historical store

Because Kyua had the eventual goals of becoming a Continuous Integration system, the initial implementation of the backing database (aka store) had support for historical data. In other words: a single database had the ability to hold the results of all of your test suite runs, regardless of the test suite you were running.

In all honesty, this feature of the backing database has never been useful —at all, really— due to the lack of a user interface to put such data to use. For example, it has not yet been possible to generate single reports with the results of more than one action. It has not been possible either to examine the history of a test case's results. But why should all this be possible? The most common way in which Kyua is executed is within a shiny-new virtual machine to run a test suite and then to extract a report: there is zero history in this case. History management features come from the "wrapping" system, which is whichever Continuous Integration framework you choose to use.

So this is an area that deserves severe simplification. Kyua is going to change to generate an individual results file per test suite run (and you should not care that such file is, internally, a SQLite database). To minimize disruption, these result files will be time-stamped to provide a historical view of all test suites and will be stored in a centralized depot (aka a well-defined directory). However, Kyua won't let you (at least for the time being) run commands that span more than one results file.

Unification of all the Kyua packages

But let's backtrack a little bit. Almost two years ago, I decided to split out the code that executes a single test case out of the kyua-cli package and instead ship those code snippets as a kyua-testers package. The goal was to decouple the tricky, operating system-dependent code required to run test cases from the higher-level interface code so that the former could be written in C and the latter could maybe be written in Lua.

Guess what the result is though? The code in kyua-cli is not simpler, and in fact the indirection via the testers hinders error diagnostics, debugging and hurts performance.

Worse than that: having to ship Kyua as three different packages (yes, remember that there also is a kyua-atf-compat!) is incredibly time-consuming due to the trickiness of performing integration tests on all supported packaging systems.

So: kyua-atf-compat is going to see the axe as, today, atf-run and atf-report do not exist any longer and, to my knowledge, nobody has successfully used kyua-atf-compat.

kyua-cli and kyua-testers are going to be merged as a single component and everything will ship as a unified kyua distfile. Result? Distributions will only have to maintain a single package, which helps integration validation.

To provide a runtime engine for the test suites of BSD-like operating systems.