Testing - Julio Merino (jmmv.dev)

End-to-end tool testing with Bazel and shtk

If you use Bazel, your project is of moderate size. And because your project is of moderate size, it almost-certainly builds one or more binaries, at least one of which is a CLI tool. But let’s face it: you don’t have end-to-end testing for those tools, do you? I’m sure you have split the binary’s main function into its own file so that the rest of the tool can be put in a library, and I’m extra-sure that you have unit tests for such library. But… those tests do little to verify the functionality and quality of the tool as experienced by the end user. Consider: What exactly does the tool print to the console on success? Does it show errors nicely when they happen, or does it dump internal stack traces? How does it handle unknown flags or bad arguments? Is the built-in help message nicely rendered when your terminal is really wide? What if the terminal is narrow? You must write end-to-end tests for your tools but, usually, that isn’t easy to do. Until today. Combining shtk with Bazel via the new rules_shtk ruleset makes it trivial to write tests that verify the behavior of your CLI tools—no matter what language they are written in—and in this article I’m going to show you how.

November 4, 2023 · Tags: <a href="/tags/bazel">bazel</a>, <a href="/tags/blogsystem5">blogsystem5</a>, <a href="/tags/shell">shell</a>, <a href="/tags/shtk">shtk</a>, <a href="/tags/testing">testing</a>
Continue reading (about 7 minutes)

Unit-testing shell scripts and tools with shtk

While working on this static blog a few days ago, I made a change to its templates that warranted an automated test. I could have written a trivial shell script to do it, but instead I reached out for shtk’s unit-testing module. I tweeted about it right away to just say that you can, in fact, write tests in shell because lots of developers are skeptical about any script longer than 10 lines of code. Interestingly, this reply came through: a pointer to a contemporary, under-development library for writing tests in Bash. Which made me think: “Hey, I had already done that years ago… but nobody knows about it. Gotta fix that with a blog post!” But first, I had to bring shtk back from its ashes because I had not touched it for more than 6 years and it wasn’t read for show and tell. So I did something that I wanted to do back in the day but never did: I put together a website for shtk to host its reference manual and I fixed a few obvious rough edges. With those tweaks out of the way, we come to this article. In here, I want to show you how writing decent tests in shell is entirely possible and how shtk’s testing platform provides unique features to do integration testing of CLI apps written in any language.

October 11, 2023 · Tags: <a href="/tags/shell">shell</a>, <a href="/tags/shtk">shtk</a>, <a href="/tags/testing">testing</a>
Continue reading (about 12 minutes)

BSDCan 2014 summary

BSDCan 2014 and the accompanying FreeBSD devsummit are officially over. Let's recap. FreeBSD devsummit The FreeBSD devsumit at BSDCan is, by far, the largest of them all. It is true that I already visited a devsummit once —the one in EuroBSDCon 2013—, but this is the first time I participate in the "real deal" while also being a committer. The first impressive thing about this devsummit is that there were about 120 attendees. The vast majority of these were developers, of course, but there was also a reasonable presence from vendors — including, for example, delegates from Netflix, Isilon, NetApp and even smaller parties like Tarsnap.

May 21, 2014 · Tags: <a href="/tags/bsdcan">bsdcan</a>, <a href="/tags/conference">conference</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/netbsd">netbsd</a>, <a href="/tags/testing">testing</a>
Continue reading (about 6 minutes)

GSoC 2014 idea: Port FreeBSD's old-style tests to ATF

Are you a student interested in contributing to a production-quality operating system by increasing its overall quality? If so, you have come to the right place! As you may already know, the Google Summer of Code 2014 program is on and FreeBSD has been accepted as a mentoring organization. As it so happens, I have a project idea that may sound interesting to you. During the last few months, we have been hard at work adding a standardized test suite to the FreeBSD upstream source tree as described in the TestSuite project page. However, a test suite is of no use if it lacks a comprehensive collection of tests!

March 12, 2014 · Tags: <a href="/tags/atf">atf</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/soc">soc</a>, <a href="/tags/testing">testing</a>
Continue reading (about 3 minutes)

FreeBSD Test Suite tutorial at AsiaBSDCon 2014

I am pleased to announce that the tutorial on the FreeBSD Test Suite that I proposed for AsiaBSDCon 2014 has been accepted! The conference website will soon include more details, but allow me to spoil your wait: Goals: Learn how to use the test suite, how it is internally organized and how new tests can be written. Stretch goal: Get attendees to contribute one or more tests to the project. Audience: Mostly developers of FreeBSD that want to learn how the new test suite plugs into the system. That said, and because a major part of the tutorial will revolve around using the test suite for one's own benefit, everyone is welcome really. In particular, system administrators may get a useful tool out of this. The main surprise in the acceptance confirmation email is that materials are due by January 20th... which is around the corner! Time to rush in getting things ready. In the meantime, you can find more details on the tutorial by reading the proposal itself.

January 8, 2014 · Tags: <a href="/tags/conference">conference</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/testing">testing</a>
Continue reading (about 1 minute)

FreeBSD Test Suite goals and planning

I was really impressed and delighted to see how popular my previous (and first) post on FreeBSD testing, titled Introducing the FreeBSD Test Suite, was. Looks like this project may be of great interest to developers and users out there (not unsurprisingly) so I'll try to keep you all up-to-date with any key developments. A first question that arises from the announcement is: where are the test suite and infrastructure headed? After all, the continuous testing machines for amd64 are already up and running, so what else is there to do?

January 5, 2014 · Tags: <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/testing">testing</a>
Continue reading (about 2 minutes)

Introducing the FreeBSD Test Suite

I joined the FreeBSD committer ranks a couple of months ago with the intention to equip FreeBSD with an out-of-the-box test suite and with a testing infrastructure. The time until now has been quite fruitful and I have been rushing to get something ready for you before the year end. With that, I am very pleased to announce that the first mockup of the FreeBSD testing cluster is up and running! Point your browser at:

December 31, 2013 · Tags: <a href="/tags/featured">featured</a>, <a href="/tags/freebsd">freebsd</a>, <a href="/tags/kyua">kyua</a>, <a href="/tags/testing">testing</a>
Continue reading (about 4 minutes)

Testing NetBSD's share/mk

For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends. One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story: Other projects depend on these build files and thus rely on their behavior. This can either be by just relying on NetBSD having these files installed, or by using the separate mk-files package (available e.g. in Fedora). The functionality of the build infrastructure should not regress, or it would break these third-party projects. (Think about these files as having a public API outside of NetBSD.)Even if the build of NetBSD is the real test of the functionality of the build infrastructure, the build infrastructure supports dozens of options and tweaks to change its behavior. One would need to rebuild NetBSD tens, if not hundreds of times, each with a different combination of build options, to ensure the files work.Lastly, when a new functionality is added to the build infrastructure, it is because some component of the source tree needs such functionality. It may latter happen that this component disappears or stops needing such functionality. The functionality becomes unused by the source tree, and thus can regress unexpectedly (breaking third-party packages or introducing bugs, as mentioned earlier).With this in mind, it's clear that we should have some tests for the share/mk files. Unfortunately, that's easier said than done: the files in share/mk are extremely complex and expose hundreds, if not thousands, of different behaviors each with its own subtleties. Adding tests for these is hard. The fact that this code that was never designed to be unit-tested doesn't help either. Regardless, I have just submitted some "placeholder" tests to the tree. The major point of these new test programs is to lower the barrier of entry to writing tests for share/mk and, therefore, maybe get other people to write some tests. To predicate with the example, I have populated these skeleton test programs with a couple of test cases each, although as you will see they are very trivial tests. And, to conclude, why have I done this now? Well: I'm working on integrating Kyua into the source tree and, to make this happen, I need to do a couple of changes to the build infrastructure files. The changes are tricky, so I want to write tests to have some assurance that my modifications work and that, specially, they do not regress over time. Having the placeholder test programs in place makes this much easier, and the real functionality changes easier to review.

August 26, 2012 · Tags: <a href="/tags/netbsd">netbsd</a>, <a href="/tags/testing">testing</a>
Continue reading (about 3 minutes)

Dependency injection: simple class constructors

Following my previous post on dependency injection (DI for short), I wanted to show you today another example of code in which DI helps in making the code clearer and easier to validate. In this case, the person to blame for the original piece of code being criticized is me. The atffile module in ATF provides a class to represent the contents of Atffiles. An Atffile is, basically, a file containing a list of test programs to run and a list of properties associated to these test programs. Let’s consider the original implementation of this module:

December 26, 2010 · Tags: <a href="/tags/testing">testing</a>
Continue reading (about 4 minutes)

Dependency injection and testing: an example

A coworker just sent me some Python code for review and, among such code, there was the addition of a function similar to: def PathWithCurrentDate(prefix, now=None): """Extend a path with a year/month/day subdirectory layout. Args: prefix: string, The path to extend with the date subcomponents. now: datetime.date, The date to use for the path; if None, use the current date. Returns: string, The new computed path with the date appended. """ path = os.path.join(prefix, '%Y', '%m', '%d') if now: return now.strftime(path) else: return datetime.datetime.now().strftime(path) The purpose of this function, as the docstring says, is to simplify the construction of a path that lays out files on disk depending on a given date.

December 23, 2010 · Tags: <a href="/tags/testing">testing</a>
Continue reading (about 5 minutes)