testing - Julio Merino (jmmv.dev)

End-to-end tool testing with Bazel and shtk

If you use Bazel, your project is of moderate size. And because your project is of moderate size, it almost-certainly builds one or more binaries, at least one of which is a CLI tool. But let’s face it: you don’t have end-to-end testing for those tools, do you? I’m sure you have split the binary’s main function into its own file so that the rest of the tool can be put in a library, and I’m extra-sure that you have unit tests for such library. But… those tests do little to verify the functionality and quality of the tool as experienced by the end user. Consider: What exactly does the tool print to the console on success? Does it show errors nicely when they happen, or does it dump internal stack traces? How does it handle unknown flags or bad arguments? Is the built-in help message nicely rendered when your terminal is really wide? What if the terminal is narrow? You must write end-to-end tests for your tools but, usually, that isn’t easy to do. Until today. Combining shtk with Bazel via the new rules_shtk ruleset makes it trivial to write tests that verify the behavior of your CLI tools—no matter what language they are written in—and in this article I’m going to show you how.

November 4, 2023 · Tags: bazel, blogsystem5, shell, shtk, testing
Continue reading (about 7 minutes)

Unit-testing shell scripts and tools with shtk

While working on this static blog a few days ago, I made a change to its templates that warranted an automated test. I could have written a trivial shell script to do it, but instead I reached out for shtk’s unit-testing module. I tweeted about it right away to just say that you can, in fact, write tests in shell because lots of developers are skeptical about any script longer than 10 lines of code. Interestingly, this reply came through: a pointer to a contemporary, under-development library for writing tests in Bash. Which made me think: “Hey, I had already done that years ago… but nobody knows about it. Gotta fix that with a blog post!” But first, I had to bring shtk back from its ashes because I had not touched it for more than 6 years and it wasn’t read for show and tell. So I did something that I wanted to do back in the day but never did: I put together a website for shtk to host its reference manual and I fixed a few obvious rough edges. With those tweaks out of the way, we come to this article. In here, I want to show you how writing decent tests in shell is entirely possible and how shtk’s testing platform provides unique features to do integration testing of CLI apps written in any language.

October 11, 2023 · Tags: shell, shtk, testing
Continue reading (about 12 minutes)

BSDCan 2014 summary

BSDCan 2014 and the accompanying FreeBSD devsummit are officially over. Let's recap. FreeBSD devsummit The FreeBSD devsumit at BSDCan is, by far, the largest of them all. It is true that I already visited a devsummit once —the one in EuroBSDCon 2013—, but this is the first time I participate in the "real deal" while also being a committer. The first impressive thing about this devsummit is that there were about 120 attendees.

May 21, 2014 · Tags: bsdcan, conference, freebsd, netbsd, testing
Continue reading (about 6 minutes)

GSoC 2014 idea: Port FreeBSD's old-style tests to ATF

Are you a student interested in contributing to a production-quality operating system by increasing its overall quality? If so, you have come to the right place! As you may already know, the Google Summer of Code 2014 program is on and FreeBSD has been accepted as a mentoring organization. As it so happens, I have a project idea that may sound interesting to you. During the last few months, we have been hard at work adding a standardized test suite to the FreeBSD upstream source tree as described in the TestSuite project page.

March 12, 2014 · Tags: atf, freebsd, soc, testing
Continue reading (about 3 minutes)

FreeBSD Test Suite tutorial at AsiaBSDCon 2014

I am pleased to announce that the tutorial on the FreeBSD Test Suite that I proposed for AsiaBSDCon 2014 has been accepted! The conference website will soon include more details, but allow me to spoil your wait: Goals: Learn how to use the test suite, how it is internally organized and how new tests can be written. Stretch goal: Get attendees to contribute one or more tests to the project. Audience: Mostly developers of FreeBSD that want to learn how the new test suite plugs into the system.

January 8, 2014 · Tags: conference, freebsd, kyua, testing
Continue reading (about 1 minute)

FreeBSD Test Suite goals and planning

I was really impressed and delighted to see how popular my previous (and first) post on FreeBSD testing, titled Introducing the FreeBSD Test Suite, was. Looks like this project may be of great interest to developers and users out there (not unsurprisingly) so I'll try to keep you all up-to-date with any key developments. A first question that arises from the announcement is: where are the test suite and infrastructure headed?

January 5, 2014 · Tags: freebsd, kyua, testing
Continue reading (about 2 minutes)

Introducing the FreeBSD Test Suite

I joined the FreeBSD committer ranks a couple of months ago with the intention to equip FreeBSD with an out-of-the-box test suite and with a testing infrastructure. The time until now has been quite fruitful and I have been rushing to get something ready for you before the year end. With that, I am very pleased to announce that the first mockup of the FreeBSD testing cluster is up and running!

December 31, 2013 · Tags: featured, freebsd, kyua, testing
Continue reading (about 4 minutes)

Testing NetBSD's share/mk

For a long time, a pet peeve of mine has been the lack of tests for the build infrastructure files of NetBSD: i.e. those bsd.*.mk files that live under /usr/share/mk/ and on which the whole source tree depends. One could argue that writing tests for these files is not strictly necessary because the successful build of NetBSD is the real final test of whether the files work or not. That's partly true, but unfortunately is not the whole story:

August 26, 2012 · Tags: netbsd, testing
Continue reading (about 3 minutes)

Dependency injection: simple class constructors

Following my previous post on dependency injection (DI for short), I wanted to show you today another example of code in which DI helps in making the code clearer and easier to validate. In this case, the person to blame for the original piece of code being criticized is me. The atffile module in ATF provides a class to represent the contents of Atffiles. An Atffile is, basically, a file containing a list of test programs to run and a list of properties associated to these test programs.

December 26, 2010 · Tags: testing
Continue reading (about 4 minutes)

Dependency injection and testing: an example

A coworker just sent me some Python code for review and, among such code, there was the addition of a function similar to: def PathWithCurrentDate(prefix, now=None): """Extend a path with a year/month/day subdirectory layout. Args: prefix: string, The path to extend with the date subcomponents. now: datetime.date, The date to use for the path; if None, use the current date. Returns: string, The new computed path with the date appended. """ path = os.

December 23, 2010 · Tags: testing
Continue reading (about 5 minutes)