Using the builder pattern to define test scenarios

I’ve been playing with the builder patter to express test scenarios in a succinct and declarative manner. I’ve liked the outcome and feel that this design can yield to pretty good test code, so I’ll dig through this idea here. Note that, while this post and the associated code talk about Rust, the ideas presented here apply to any language. So don’t leave just because I said Rust!

While Rust provides excellent facilities to execute unit and integration tests via cargo test, the APIs to write such tests are… extremely limited. In essence, we can tag tests functions with #[test] and we can then use the assert* family of macros plus unwrap()/expect() calls to validate values. But that’s about it. No fixtures. No JUnit-like advanced features. No Truth-like expressive assertions. No nothing.

These limitations, and in particular the lack of fixtures, can lead to convoluted test code. The fact is that some tests, even if they are true unit tests, require quite a bit of boilerplate: prepare fake objects with golden values, inject those into the code under test, and validate results. If we want to adhere to the principle of making each individual test case (aka each #[test] function) focus on a single behavior, we find ourselves with a lot of code duplication. Code duplication blurs the essence behind each test, which then makes tests harder to maintain—and hard to maintain tests is something you really do not want in a codebase.

To mitigate these limitations and keep the content of test functions focused on behavior, I’ve found myself writing helpers of the form do_ok_test_for_blah(). These helpers wrap the blah() under test and abstract away all of the uninteresting setup/teardown noise, customizing these based on input parameters.

The problem is that functions like do_ok_test_for_blah() quickly become complex. As you add more test cases, the amount of parameters to pass to these functions grows too, and the tests that used to be succinct aren’t any more. To compensate, you might add extra functions like do_ok_test_for_blah_with_bleh() that provide even further wrapping… but then you end up with a real mess that goes counter the original goal of simplifying the test code. Mind you, a lot of the older tests in EndBASIC suffer from this problem and it’s a pain to touch them, so I had to find a solution.

Of course, there are crates out there to provide extra testing facilities for Rust. But how far can we get with the simple Rust primitives? Pretty far actually.

What if we used the builder pattern to define the test scenario with required and optional properties, set expectations in a declarative manner, and then hid the test logic within it?

The idea is to define a type (or struct, or class, or whatever your language of choice offers to encapsulate data) that holds all details needed to set up a test and also carries the expectations of the test. The type requires the minimum amount of parameters to run a “null” test scenario, and allows passing all other parameters in an optional manner. Lastly, a single run()-like function takes care of preparing the test scenario and running through it.

Let’s illustrate all these words with a trivial example.

Consider a simple and non-generic sum_all function that takes an array of i32 values and sums them all:

#[cfg(test)] mod tests;

/// Sums all input `values` and returns the total.
pub fn sum_all(values: &[i32]) -> i32 {
    let mut result = 0;
    for v in values {
        result += v;
    }
    result
}

Yes, this is a simplistic function that can be trivially tested: using the builder pattern here is overkill, but hopefully you get the idea of how this helps verify more complex interfaces.

To test this function, we define a SumAllTest type to represent the builder pattern, with these properties:

• Given that sum_all() always returns a value, the test must always know what value to expect; therefore, we express this requirement as part of the type’s constructor.
• The data to pass to the function is variable, though, so we make it optional via an add_value() method. I’ve chosen to use an accumulator method here to further illustrate how a builder might be helpful.
• We define a run() method that consumes the builder and executes the test.

Our test code looks like this:

#![deny(warnings)]

use crate::*;

/// Builder pattern for tests that validate `sum_all`.
#[must_use]
struct SumAllTest {
    expected: i32,
    values: Vec<i32>,
}

impl SumAllTest {
    /// Creates the test scenario and initializes it with the result we expect.
    fn expect(value: i32) -> Self {
        Self { expected: value, values: vec!() }
    }

    /// Registers a value to pass to `sum_all` as an input.
    fn add_value(mut self, value: i32) -> Self {
        self.values.push(value);
        self
    }

    /// Runs `sum_all` with all recorded values and checks the result.
    fn run(self) {
        let actual = sum_all(&self.values);
        assert_eq!(self.expected, actual);
    }
}

#[test]
fn test_sum_all_with_no_values() {
    SumAllTest::expect(0).run();
}

#[test]
fn test_sum_all_with_one_value() {
    SumAllTest::expect(5).add_value(5).run();
}

#[test]
fn test_sum_all_with_many_values() {
    SumAllTest::expect(8).add_value(3).add_value(4).add_value(1).run();
}

The beauty of this design is that each test case is now declarative. The test case expresses, in code, what the test scenario looks like and what the expectations are. And given the simplicity of the calls, we can trivially express each scenario in its own test case. When we run the tests, we get what we expect:

running 3 tests
test tests::test_sum_all_many_values ... ok
test tests::test_sum_all_no_values ... ok
test tests::test_sum_all_one_value ... ok

test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Great!

But before we go, there is one key detail I glanced over. This detail is critical to ensure that your tests always do the right thing:

IMPORTANT: The test builder has to be annotated with #[must_use]. You may want also want to accompany that with the optional #[deny(warnings)].

You see: given the above design, it’s all too easy to forget to call run() on the test builder—and if you forget to do that, the test will do nothing and will always pass. That’s too risky for test code. Fortunately, by using the #[must_use] annotation on the builder type, the compiler will catch such problems.

To witness: if we remove the run() call from any of the tests above and try to build:

error: unused `SumAllTest` that must be used
  --> src/tests.rs:33:5
   |
33 |     SumAllTest::expect(0);
   |     ^^^^^^^^^^^^^^^^^^^^^^
   |

Rust fails the build and tells us that we forgot to consume the builder object. There is no way we can forget to call run() once we have initialized a SumAllTest object, so the test code will always be exercised.

What do you think? Interesting? Problematic?

For a more realistic example, check out EndBASIC read_line’s own tests.