what should all tests passing look like?

I was fixing a fault in cyber-dojo the other day and I happened to be looking at the starting test code for C++. I'd written it using an array of function pointers. At the end of a practice the array might look like this (in main I simply iterate through the array and call each of the functions):
typedef void test();

static test * tests[ ] =
{
    a_recently_used_list_is_initially_empty,
    the_most_recently_added_item_is_always_first,
    items_can_be_looked_up_by_index_which_counts_from_zero,
    items_in_the_list_are_unique,
};
I imagined it written like this instead:
int main()
{
    a_recently_used_list_is_initially_empty();
    the_most_recently_added_item_is_always_first();
    items_can_be_looked_up_by_index_which_counts_from_zero();
    items_in_the_list_are_unique();
}
I wondered why I was putting the function pointers into an array at all. The most obvious thing I lose by not using an array is that I can no longer print out the number of tests that have passed. I thought about that a bit. I started to wonder what benefits I actually get by being told how many tests had run. It's a 100% quantity and 0% quality measurement.

The most obvious benefit I can think of is that, after writing a new test, I can verify it has run by seeing the number of tests increase by one. The problem is I don't think that's a benefit at all. I think that interaction could easily encourage me to not start with a failing test.

If all tests passing produces any output I will look at the output and start to rely on it. That doesn't feel right for a supposedly automated test.

If each passing test introduces any new output, even if it's just a single dot, then I'm in danger of falling into a composition trap. Those dots don't make a difference individually, but sooner or later they will collectively.

If the code I'm testing doesn't produce any output when it runs (which is very likely for unit-tests) then is it right for my tests to introduce any output?

If I want to time how long the tests take to run shouldn't that be part of the script that runs the tests?

I think there is something to be said for all tests passing producing no output at all. None. Red means a test failed and the output should be as helpful as possible in identifiying the failure. No output means green. Green means no output.

Thoughts?

2 comments:

  1. No output is indistinguishable from no-run which will also produce no output, but definition. Error code or system messages may help detection of no-run, but a system message shouldn't be seen as output from a test run, IMO.

    ReplyDelete
    Replies
    1. If I have a failing test then I will get output. If I now make it pass the output will go away. However, this is only true _during_ the process of writing the code and tests. I need to also consider the situation where you come back to the tests after a while. Thanks

      Delete