Another C/C++ Testing Idea - 2 phase execution

Languages that support reflection allow you to mark a function as a test which the test framework then automatically runs. Not having to explicitly run the test is nice. C and C++ don't support runtime reflection so at some level you have to also explicitly run the function. That's not so nice. But... I've an idea. One that I mentioned it to my friend Kevlin Henney and it turns out he was thinking the same kind of thoughts. As usual he was a few steps ahead of me. The idea is don't use functions! Make the basic unit of testing a block instead of a function! For example, instead of having three separate functions naming three separate tests why not have three separate named blocks inside a single function. Something like this:
void test_holder()
{
   TEST("first")
   {  ... /* assertion */
   }
   TEST("second")
   {  ... /* assertion */
   }
   TEST("third")
   {  ... /* assertion */
   }
}
The test framework then calls the function three times, ensuring each block gets run once. How does it know to call the function three times? The idea is to have two phases, a gather phase and a run phase. First set the phase to gather and call the function. In this phase the TEST macro does not execute, instead it inserts itself into a collection of test blocks. When all the test blocks have been gathered switch to run phase and run the test blocks one at a time. Here's a bare bones implementation of the idea (in C99)
#include <assert.h>
#include <stdbool.h>

#define TEST(name)   if (run_after_gather(blocks, #name))
#define IGNORE(name) if (ignore())

struct test_blocks
{
    int size;
    const char * names[1024];
    bool in_gather_phase;
    const char * run_name;
};

bool run_after_gather(struct test_blocks * blocks, const char * name)
{
    if (blocks->in_gather_phase) 
    {
        blocks->names[blocks->size] = name;
        blocks->size++;
        return false;
    }
    else
        return blocks->run_name == name;
}

bool ignore(void) { return false; }

void run(void (*test)(struct test_blocks *))
{
    struct test_blocks blocks = { .size = 0 };
    blocks.in_gather_phase = true;
    test(&blocks);
    blocks.in_gather_phase = false;
    for (int at = 0; at != blocks.size; at++)
    { 
        blocks.run_name = blocks.names[at];
        test(&blocks);
    }
}

/* - - - - - - - - - - - - - */

void tests(struct test_blocks * blocks)
{
    TEST(first)
    {
        assert(1==1);
    }
    IGNORE(second)
    {
        assert(2==21);
    }
    TEST(third and last)
    {
        assert(3==31);
    }
}

int main()
{
    run(tests);
    return 0;
}

15 comments:

  1. Another interesting concept!

    You've added an extra level of indirection so you only need to define one "test method" per test file. So you've pushed the problem down one level. But unless I'm missing something, the original problem's still there.

    You still have to specifying that ONE test method exists in each test file somehow, rather than that A NUMBER OF methods exist.

    It's a real shame that we have to try to jump through such hoops in C and C++.

    ReplyDelete
  2. Yes you still have to register the actual function. But at least you can choose the function/block granularity mix.

    ReplyDelete
  3. My unit testing framework uses a single function per file, and then each test case is a spearate scope. This allows you to share data and objects across 'related' tests by defining them at the top of the tests scope instead of using a 'setup' method. Then RAII kicks in at the end instead of neading a 'teardown' method.

    I know that each test should be independent with setup and teardown called between tests to avoid any residue, but in reality does it matter? As long as the test dummies are immutable or the method/function is stateless I don't see the problem.

    Presuambly the other benefit of running tests independently is speed. You can concentrate on a single test. Is it so bad that you actually run all tests in a single file each time? If they run quickly (which they should) it won't affect you.

    In short, this is the direction I have been heading, but I've not been doing it very long, so feel free to point out my folly :-)

    ReplyDelete
  4. Yes the ability to put code above the first block creates a nice way to do a set-up. Similarly after the last block a tear-down.
    The ability to run each block individually is not really about speed. It's more about control. For example, you can quite easily create a new variant macro called TEST_ONLY which disables all the other TEST blocks. Similarly rather than physically commenting out tests you can create another variant macro TEST_IGNORE that disables just that one.

    ReplyDelete
  5. What problem are you trying to solve? It seems like a solution looking for an application.

    If you have to write tests like this most likely your code is badly factored.

    ReplyDelete
  6. The tests would be the same regardless of whether they were written in separate functions on in separate blocks in one function so I don't see what you mean about the code being badly factored.
    This is simply an alternative way of structuring the tests. One which allows you to spend less time manually calling test functions and more time writing tests.

    ReplyDelete
  7. Indeed, this has absolutely nothing to do with badly factored code.

    And I *really* like the idea of setup/teardown in the one function body, clearly related in scope terms with each test. I see that as a very neat factoring, and a rather compelling reason to follow this test code structure. It shows the object lifetimes in the test harness much more clearly than doing it constructors/destructors or setUp/tearDown functions that might be elsewhere in your test file.

    Neat.

    However, I wonder whether this scheme would become rather unwieldy once you get a large set of tests in the file? Your test code is already two blocks of indentation in before you've started writing them. Therefore this might become clumsy?

    Of course, your test code is always so simple and short as to not make this a problem? :-)

    ReplyDelete
  8. One thing to be aware of is that any statements in the set-up and tear-down bits will be run once in the gather phase _before_ being run in the run phase.

    ReplyDelete
  9. Yeah, that is a shame, although for most tests oughtn't be a problem.

    Of course, you could always have *another* macro wrapped around them so they're only created in the run phase? I'll get my coat...

    ReplyDelete
  10. A problem with trying to make the setup bit not run in the gather phase is that you can't put it in a block. If you do that anything inside it is not in scope for the test blocks.
    I have an idea that might work but I haven't worked it through yet.

    ReplyDelete
  11. This is a good idea. However, I think it can be simplified even more - why do you have to do two passes? Just run everything in one pass, using TEST macro to add current test info and such; you can use setjmp/try-catch to catch errors at this level also.

    Note that if you're using C++ this is not necessary since you can declare a static initializer object in TEST macro to auto-register the tests; it'll work fine as long as linker does not strip the object files away.

    ReplyDelete
  12. Oh, Chris described the same thing with one pass run. My apologies.

    ReplyDelete
  13. Anonymous11:45 am

    This comment has been removed by a blog administrator.

    ReplyDelete
  14. Again an ugly approach (IMHO ;-) Check out CUTE and its plug-in for Eclipse CDT. (just to make sure everyone gets it). The plug-in provides the required registration without too many ugly macro tricks (only to get __FILE__ and __LINE__)

    Regards
    Peter Sommerlad
    (sorry for advertising)

    ReplyDelete
  15. Peter, how does CUTE auto register tests if you are working in C rather than C++ and not using Eclipse?

    ReplyDelete