A C++11 Unit Testing Framework based on Test::More
Modern software development knows the benefits of automated testing over (or as a complement to) dedicated QA staff. Automated testing gives a quicker bug cycle for developers and allows them to catch bugs introduced in code changes by other modules or modifications to the same module. Indeed, automated unit tests are a corner stone of Extreme Programming, and many software projects don't allow new code to be checked in without accompanying automated tests.
On the other hand, most C++ unit testing frameworks are heavy-weight and restricting. They are filled with unnecessary macros which can present problems, and they are often not thread-safe, which makes testing concurrent code even more difficult.
cipra was designed as a response to this, with the following goals in mind:
cipraclosely mirrors that of Perl's
Test::More, a widely-used testing module in the Perl community that elegantly works well in C++.
cipramanage this themselves only if they need it.
Test::Moremodule as a guide,
ciprais thoroughly a C++ library. Written in C++11, it eschews macros and is thread-safe, using modern C++ techniques that aren't intrusive on the code you want to test. We don't require anything other than a C++11 compiler.
This tutorial should give you an introduction to using
cipra. It expects C++11 experience, but doesn't require any knowledge of
Test::More or Perl testing.
cipra is a header-only library, which means you don't have to compile anything to use it. Simply install the header files to your system or package them in your project, and you can start using
cipra as a test framework. Make sure you are using a C++11 compliant compiler.
You will need a TAP13 test harness to track and pretty-print the output of
cipra test programs. Right now, the best option is to use the
prove program that comes with Perl. In a bind, the following UNIX command can also be used to check for failures, assuming your test program is named
In the future,
cipra will provide a test harness written entirely in C++11, making
cipra tests dependency-free. In this tutorial, we will be showing the output of
cipra itself, not run through any test harness.
Let's start with a very simple test program using
cipra. We'll assume that the header files are in your system include path (placed in
/usr/include, for instance, or compiled in with a
-I flag on GCC). Start off by including
cipra, all test programs need to have a class derived from
cipra::fixture, which contains all the test functions in
cipra. The actual tests are contained in the overridden
test() function. Let's make a class that will contain our tests and then make an instance of this class:
By calling the
run() method of
test, we get the return value of the test, running all the test functions and then quitting.
test() function, we need to declare how many tests we are planning to run, just in case we exit from the test prematurely. For our first test program, we'll just have a single test, so we call the
plan() function at the beginning of the
test() function with the argument
We're finally ready to actually run a test. It will be a simple test, making sure that
5 == 5 is
true. To do this, we use
cipra's most basic test function template,
ok(), which is declared as follows:
funcT is anything that has an
operator() whose return value is convertible to a
bool. This could be a function pointer, a functor, the result of
std::bind(), or a C++11 lambda. Instead of declaring our test code separately, we prefer to use lambdas and declare the test directly in the call to
ok(). The second argument is a description of the test that will be printed with the test output. When reading the test output at a glance, it's much easier to see which test is failing if you include these descriptions. We recommend that you always do this. Let's write a call to
ok() that checks whether
5 == 5:
Here is the full code of our first test file:
If we run the above code, we get the following output:
This output is in a format called TAP13, short for the "Test Anything
Protocol, version 13". TAP is standard format used by many test harnesses, especially those within the Perl community.
Test::More, the Perl module on which
cipra is based, uses TAP. Because of the wide support for TAP,
cipra chooses it as a standard output format.
For comparison, here is a Perl script using
Test::More that performs the same test:
The C++ code using
cipra does contain a lot of boilerplate code compared with the Perl code using
Test::More. Ideally, this code should be reduced. The goal for a future version of
cipra is to reduce the code to only the following:
In real testing environments, though, tests fail. Let's add a failing test:
5 == 6 is obviously false, so this test should fail. If we run the test program, we'll get the following output:
This is TAP's way of telling us that the test failed.
So far, we've called the
plan() function using an integer argument, indicating the number of test functions that we are expecting to call in the test program. This function is responsible for the
1..n line in the above output, where
n is the number you pass in. That line is called the test plan in TAP terms. By including it, you declare to the test harness how many tests you are expecting to run, so it can tell whether the test was aborted prematurely.
In our tests so far, the test plan has come at the very beginning of the output, before any of the test functions ran. Sometimes, though, you don't know how many tests are going to run. Consider the following test program snippet:
We don't know how many tests should have run until after the file has been read. In this case, we can put the test plan at the end of the test. This does not mean that the test plan can go anywhere in the test program. The test plan must come at either the beginning or the end of the test output. We modify our code and get the right plan:
and the output
If we want to skip all the tests in the test program (for example, if the test is testing behavior that isn't implemented yet), we can call the function with
skip_all as the argument. It only makes sense to do this before all test functions.
TAP supports outputting information from the test program to the test harness that won't be interpreted, but will instead be considered comments to be displayed to the user.
cipra implements these with the
note() functions, as well as the
explain() function template.
The difference between
note() is subtle. While both output a TAP diagnostic,
diag() prints the message to
note() prints the message to
stdout. Consider the following code:
The output of this code will be
If you want to pretty-print an object using its streaming
operator<<(), use the function template
explain(). You can also specialize the
cipra::print_trait template to use a different function.
As we stated earlier, the
ok() function is the only test function you really need. Every one of the other test functions in
cipra could be replaced by calls to
ok(). These other test functions are simply conveniences that express intent. Because they express intent, they can make the test source code easier to read and can provide better diagnostic information when the test fails.
The two most simple test functions are
fail(). Their signatures are below:
The the function call
pass(name) is equivalent to
ok(true, name), while the function call
fail(name) is equivalent to
Another test function template is
is(), which compares two values with their
We could rewrite our first test using
Running this, we get the same output as before:
If we have two types that don't have an
operator== function, we can specialize the
cipra::equal_traits template on our types.
What about our failing test?
Compared to the output we got with
ok(), a failing
is() gives much better diagnostics. You need to provide an
operator<<() function for both types so it can be outputted or specialize the
Similarly, there is an
isnt() function template which checks whether the two values are not equal.
In addition to simple truth values,
cipra can also test exceptions from a function. Normally, exceptions cause the test functions we've seen to fail in the same way that a
false value does. The exceptions are not propagated outside the test function, so we can't easily check whether the test function failed because of an exception or because of a
false return value. If we need to, we can wrap our lambda code in a
catch block. Let's say the function
double div(double, double) throws an exception when its second argument is
0.0. If we want to check that this correctly throws an exception, we might write the following test:
Though this code does exactly what we want it to, the actual test is very verbose, because the
ok() function doesn't semantically encode the exception test. Luckily, though,
cipra provides the function templates
throws() that do:
The first overload just checks that the
expr throws some exception, while the second overload checks that
expr throws an exception of the type
funcT. In our case, we can simplify our code with the first overload:
If we want to test that it is of the type
std::runtime_error (or a derived class from this type), we can use the second overload:
This overload succeeds only when a
std::runtime_error is thrown. If any other type of exception is thrown, it will catch it and fail, printing diagnostic information. If no exception is thrown, the test function will also fail.
On compilers that support the CXA ABI (for instance, GCC), you can define the preprocessor symbol
CIPRA_CXA_ABI before including
cipra will then use information available from the
abi::__cxa_current_exception_type() function to determine the name of the exception type and will output it with the diagnostics. Otherwise, the name of the exception won't be available.
cipra also has two function templates
nothrows() which do the inverse of the
Take the following class:
If we want to test the constructor, we could write the following code:
v's constructor will fail and throw an exception, which causes the first test function to succeed.
w's constructor will succeed, and
w will be properly constructed, so the second test function will also succeed. Because
w is inside a lambda, though, we can't use it in subsequent code. The third test statement will fail to compile, because
w isn't in scope. We could write the following code to reconstruct it in the outer scope:
w failed to construct the first time, it will fail to construct again, and we'll have leaked an exception.
cipra provides a function template specifically to test constructors. The function template, called
new_ok, has the following signature:
new_ok constructs a object of type
T's constructor that takes the given
args. If the constructor of the object fails (that is, if the constructor throws an exception),
new_ok will abort from the test program. This function requires that
T either be
Movable. Since the
value<double> class in our example is
Copyable, we can simplify our test snippet:
w is in the outer scope now, and it can be used in subsequent test calls, like the
is() call. If the constructor were to fail, the
is() would never be called, and the test would exit.
Copyright (C) 2013, Patrick M. Niedzielski
This work is licensed under a Creative Commons Attribution 3.0 United States License. You should have received a copy of the CC-BY 3.0 US license along with this file. If not, see the Creative Commons website.