Guideline on how and when to write a test

Why?

MoleCuilder is by now a quite complex piece of code. As of this writing, it has around 90k lines of code and about 700 modules. I.e. the code has lots of cross references, lots of functions are need there and then also quite somewhere else. Hence, if you change the inner workings of one function, maintaining its signature, it will compile, but most likely these functions somewhere else will not continue working as intended as they did before.

That's where software testing comes into play!

Via a test you basically define what kind of behavior your function/class will have. You test that for given inputs, required outputs are returned. Such when you change the inner workings, you immediately notice if you changed something crucial as the associated test will (hopefully) fail. Hopefully because no test can ever be complete, this is NP-hard or something alike to the ​halting problem of computer science.

You might answer to this: I'll just never change the functions ... Oh, yes you will ... sometimes you have written a function and then stumble upon another section of code where it would fit beautifully if only it were written a bit more generally. And there are lots of more examples ... especially, when you believe in ​refactoring of code! ... Which we do here ...

When?

Always! ... when in doubt, write a test. When only half in doubt, do write a test. When trying something new, first write a test that encompasses all the stuff the new code should be able to do. When thinking of a new black box of code, define via a test what the outcomes for given inputs of this black box should be.

Have a look at ​software testing and ​unit tests for general ideas on software testing techniques.

So, as the why and when questions should now be settled. Let's continue ...

What?

What kind of test should I write.

There are two test types supported:

• Regression tests
• Unit tests

And the distinction is quite simple:

• Write a regression test when you test the functionality of an Action.
• Write a unit test when the number of required classes/functions/components is very small.

Unit tests are little independent programs that are linked against the classes you want to test. Hence, if lots of other classes are required, you will have lots of dependencies and this will slow down the compilation process enormously in case of many unit tests.

Regression tests are basically scripts done via ​GNU autotest. They call the MoleCuilder executable specifically with the desired actions and check e.g. the output files to see whether everything was done as intended.

Regression test are actually designed as a tool in the refactoring of code but they also serve as a global test on the software as in most cases lots of dependencies are required: e.g. removing an atom needs the World, atom with all inherited, molecule ActionHistory, ActionRegistry, ... That is also the reason why these kind of tests are sort of orthogonal to unit tests.

How?

Finally, we answer the questions how to write a test ...

Regression test

Regression tests can be found in tests/regression. The basic working is: have a input file, do something with it and compare to output file against a stored one.

Note: There is specific test set on tesselations in tests/Tesselations where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now. The same is present for fragmentation in tests/Fragmentations. We won't go into the details of these tests here, see section Massive tests.

Basically, the regression test is driven via a testsuite script which is created by autotools. In tests/regression/Makefile.am​ you will notice it as the only element of the TESTSUITE variable given. Note however that some EXTRADIRS are specified. Hence, if you ever create a new directory for tests, add it hereto.

In tests/regression you notice a single testsuite.at, in its subfolders there are lots of .at suffixed files. These are the autotest scripts per test category. So far, we have:

• Analysis - analysis functons such as pair correlation, ...
• Atoms - Actions that add, remove atoms, ...
• Domain - Actions that change the domain by adding empty boundary, ...
• Filling - Actions that fill the domain with molecules, such as fill void with molecule, ...
• Fragmentation - fragmentation of a bonding graph into subgraphs.
• Graph - graph routines that recognize the bonding graph.
• Molecules - Actions that change molecules names, ...
• Options - tests for options such as help, verbosity, version, ...
• Parser - Actions that parse atoms contained in files into the world, also parser-specific actions.
• RandomNumbers - Action that set random number engine or distribution.
• Selection - Action that (un)select atoms or molecules.
• Tesselation - Actions for creating the tesselated surface of a molecule.

Input and output files are stored in the subfolders of tests/regression. Each specific testsuite-....at is contained in its own folder called alike. Have a look at testsuite.at wherefrom testsuite is constructed bu autotools. It only contains m4_includes to all the test parts.

The folder/file hierarchy is as this:

• tests/regression is the folder for all regression tests.
• Each test themes has its own subfolder, e.g. tests/regression/Analysis.
• Therein, we need a testsuite file which m4_include's all tests to be done within that theme, called here testsuite-analysis.at.
• Each test has then its own folder and should be called according to the Action under test, e.g. PairCorrelation. Therein another autotest file called here testsuite-analysis-pair-correlation.at.
• These folders contain each a pre and a post folder. pre contains input and post contains output files to test/diff against.

The naming convention is as this:

• Folder should be called as the Action NAME's, e.g. PairCorrelation.
• Files should be called as the Action TOKEN's, e.g. pair-correlation. However filenames should contain absolute path (with theme and all), e.g. not testsuite-pair-correlation.at but testsuite-analysis-pair-correlation.at.

On how to write a testsuite test, we refer you to one of these .at files to get a notion and ​here to have a reference of the possible autotest commands at hand. The scheme is always the same basically:

AT_BANNER([Global theme of the test suite section])
AT_KEYWORDS([<some keywords>,<actionname>,[undo/redo]])
AT_SETUP([small theme of your test])
...
AT_CHECK([this], <return value>, [ignore], [ignore])
AT_CHECK([that], <return value>, [stdout], [stderr])
AT_CHECk([fgrep "test fine" stdout], 0, [ignore], ignore])
...
AT_CLEANUP #remove all temporary files

where <return value> is some number the code returns to indicate everything worked fine. The global theme is specified only once per .at file, file AT_SETUP and AT_CLEANUP sort of embrace every specific test that you want to do. Note that it is required to list the action name under AT_KEYWORDS and also give undo oder redo as an additional keyword if the undo or redo of the action is tested. It is advised to give further keywords, e.g. the directory name giving the general theme of the tests (selection, analysis, ...). Also note that all keywords are always lower-case!

Note that testing the undo/redo-functionality of an Action is always placed into the same test file along with the normal functionality but in different tests (i.e. undo and redo each have their own AT_SETUP .. AT_CLEANUP wrapping).

Done. Test via make check which will also re-create the testsuite script.

Unit test

Unit tests are written using the ​CppUnit testing framework.

Have a look at subfolders called unittests in src. These folders containing the unit test soure code are located in the folders with the associated component or unit to test. Hence, they are also refered to as component tests and this is also their main intention: Testing small components of the big code for its specific functionality.

Let us take the SingletonUnitTest as an example, located in ThirdParty/CodePatterns/src/Patterns/unittests that tests correct implementation of the Singleton pattern in ThirdParty/CodePatterns/src/CodePatterns/Singleton_impl.hpp​. The unit test consists of the following files:

• source file with the test implementation, e.g. ThirdParty/CodePatterns/src/Patterns/unittests/SingletonUnitTest.cpp​
• header file with the test definition, i.e. the class containing the test and all test functions to call, , e.g. ThirdParty/CodePatterns/src/Patterns/unittests/SingletonUnitTest.hpp​
• ThirdParty/CodePatterns/src/unittests/UnitTestMain.cpp​ with the main function calling the unit test runner.
• additional lines in Makefile.am to mark the small program as a test and to define the program to consist of your source, header and the UnitTestMain.cpp file.

Writing a new unit test then consists of writing the source and the header file, where you should guide yourself by the already present files. Naming convention is usually to suffix the name with UnitTest, e.g. SingletonUnitTest.cpp.

The test class should be named in a similar manner and has the following look:

class SingletonTest : public CppUnit::TestFixture
{
  CPPUNIT_TEST_SUITE( SingletonTest );
  CPPUNIT_TEST ( blaTest );
  CPPUNIT_TEST ( blablaTest );
  CPPUNIT_TEST_SUITE_END();

public:
  void setUp();
  void tearDown();

  void blaTest();
  void blablaTest();
};

It inherits CppUnit::TestFixture and defines it as a TEST_SUITE via the initial macros which also give the test functions to call. Below the test functions are defined along with setUp() and tearDown() which embrace the call of each test function, i.e. which create a common test environment for each test function, by allocating some memory, setting some stuff and cleaning up in the end again.

See the ​CppUnit Cookbook for a guide on how to write these tests.

Massive test

Note: The Massive tests use purely automake constructs and has nothing to do with autotest.

There are two more directories within tests, tesselation and fragmentation. There we want to test the algorithms on a whole range of different molecules to check whether all of these are working. This is too large a part for use within tests/regression, hence tests as these are placed into an extra folder and have a different structure:

• Have a Makefile.am, where all TESTS are listed (needs to be installed in configure.ac).
• Each entry in TESTS is a small file suffixed by .test.
• It contains a certain call of the code along with all necessary checks, returning 0 if the test is ok, else if it failed.
• In a helper file, called defs.in specific variables and functions might be implemented as helpers (needs to be installed in configure.ac)

Automake will then dive into the subfolder and execute each file in TESTS, telling whether the test was ok or failed.

How to use/call these tests?

Eventually, you may want to call these tests, either all at once or individually ...

To call them all, simply type

make check

which will call both all unit tests and the testsuite.

Regression test

If you want to call an individual regression test, you can access the testsuite program in tests/regression/testsuite directly by giving it the number or range of a test. We sssume your build directory is called build, and you are right now in ./build/tests/regression/ and you want to start tests 5-8 and 10, enter

../../../tests/regression/testsuite 5-8 10 AUTOTEST_PATH=path/to/molecuilder/build/src/

and only these tests will get executed. You can also specify a number of tests by its keywords, try

../../../tests/regression/testsuite --help

get obtain more information or see ​here.

Unit test

Calling single unit tests is even easier. Assumptions from above, we want to call FormulaUnittest and reside in ./build, enter

src/unittests/FormulaUnittest

The test will state (OK) when without error. Otherwise it will state FAIL and the code lines of all tests that failed

Massive test

Calling the massive tests only involves stepping into the respective subfolder and calling

../../../tests/Tesselations/benzene.test

which will then execute the benzene.test test.

If something is broken

At the very last, we have to talk about what to do with the test if something is broken. If a unit test is broken, you might only modify the code directly. When it is a regression test, you can modify the return value to state that something will not work. However ...

Beware when changing tests!

These should never be done lightly and changes to tests HAVE to be given in full in the description of the commit to the repository, also state why the change was necessary. Maybe the test was not complete or faulty itself.

The idea is that even when some tests are not working, make check should run through smoothly. However, it should indicate that something is amiss here, e.g.

• put a line just after AT_SETUP as follows:

  AT_XFAIL_IF([/bin/true])
  

  this will tell that the test is supposed to fail and allow for the continuation of the remainder of test suite. Fix the test as soon as possible.
• unit tests should print a warning message.