Guideline on how and when to write a test

Why?

MoleCuilder is by now a quite complex piece of code. as of this writing, it has around 90k lines of code and about 700 modules. I.e. the code has lots of cross references, lots of functions are need there and then also quite somewhere else. Hence, if you change the inner workings of one function, keeping its signature, it will compile, but most likely these functions somewhere else will not continue working as before.

That's where software testing comes into play!

Via a test you basically define what kind of behavior your function/class will have. You test that for given inputs, required outputs are returned. Such when you change the inner workings, you immediately notice if you changed something crucial as the associated test will (hopefully) fail. Hopefully because no test can ever be complete, this is NP-hard or something alike to the ​halting problem of computer science.

You might answer to this: I'll just never change the functions ... Oh, yes you will ... sometimes you have written a function and then stumble upon another section of code where it would fit beautifully if only it were written a bit more generally. And there are lots of more examples ... especially, when you believe in ​refactoring of code! ... Which we do here ...

When?

Always! ... when in doubt, write a test. When only half in doubt, do write a test. When trying something new, first write a test that encompasses all the stuff the new code should be able to do. When thinking of a new black box of code, define via a test what the outcomes for given inputs of this black box should be.

Have a look at ​software testing and ​unit tests for general ideas on software testing techniques.

So, as the when question should now be settled. Let's continue ...

What?

What kind of test should I write.

There are two test types supported:

• Regression tests
• Unit tests

And the distinction is quite simple:

• Write a regression test when you test the functionality of an Action.
• Write a unit test when the number of required class is very small.

Unit tests are little independent programs that are linked against the classes you want to test. Hence, if lots of other classes are required, you will have lots of dependencies and this will slow down the compilation process in case of many unit tests enormously.

Regression tests are basically scripts done via ​GNU autotest. They call the MoleCuilder executable specifically with the desired actions and check e.g. the output files to see whether everything was done as intended.

Regression test are actually designed as a tool in the refactoring of code bu they also serve as a global test on the software as in most cases lots of dependencies are required: e.g. removing an atom needs the World, atom with all inherited, molecule ActionHistory, ActionRegistry, ...

How?

Finally, we answer the questions how to write a test ...

Regression test

Regression tests can be found in tests/regression.

Note: There is specific test set on tesselations in tests/Tesselations where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now.

Basically, the regression test is driven via a testsuite script which is created by autotools. In tests/regression/Makefile.am​ you will notice it as the only element of the TESTSUITE variable given. Note however that some EXTRADIRS are specified.

In tests/regression you notice a lot of .at suffixed files. These are the autotest scripts per test category. So far, we have:

• Analysis - analysis functons such as pair correlation, ...
• Domain - Actions that change the domain by adding empty boundary, ...
• Filling - Actions that fill the domain with molecules, such as fill void with molecule, ...
• Fragmentation - fragmentation of a bonding graph into subgraphs
• Graph - graph routines that recognize the bonding graph
• Molecules - Actions that change molecules names, ...
• Simple Configuration - tests for loading and saving files, adding, removing atoms
• Specifics - very specific stuff as specifying the basis set stored in MPQC-type files
• Standard Options - tests for options such as help, verbosity, version, ...
• Tesselation - Actions for creating the tesselated surface of a molecule

On how to write a testsuite test, we refer you to these files to get a notion and ​here to have a reference of the possible autotest commands at hand. The scheme is always the same basically:

AT_BANNER([Global theme of the test suite section])
AT_SETUP([small theme of your test])
...
AT_CHECK([this], <return value>, [ignore], [ignore])
AT_CHECK([that], <return value>, [stdout], [stderr])
AT_CHECk([fgrep "test fine" stdout], 0, [ignore], ignore])
...
AT_CLEANUP #remove all temporary files

where <return value> is some number the code returns to indicate everything worked fine.

Unit test

How to use/call these tests?

Eventually, you may want to call these tests, either all at once or individually ...

To call them all, simply type

make check

which will call both all unit tests and the testsuite.

Regression test

If you want to call an individual regression test, you can access the testsuite program in tests/regression/testsuite directly by giving it the number or range of a test. We sssume your build directory is called build, and you are right now in ./build/tests/regression/ and you want to start tests 5-8 and 10, enter

../../../tests/regression/testsuite 5-8 10

and only these tests will get executed. You can also specify a number of tests by its keywords, try

../../../tests/regression/testsuite --help

get obtain more information or see ​here.

Unit test

Calling single unit tests is even easier. Assumptions from above, we want to call FormulaUnittest and reside in ./build, enter

src/unittests/FormulaUnittest

If something is broken

At the very last, we have to talk about what to do with the test if something is broken. If a unit test is broken, you might only modify the code directly. When it is a regression test, you can modify the return value to state that something will not work.

However beware when changing tests!

These should never be done lightly and changes to tests HAVE to be given in full in the description of the commit to the repository, also state why the change was necessary. Maybe the test was not complete or faulty itself.

The idea is that even when some tests are not working, make check should run through smoothly. However, it should indicate that something is amiss here, e.g.

• change the message in AT_SETUP to something that contains BROKEN or alike
• unit tests should