== Guideline on how and when to write a test == === Why? === MoleCuilder is by now a quite complex piece of code. As of this writing, it has around 90k lines of code and about 700 modules. I.e. the code has lots of cross references, lots of functions are need there and then also quite somewhere else. Hence, if you change the inner workings of one function, maintaining its signature, it will compile, but most likely these functions somewhere else will not continue working as intended as they did before. That's where ''software testing'' comes into play! Via a test you basically define what kind of behavior your function/class will have. You test that for given inputs, required outputs are returned. Such when you change the inner workings, you immediately notice if you changed something crucial as the associated test will (hopefully) fail. ''Hopefully'' because no test can ever be complete, this is NP-hard or something alike to the [http://en.wikipedia.org/wiki/Halting_problem halting problem] of computer science. You might answer to this: I'll just never change the functions ... ''Oh, yes you will'' ... sometimes you have written a function and then stumble upon another section of code where it would fit beautifully if only it were written a bit more generally. And there are lots of more examples ... especially, when you believe in [http://en.wikipedia.org/wiki/Refactoring refactoring] of code! ... Which we do here ... === When? === '''Always'''! ... when in doubt, write a test. When only half in doubt, do write a test. When trying something new, first write a test that encompasses all the stuff the new code should be able to do. When thinking of a new black box of code, define via a test what the outcomes for given inputs of this black box should be. Have a look at [http://en.wikipedia.org/wiki/Software_testing software testing] and [http://en.wikipedia.org/wiki/Unit_tests unit tests] for general ideas on software testing techniques. So, as the why and when questions should now be settled. Let's continue ... === What? === What kind of test should I write. There are two test types supported: * ''Regression tests'' * ''Unit tests'' And the distinction is quite simple: * Write a regression test when you test the ''functionality of an Action''. * Write a unit test when the ''number of required classes''/functions/components is very ''small''. Unit tests are little independent programs that are linked against the classes you want to test. Hence, if lots of other classes are required, you will have lots of dependencies and this will slow down the compilation process enormously in case of many unit tests. Regression tests are basically scripts done via [http://www.gnu.org/software/hello/manual/autoconf/Using-Autotest.html GNU autotest]. They call the MoleCuilder executable specifically with the desired actions and check e.g. the output files to see whether everything was done as intended. Regression test are actually designed as a tool in the refactoring of code but they also serve as a global test on the software as in most cases lots of dependencies are required: e.g. removing an atom needs the World, atom with all inherited, molecule !ActionHistory, !ActionRegistry, ... That is also the reason why these kind of tests are sort of orthogonal to unit tests. === How? === Finally, we answer the questions how to write a test ... ==== Regression test ==== Regression tests can be found in [source:tests/regression]. The basic working is: have a input file, do something with it and compare to output file against a stored one. '''Note:''' There is specific test set on ''tesselations'' in [source:tests/Tesselations] where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now. The same is present for ''fragmentation'' in [source:tests/Fragmentations]. We won't go into the details of these tests here, see section ''Massive tests''. Basically, the regression test is driven via a '''testsuite''' script which is created by autotools. In [source:tests/regression/Makefile.am] you will notice it as the only element of the ''TESTSUITE'' variable given. Note however that some ''EXTRADIRS'' are specified. Hence, if you ever create a new directory for tests, add it hereto. In [source:tests/regression] you notice a single '''testsuite.at''', in its subfolders there are lots of '''.at''' suffixed files. These are the '''a'''uto'''t'''est scripts per test category. So far, we have: * Analysis - analysis functons such as pair correlation, ... * Atoms - Actions that add, remove atoms, ... * Domain - Actions that change the domain by adding empty boundary, ... * Filling - Actions that fill the domain with molecules, such as fill void with molecule, ... * Fragmentation - fragmentation of a bonding graph into subgraphs. * Graph - graph routines that recognize the bonding graph. * Molecules - Actions that change molecules names, ... * Options - tests for options such as help, verbosity, version, ... * Parser - Actions that parse atoms contained in files into the world, also parser-specific actions. * !RandomNumbers - Action that set random number engine or distribution. * Selection - Action that (un)select atoms or molecules. * Tesselation - Actions for creating the tesselated surface of a molecule. Input and output files are stored in the subfolders of [source:tests/regression]. Each specific '''testsuite-....at''' is contained in its own folder called alike. Have a look at '''testsuite.at''' wherefrom '''testsuite''' is constructed bu autotools. It only contains ''m4_include''s to all the test parts. The folder/file hierarchy is as this: * '''tests/regression''' is the folder for all regression tests. * Each test themes has its own subfolder, e.g. '''tests/regression/Analysis'''. * Therein, we need a testsuite file which m4_include's all tests to be done within that theme, called here '''testsuite-analysis.at'''. * Each test has then its own folder and should be called according to the Action under test, e.g. '''!PairCorrelation'''. Therein another autotest file called here '''testsuite-analysis-pair-correlation.at'''. * These folders contain each a '''pre''' and a '''post''' folder. '''pre''' contains input and '''post''' contains output files to test/diff against. The naming convention is as this: * Folder should be called as the Action NAME's, e.g. !PairCorrelation. * Files should be called as the Action TOKEN's, e.g. pair-correlation. ''However'' filenames should contain absolute path (with theme and all), e.g. ''not'' '''testsuite-pair-correlation.at''' ''but'' '''testsuite-analysis-pair-correlation.at'''. On how to write a testsuite test, we refer you to one of these '''.at''' files to get a notion and [http://www.gnu.org/software/hello/manual/autoconf/Writing-Testsuites.html#Writing-Testsuites here] to have a reference of the possible autotest commands at hand. The scheme is always the same basically: {{{ AT_BANNER([Global theme of the test suite section]) AT_KEYWORDS([,,[undo/redo]]) AT_SETUP([small theme of your test]) ... AT_CHECK([this], , [ignore], [ignore]) AT_CHECK([that], , [stdout], [stderr]) AT_CHECk([fgrep "test fine" stdout], 0, [ignore], ignore]) ... AT_CLEANUP #remove all temporary files }}} where '''' is some number the code returns to indicate everything worked fine. The global theme is specified only once per '''.at''' file, file ''AT_SETUP'' and ''AT_CLEANUP'' sort of embrace every specific test that you want to do. Note that it is required to list the action name under ''AT_KEYWORDS'' and also give undo oder redo as an additional keyword if the undo or redo of the action is tested. It is advised to give further keywords, e.g. the directory name giving the general theme of the tests (selection, analysis, ...). Also note that all keywords are always lower-case! Note that testing the undo/redo-functionality of an Action is always placed into the same test file along with the normal functionality but in different tests (i.e. undo and redo each have their own ''AT_SETUP .. AT_CLEANUP'' wrapping). Done. Test via '''make check''' which will also re-create the '''testsuite''' script. ==== Unit test ==== Unit tests are written using the [http://sourceforge.net/apps/mediawiki/cppunit/index.php?title=Main_Page CppUnit] testing framework. Have a look at subfolders called ''unittests'' in [source:src]. These folders containing the unit test soure code are located in the folders with the associated ''component'' or unit to test. Hence, they are also refered to as component tests and this is also their main intention: ''Testing small components of the big code for its specific functionality.'' Let us take the '''!SingletonUnitTest''' as an example, located in [source:ThirdParty/CodePatterns/src/Patterns/unittests] that tests correct implementation of the Singleton pattern in [source:ThirdParty/CodePatterns/src/CodePatterns/Singleton_impl.hpp]. The unit test consists of the following files: * source file with the test implementation, e.g. [source:ThirdParty/CodePatterns/src/Patterns/unittests/SingletonUnitTest.cpp] * header file with the test definition, i.e. the class containing the test and all test functions to call, , e.g. [source:ThirdParty/CodePatterns/src/Patterns/unittests/SingletonUnitTest.hpp] * [source:ThirdParty/CodePatterns/src/unittests/UnitTestMain.cpp] with the main function calling the unit test runner. * additional lines in '''Makefile.am''' to mark the small program as a test and to define the program to consist of your source, header and the !UnitTestMain.cpp file. Writing a new unit test then consists of writing the source and the header file, where you should guide yourself by the already present files. ''Naming convention'' is usually to suffix the name with '''!UnitTest''', e.g. '''!SingletonUnitTest.cpp'''. The test class should be named in a similar manner and has the following look: {{{ class SingletonTest : public CppUnit::TestFixture { CPPUNIT_TEST_SUITE( SingletonTest ); CPPUNIT_TEST ( blaTest ); CPPUNIT_TEST ( blablaTest ); CPPUNIT_TEST_SUITE_END(); public: void setUp(); void tearDown(); void blaTest(); void blablaTest(); }; }}} It inherits CppUnit::!TestFixture and defines it as a TEST_SUITE via the initial macros which also give the test functions to call. Below the test functions are defined along with ''setUp()'' and ''tearDown()'' which embrace the call of each test function, i.e. which create a common test environment for each test function, by allocating some memory, setting some stuff and cleaning up in the end again. See the [http://cppunit.sourceforge.net/doc/lastest/cppunit_cookbook.html CppUnit Cookbook] for a guide on how to write these tests. ==== Massive test ==== '''Note:''' The Massive tests use purely automake constructs and has nothing to do with autotest. There are two more directories within '''tests''', ''tesselation'' and ''fragmentation''. There we want to test the algorithms on a whole range of different molecules to check whether all of these are working. This is too large a part for use within '''tests/regression''', hence tests as these are placed into an extra folder and have a different structure: * Have a '''Makefile.am''', where all ''TESTS'' are listed (needs to be installed in configure.ac). * Each entry in ''TESTS'' is a small file suffixed by '''.test'''. * It contains a certain call of the code along with all necessary checks, returning 0 if the test is ok, else if it failed. * In a helper file, called '''defs.in''' specific variables and functions might be implemented as helpers (needs to be installed in configure.ac) Automake will then dive into the subfolder and execute each file in ''TESTS'', telling whether the test was ok or failed. === How to use/call these tests? === Eventually, you may want to call these tests, either all at once or individually ... To call them all, simply type {{{ make check }}} which will call both all unit tests and the testsuite. ==== Regression test ==== If you want to call an individual regression test, you can access the '''testsuite''' program in [source:tests/regression/testsuite] directly by giving it the number or range of a test. We sssume your build directory is called '''build''', and you are right now in '''./build/tests/regression/''' and you want to start tests 5-8 and 10, enter {{{ ../../../tests/regression/testsuite 5-8 10 AUTOTEST_PATH=path/to/molecuilder/build/src/ }}} and only these tests will get executed. You can also specify a number of tests by its keywords, try {{{ ../../../tests/regression/testsuite --help }}} get obtain more information or see [http://www.gnu.org/software/hello/manual/autoconf/testsuite-Invocation.html#testsuite-Invocation here]. ==== Unit test ==== Calling single unit tests is even easier. Assumptions from above, we want to call '''!FormulaUnittest''' and reside in '''./build''', enter {{{ src/unittests/FormulaUnittest }}} The test will state '''(OK)''' when without error. Otherwise it will state '''FAIL''' and the code lines of all tests that failed ==== Massive test ==== Calling the massive tests only involves stepping into the respective subfolder and calling {{{ ../../../tests/Tesselations/benzene.test }}} which will then execute the '''benzene.test''' test. == If something is broken == At the very last, we have to talk about what to do with the test if something is broken. If a unit test is broken, you might only modify the code directly. When it is a regression test, you can modify the return value to state that something will not work. However ... '''Beware when changing tests! ''' These should never be done lightly and changes to tests HAVE to be given in full in the description of the commit to the repository, also state why the change was necessary. Maybe the test was not complete or faulty itself. The idea is that even when some tests are not working, '''make check''' should run through smoothly. However, it should indicate that something is amiss here, e.g. * put a line just after '''AT_SETUP''' as follows: {{{ AT_XFAIL_IF([/bin/true]) }}} this will tell that the test is supposed to fail and allow for the continuation of the remainder of test suite. Fix the test as soon as possible. * unit tests should print a warning message.