Changes between Version 2 and Version 3 of TestingGuidelines


Ignore:
Timestamp:
Dec 10, 2010, 2:02:36 PM (15 years ago)
Author:
FrederikHeber
Comment:

extended to yet missing parts

Legend:

Unmodified
Added
Removed
Modified
  • TestingGuidelines

    v2 v3  
    44=== Why? ===
    55
    6 MoleCuilder is by now a quite complex piece of code. as of this writing, it has around 90k lines of code and about 700 modules. I.e. the code has lots of cross references, lots of functions are need there and then also quite somewhere else. Hence, if you change the inner workings of one function, keeping its signature, it will compile, but most likely these functions somewhere else will not continue working as before.
     6MoleCuilder is by now a quite complex piece of code. As of this writing, it has around 90k lines of code and about 700 modules. I.e. the code has lots of cross references, lots of functions are need there and then also quite somewhere else. Hence, if you change the inner workings of one function, maintaining its signature, it will compile, but most likely these functions somewhere else will not continue working as intended as they did before.
    77
    8 That's where software testing comes into play!
     8That's where ''software testing'' comes into play!
    99
    1010Via a test you basically define what kind of behavior your function/class will have. You test that for given inputs, required outputs are returned. Such when you change the inner workings, you immediately notice if you changed something crucial as the associated test will (hopefully) fail. ''Hopefully'' because no test can ever be complete, this is NP-hard or something alike to the [http://en.wikipedia.org/wiki/Halting_problem halting problem] of computer science.
     
    1919Have a look at [http://en.wikipedia.org/wiki/Software_testing software testing] and [http://en.wikipedia.org/wiki/Unit_tests unit tests] for general ideas on software testing techniques.
    2020
    21 So, as the when question should now be settled. Let's continue ...
     21So, as the why and when questions should now be settled. Let's continue ...
    2222
    2323=== What? ===
     
    3030
    3131And the distinction is quite simple:
    32  * Write a regression test when you test the functionality of an Action.
    33  * Write a unit test when the number of required class is very small.
     32 * Write a regression test when you test the ''functionality of an Action''.
     33 * Write a unit test when the ''number of required classes''/functions/components is very ''small''.
    3434
    35 Unit tests are little independent programs that are linked against the classes you want to test. Hence, if lots of other classes are required, you will have lots of dependencies and this will slow down the compilation process in case of many unit tests enormously.
     35Unit tests are little independent programs that are linked against the classes you want to test. Hence, if lots of other classes are required, you will have lots of dependencies and this will slow down the compilation process enormously in case of many unit tests.
    3636
    3737Regression tests are basically scripts done via [http://www.gnu.org/software/hello/manual/autoconf/Using-Autotest.html GNU autotest]. They call the MoleCuilder executable specifically with the desired actions and check e.g. the output files to see whether everything was done as intended.
    3838
    39 Regression test are actually designed as a tool in the refactoring of code bu they also serve as a global test on the software as in most cases lots of dependencies are required: e.g. removing an atom needs the World, atom with all inherited, molecule !ActionHistory, !ActionRegistry, ...
     39Regression test are actually designed as a tool in the refactoring of code but they also serve as a global test on the software as in most cases lots of dependencies are required: e.g. removing an atom needs the World, atom with all inherited, molecule !ActionHistory, !ActionRegistry, ... That is also the reason why these kind of tests are sort of orthogonal to unit tests.
    4040
    4141=== How? ===
     
    4545==== Regression test ====
    4646
    47 Regression tests can be found in [source:tests/regression].
     47Regression tests can be found in [source:tests/regression]. The basic working is: have a input file, do something with it and compare to output file against a stored one.
    4848
    49 '''Note:''' There is specific test set on ''tesselations'' in [source:tests/Tesselations] where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now.
     49'''Note:''' There is specific test set on ''tesselations'' in [source:tests/Tesselations] where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now. We won't go into the details of these tests here.
    5050
    51 Basically, the regression test is driven via a '''testsuite''' script which is created by autotools. In [source:tests/regression/Makefile.am] you will notice it as the only element of the ''TESTSUITE'' variable given. Note however that some ''EXTRADIRS'' are specified.
     51Basically, the regression test is driven via a '''testsuite''' script which is created by autotools. In [source:tests/regression/Makefile.am] you will notice it as the only element of the ''TESTSUITE'' variable given. Note however that some ''EXTRADIRS'' are specified. Hence, if you ever create a new directory for tests, add it hereto.
    5252
    5353In [source:tests/regression] you notice a lot of '''.at''' suffixed files. These are the '''a'''uto'''t'''est scripts per test category. So far, we have:
     
    6363 * Tesselation - Actions for creating the tesselated surface of a molecule
    6464
    65 On how to write a testsuite test, we refer you to these files to get a notion and [http://www.gnu.org/software/hello/manual/autoconf/Writing-Testsuites.html#Writing-Testsuites here] to have a reference of the possible autotest commands at hand. The scheme is always the same basically:
     65Input and output files are stored in the subfolders of [source:tests/regression]. Each specific '''testsuite-....at''' has its own folder called alike. Have a look at '''testsuite.at''' wherefrom '''testsuite''' is constructed bu autotools. It only contains ''m4_include''s to all the test parts.
     66
     67These folders contain again folders with cardinal numbers, in each a '''pre''' and a '''post''' folder. '''pre''' contains input and '''post''' contains output files to test against. The numbers are arbitrary and are unique to the specific test.
     68
     69On how to write a testsuite test, we refer you to one of these '''.at''' files to get a notion and [http://www.gnu.org/software/hello/manual/autoconf/Writing-Testsuites.html#Writing-Testsuites here] to have a reference of the possible autotest commands at hand. The scheme is always the same basically:
    6670{{{
    6771AT_BANNER([Global theme of the test suite section])
     
    7478AT_CLEANUP #remove all temporary files
    7579}}}
    76 where ''<return value>'' is some number the code returns to indicate everything worked fine.
     80where ''<return value>'' is some number the code returns to indicate everything worked fine. The global theme is specified only once per '''.at''' file, file ''AT_SETUP'' and ''AT_CLEANUP'' sort of embrace every specific test that you want to do.
     81
     82Note that testing the undo/redo-functionality of an Action is always placed into two extra tests, same small theme only with '''with Undo/Redo''' added.
     83
     84So, what to do step by step:
     85 1. Pick the '''testsuite-....at''' file that you think your Action fits best.
     86 1. Create your own folder within one of the subfolder list above. E.g. if 1 to 5 are used, take 6. Therein, create folders '''pre''' and '''post'''
     87 1. In '''pre''' place all the inputs files your tests needs. In '''post''' place all the outputs files as you expect your Actions to produce (note: if you create them via a different manner than by just calling your Action and before you actually create the Action, even better so. If not, ''each'' of  the stored output files should have been tested by some other means (visually at the very least).)
     88 1. Add an setup/cleanup section to the chosen '''.at''' file, place ''AT_CHECK''s inside with calls to molecuilder and your action and some fgrep for specific messages of the molecuilder run and/or diff comparing the output files.
     89
     90Done. Test via '''make check''' which will also re-create the '''testsuite''' script.
    7791
    7892==== Unit test ====
    7993
     94Unit tests are written using the [http://sourceforge.net/apps/mediawiki/cppunit/index.php?title=Main_Page CppUnit] testing framework.
     95
     96Have a look at subfolders called ''unittests'' in [source:src]. These folders containing the unit test soure code are located in the folders with the associated ''component'' or unit to test. Hence, they are also refered to as component tests and this is also their main intention: ''Testing small components of the big code for its specific functionality.''
     97
     98Let us take the '''!SingletonUnitTest''' as an example, located in [source:src/Patterns/unittests] that tests correct implementation of the Singleton pattern in [source:src/Patterns/Singleton_impl.hpp]. The unit test consists of the following files:
     99 * source file with the test implementation, e.g. [source:src/Patterns/unittests/SingletonTest.cpp]
     100 * header file with the test definition, i.e. the class containing the test and all test functions to call, , e.g. [source:src/Patterns/unittests/SingletonTest.hpp]
     101 * [source:src/unittests/UnitTestMain.cpp] with the main function calling the unit test runner.
     102 * additional lines in '''Makefile.am''' to mark the small program as a test and to define the program to consist of your source, header and the !UnitTestMain.cpp file.
     103
     104Writing a new unit test then consists of writing the source and the header file, where you should guide yourself by the already present files. ''Naming convention'' is usually to suffix the name with '''!UnitTest''', e.g. '''!SingletonUnitTest.cpp'''.
     105
     106The test class should be named in a similar manner and has the following look:
     107{{{
     108class SingletonTest : public CppUnit::TestFixture
     109{
     110  CPPUNIT_TEST_SUITE( SingletonTest );
     111  CPPUNIT_TEST ( blaTest );
     112  CPPUNIT_TEST ( blablaTest );
     113  CPPUNIT_TEST_SUITE_END();
     114
     115public:
     116  void setUp();
     117  void tearDown();
     118
     119  void blaTest();
     120  void blablaTest();
     121};
     122}}}
     123
     124It inherits CppUnit::!TestFixture and defines it as a TEST_SUITE via the initial macros which also give the test functions to call. Below the test functions are defined along with ''setUp()'' and ''tearDown()'' which embrace the call of each test function, i.e. which create a common test environment for each test function, by allocating some memory, setting some stuff and cleaning up in the end again.
     125
     126See the [http://cppunit.sourceforge.net/doc/lastest/cppunit_cookbook.html CppUnit Cookbook] for a guide on how to write these tests.
    80127
    81128=== How to use/call these tests? ===
     
    107154src/unittests/FormulaUnittest
    108155}}}
    109 
     156The test will state '''(OK)''' when without error. Otherwise it will state '''FAIL''' and the code lines of all tests that failed
    110157
    111158== If something is broken ==