Changes between Initial Version and Version 1 of TestingGuidelines

Timestamp:

Dec 10, 2010, 11:13:46 AM (15 years ago)

Author:

FrederikHeber

Comment:

first version

TestingGuidelines

@@ +1 @@
+
+== Guideline on how and when to write a test ==
+
+=== Why? ===
+
+MoleCuilder is by now a quite complex piece of code. as of this writing, it has around 90k lines of code and about 700 modules. I.e. the code has lots of cross references, lots of functions are need there and then also quite somewhere else. Hence, if you change the inner workings of one function, keeping its signature, it will compile, but most likely these functions somewhere else will not continue working as before.
+
+That's where software testing comes into play!
+
+Via a test you basically define what kind of behavior your function/class will have. You test that for given inputs, required outputs are returned. Such when you change the inner workings, you immediately notice if you changed something crucial as the associated test will (hopefully) fail. ''Hopefully'' because no test can ever be complete, this is NP-hard or something alike to the [http://en.wikipedia.org/wiki/Halting_problem halting problem] of computer science.
+
+You might answer to this: I'll just never change the functions ... ''Oh, yes you will'' ... sometimes you have written a function and then stumble upon another section of code where it would fit beautifully if only it were written a bit more generally. And there are lots of more examples ... especially, when you believe in [http://en.wikipedia.org/wiki/Refactoring refactoring] of code! ... Which we do here ...
+
+
+=== When? ===
+
+'''Always'''! ... when in doubt, write a test. When only half in doubt, do write a test. When trying something new, first write a test that encompasses all the stuff the new code should be able to do. When thinking of a new black box of code, define via a test what the outcomes for given inputs of this black box should be.
+
+Have a look at [http://en.wikipedia.org/wiki/Software_testing software testing] and [http://en.wikipedia.org/wiki/Unit_tests unit tests] for general ideas on software testing techniques.
+
+So, as the when question should now be settled. Let's continue ...
+
+=== What? ===
+
+What kind of test should I write.
+
+There are two test types supported:
+ * ''Regression tests''
+ * ''Unit tests''
+
+And the distinction is quite simple:
+ * Write a regression test when you test the functionality of an Action.
+ * Write a unit test when the number of required class is very small.
+
+Unit tests are little independent programs that are linked against the classes you want to test. Hence, if lots of other classes are required, you will have lots of dependencies and this will slow down the compilation process in case of many unit tests enormously.
+
+Regression tests are basically scripts done via [http://www.gnu.org/software/hello/manual/autoconf/Using-Autotest.html GNU autotest]. They call the MoleCuilder executable specifically with the desired actions and check e.g. the output files to see whether everything was done as intended.
+
+Regression test are actually designed as a tool in the refactoring of code bu they also serve as a global test on the software as in most cases lots of dependencies are required: e.g. removing an atom needs the World, atom with all inherited, molecule !ActionHistory, !ActionRegistry, ...
+
+=== How? ===
+
+Finally, we answer the questions how to write a test ...
+
+==== Regression test ====
+
+Regression tests can be found in [source:tests/regression]. 
+
+'''Note:''' There is specific test set on ''tesselations'' in [source:tests/Tesselations] where many molecules are tesselated for their surface that are done in different manner than the autotest-based tests we describe now.
+
+Basically, the regression test is driven via a '''testsuite''' script which is created by autotools. In [source:tests/regression/Makefile.am] you will notice it as the only element of the ''TESTSUITE'' variable given. Note however that some ''EXTRADIRS'' are specified.
+
+In [source:tests/regression] you notice a lot of '''.at''' suffixed files. These are the '''a'''uto'''t'''est scripts per test category. So far, we have:
+ * Analysis - analysis functons such as pair correlation, ...
+ * Domain - Actions that change the domain by adding empty boundary, ...
+ * Filling - Actions that fill the domain with molecules, such as fill void with molecule, ...
+ * Fragmentation - fragmentation of a bonding graph into subgraphs
+ * Graph - graph routines that recognize the bonding graph
+ * Molecules - Actions that change molecules names, ...
+ * Simple Configuration - tests for loading and saving files, adding, removing atoms
+ * Specifics - very specific stuff as specifying the basis set stored in MPQC-type files
+ * Standard Options - tests for options such as help, verbosity, version, ...
+ * Tesselation - Actions for creating the tesselated surface of a molecule
+
+On how to write a testsuite test, we refer you to these files to get a notion and [http://www.gnu.org/software/hello/manual/autoconf/Writing-Testsuites.html#Writing-Testsuites here] to have a reference of the possible autotest commands at hand. The scheme is always the same basically:
+{{{
+AT_BANNER([Global theme of the test suite section])
+AT_SETUP([small theme of your test])
+...
+AT_CHECK([this], <return value>, [ignore], [ignore])
+AT_CHECK([that], <return value>, [stdout], [stderr])
+AT_CHECk([fgrep "test fine" stdout], 0, [ignore], ignore])
+...
+AT_CLEANUP #remove all temporary files
+}}}
+where ''<return value>'' is some number the code returns to indicate everything worked fine.
+
+==== Unit test ====
+
+
+=== How to use/call these tests? ===
+
+Eventually, you may want to call these tests, either all at once or individually ...
+
+To call them all, simply type
+{{{
+make check
+}}}
+which will call both all unit tests and the testsuite.
+
+==== Regression test ====
+
+If you want to call an individual regression test, you can access the '''testsuite''' program in [source:tests/regression/testsuite] directly by giving it the number or range of a test. We sssume your build directory is called '''build''', and you are right now in '''./build/tests/regression/''' and you want to start tests 5-8 and 10, enter
+{{{
+../../../tests/regression/testsuite 5-8 10
+}}}
+and only these tests will get executed. You can also specify a number of tests by its keywords, try
+{{{
+../../../tests/regression/testsuite --help
+}}}
+get obtain more information or see [http://www.gnu.org/software/hello/manual/autoconf/testsuite-Invocation.html#testsuite-Invocation here].
+
+==== Unit test ====
+
+Calling single unit tests is even easier. Assumptions from above, we want to call '''FormulaUnittest''' and reside in '''./build''', enter
+{{{
+src/unittests/FormulaUnittest
+}}}
+
+
+== If something is broken ==
+
+At the very last, we have to talk about what to do with the test if something is broken. If a unit test is broken, you might only modify the code directly. 
+When it is a regression test, you can modify the return value to state that something will not work.
+
+ '''However beware when changing tests! '''
+
+These should never be done lightly and changes to tests HAVE to be given in full in the description of the commit to the repository, also state why the change was necessary. Maybe the test was not complete or faulty itself.
+
+The idea is that even when some tests are not working, '''make check''' should run through smoothly. However, it should indicate that something is amiss here, e.g.
+ * change the message in '''AT_SETUP''' to something that contains ''BROKEN'' or alike
+ * unit tests should