| 1 | /*
|
|---|
| 2 | * Project: MoleCuilder
|
|---|
| 3 | * Description: creates and alters molecular systems
|
|---|
| 4 | * Copyright (C) 2010 University of Bonn. All rights reserved.
|
|---|
| 5 | * Please see the LICENSE file or "Copyright notice" in builder.cpp for details.
|
|---|
| 6 | */
|
|---|
| 7 |
|
|---|
| 8 | /**
|
|---|
| 9 | * \file regression-tests.dox
|
|---|
| 10 | *
|
|---|
| 11 | * This file contains explanation how to write, launch and use regression tests.
|
|---|
| 12 | *
|
|---|
| 13 | * Created on: Oct 11, 2011
|
|---|
| 14 | * Author: heber
|
|---|
| 15 | */
|
|---|
| 16 |
|
|---|
| 17 |
|
|---|
| 18 | /**
|
|---|
| 19 | * \page regressiontest "Regression tests"
|
|---|
| 20 | *
|
|---|
| 21 | * Regression test with this project are used to check on the functionality of
|
|---|
| 22 | * all Actions. They launch MoleCuilder from the command line with a given
|
|---|
| 23 | * action and option and basically diff the output against a stored file with
|
|---|
| 24 | * the desired result. This should be the behavior of more than 90% of all
|
|---|
| 25 | * regression tests.
|
|---|
| 26 | * For some we additionally catch the output from stdout and stderr and diff against
|
|---|
| 27 | * it, but this is actually not a recommended way of testing as output depends very
|
|---|
| 28 | * much on the logging level.
|
|---|
| 29 | *
|
|---|
| 30 | * \section regressiontest-structure Directory structure
|
|---|
| 31 | *
|
|---|
| 32 | * They are contained in the source folder \b tests/regression. There are
|
|---|
| 33 | * categories containing a distinct folder for each action. The folder for the
|
|---|
| 34 | * test of a specific Action has the following structure:
|
|---|
| 35 | * \li a folder \b pre contains all files required as input to the test.
|
|---|
| 36 | * \li a folder \b post contains all output files against which we compare all
|
|---|
| 37 | * or a number of resulting files.
|
|---|
| 38 | * \li a test script \b testsuite-....at which is included in a similarly-
|
|---|
| 39 | * named test script one directory level higher.
|
|---|
| 40 | *
|
|---|
| 41 | * \section regressiontest-launch-all Launching all tests
|
|---|
| 42 | *
|
|---|
| 43 | * Launching all regression tests is as simple as:
|
|---|
| 44 | * -# Enter the build directory
|
|---|
| 45 | * -# There, enter \b tests/regression
|
|---|
| 46 | * -# Run
|
|---|
| 47 | * \code make check \endcode
|
|---|
| 48 | * (at you liberty with option \a -j8 or similar for running the tests in
|
|---|
| 49 | * parallel.
|
|---|
| 50 | *
|
|---|
| 51 | * \section regressiontest-launch-some Launching a specific tests
|
|---|
| 52 | *
|
|---|
| 53 | * Launching a single or just some of the tests is only a little bit more
|
|---|
| 54 | * complicated. There are two options: either by the test number which however
|
|---|
| 55 | * changes when new tests are added, and by keywords.
|
|---|
| 56 | *
|
|---|
| 57 | * Then proceed as follows:
|
|---|
| 58 | * -# Enter the build directory
|
|---|
| 59 | * -# There, enter \b tests/regression
|
|---|
| 60 | * -# Run
|
|---|
| 61 | * \code ../../../tests/regression/testsuite <option> AUTOTEST_PATH="<buildpath>/src" \endcode,
|
|---|
| 62 | * where \a <option> is explained in the subsections below and \a <buildpath>
|
|---|
| 63 | * is the build path (i.e. the variable \a AUTOTEST_PATH should contain the
|
|---|
| 64 | * path to the executable).
|
|---|
| 65 | *
|
|---|
| 66 | * \subsection regressiontest-launch-by-number ... by number
|
|---|
| 67 | *
|
|---|
| 68 | * Tests can be launched by specifying their test number, e.g. then \a <option>
|
|---|
| 69 | * might be \a 283 or \a 283-284 or \a 283,285-286 or alike
|
|---|
| 70 | *
|
|---|
| 71 | * \subsection regressiontest-launch-by-keyword .. by keyword
|
|---|
| 72 | *
|
|---|
| 73 | * Tests may as well be launched by some keywords, e.g. each Action has a
|
|---|
| 74 | * specific token which is also one of its keyword (see above for the
|
|---|
| 75 | * policy). I.e. we may launch the test on fill-void via the \a <option>
|
|---|
| 76 | * \a -k \a fill-void. Also multiple keywords may be given.
|
|---|
| 77 | *
|
|---|
| 78 | * \section regressiontest-results Inspecting test results
|
|---|
| 79 | *
|
|---|
| 80 | * The testsuite can be launched with the additional option of \a -d which
|
|---|
| 81 | * leaves the directory of the test present even though the test has passed
|
|---|
| 82 | * for inspection.
|
|---|
| 83 | *
|
|---|
| 84 | * If a test fails, in whatever way it was launched, will leave in the build
|
|---|
| 85 | * directory a folder \b tests/regression/testsuite.dir/<nr> where \a <nr> is
|
|---|
| 86 | * the number of the test (padded maybe with some zeros).
|
|---|
| 87 | *
|
|---|
| 88 | * \subsection regressiontest-results-failures Typical failure causes
|
|---|
| 89 | *
|
|---|
| 90 | * Here, we note down some typical failure causes:
|
|---|
| 91 | * -# A test fails on installcheck on \code make distcheck \endcode: In the test
|
|---|
| 92 | * a file is probably copied and then some work is done on it. Due to autoconf's
|
|---|
| 93 | * distcheck policy, files copied from the archive are write-protected, hence
|
|---|
| 94 | * cannot be modified. And so is the copy. The remedy is to add the line
|
|---|
| 95 | * \code
|
|---|
| 96 | * AT_CHECK([chmod u+w $file], 0)
|
|---|
| 97 | * \endcode
|
|---|
| 98 | * just after the copying to modify the write permission of the copied file \a $file
|
|---|
| 99 | * and allow its modification.
|
|---|
| 100 | *
|
|---|
| 101 | * \section regressiontest-add Adding a new test
|
|---|
| 102 | *
|
|---|
| 103 | * The basic working is: have a input file, do something with it and compare
|
|---|
| 104 | * to output file against a stored one.
|
|---|
| 105 | *
|
|---|
| 106 | * \attention Name convention of files and directories, e.g.
|
|---|
| 107 | * \b tests/regression/Parser/Pdb with \b testsuite-parser-pdb-save.at
|
|---|
| 108 | * - the test directory should either be called as the token of the Action it
|
|---|
| 109 | * tests or a unique and brief description but with no spaces, no dashes,
|
|---|
| 110 | * but CamelCase (i.e. Capitalize each new word)
|
|---|
| 111 | * - the test script file should be called as follows:
|
|---|
| 112 | * -# testsuite-...
|
|---|
| 113 | * -# ...each directory (non-capital letters) with a dash...
|
|---|
| 114 | * -# ..the name of the test directory...
|
|---|
| 115 | * -# a further description of there are multiple test scripts in the
|
|---|
| 116 | * test directory.
|
|---|
| 117 | *
|
|---|
| 118 | * Adding a new regression tests consists of the following items:
|
|---|
| 119 | * -# Create a new folder in a matching category
|
|---|
| 120 | * -# Create therein subfolders \b pre and \b post
|
|---|
| 121 | * -# Create a test script where the name begins with \b testsuite- and contains
|
|---|
| 122 | * category and the action token. See other test scripts to get an idea on how
|
|---|
| 123 | * to write these and also look here (http://www.gnu.org/s/hello/manual/autoconf/Using-Autotest.html#Using-Autotest).
|
|---|
| 124 | * -# In the command \a AT_KEYWORD which must be present you should given the
|
|---|
| 125 | * token of the Action as it would be called from the command line (see below for
|
|---|
| 126 | * the reason).
|
|---|
| 127 | * -# Include your testscript in the one present one directory-level up.
|
|---|
| 128 | * -# Also include your script in \b tests/scripts/Makefile.am such that the
|
|---|
| 129 | * testsuite is automatically re-compiled when one of the test files has changed.
|
|---|
| 130 | *
|
|---|
| 131 | * On how to write a testsuite test, we refer you to one of these .at files to get a
|
|---|
| 132 | * notion and here to have a reference of the possible autotest commands at hand.
|
|---|
| 133 | * The scheme is always the same basically:
|
|---|
| 134 | * \code
|
|---|
| 135 | * AT_SETUP([General test part - description of this testsuite section])
|
|---|
| 136 | * AT_KEYWORDS([<some keywords>,<actionname>,[undo/redo]])
|
|---|
| 137 | * ...
|
|---|
| 138 | * AT_CHECK([this], <return value>, [ignore], [ignore])
|
|---|
| 139 | * AT_CHECK([that], <return value>, [stdout], [stderr])
|
|---|
| 140 | * AT_CHECk([fgrep "test fine" stdout], 0, [ignore], ignore])
|
|---|
| 141 | * ...
|
|---|
| 142 | * AT_CLEANUP #remove all temporary files
|
|---|
| 143 | * \endcode
|
|---|
| 144 | * where <return value> is some number the code returns to indicate everything
|
|---|
| 145 | * worked fine. The global theme is specified only once per .at file, file
|
|---|
| 146 | * AT_SETUP and AT_CLEANUP sort of embrace every specific test that you want to do.
|
|---|
| 147 | * Note that it is required to list the action name under AT_KEYWORDS and also
|
|---|
| 148 | * give undo oder redo as an additional keyword if the undo or redo of the action
|
|---|
| 149 | * is tested. It is advised to give further keywords, e.g. the directory name
|
|---|
| 150 | * giving the general theme of the tests (selection, analysis, ...). Also note
|
|---|
| 151 | * that all keywords are always lower-case!
|
|---|
| 152 | *
|
|---|
| 153 | * Note that testing the undo/redo-functionality of an Action is always placed
|
|---|
| 154 | * into the same test file along with the normal functionality but in different
|
|---|
| 155 | * tests (i.e. undo and redo each have their own AT_SETUP .. AT_CLEANUP wrapping).
|
|---|
| 156 | *
|
|---|
| 157 | *
|
|---|
| 158 | * \date 2011-10-31
|
|---|
| 159 | *
|
|---|
| 160 | */
|
|---|
