CTests · Wiki · sft / lcgcmake

CTest for a generator is defined in lcgcmake/generators/CMakeLists.txt beside the definition of the generator itself.

If an executable is to be compiled for the , it is done by lcgcmake/generators/<generator>/tests/CMakeLists.txt. The test code (if it is not taken from the installed generator's share/sources) is also put in this directory.

The CMakeLists.txt for the test compilation uses the CMake system FindPackage. Some macros to find well known packages are defined in the standard CMake (see (http://www.cmake.org/cmake/help/v2.8.9/cmake.html)[CMake documentation]). If the tested package depends on some other package of GENSER or external, additional macros FindPackage are used, e.g. FindHepMC, FindThePEG. They can be defined in 2 places:

package cmaketools defined in lcgcmake and installed in /external
lcgcmake/cmake/modules. In this case this additional search path is added inside lcgcmake/generators/<generator>/tests/CMakeLists.txt

Usually new Find* macros are put in the second location, and after full testing added to cmaketools (kept in git).

When everything is prepared, a single ctest can be run in lcgcmake-build by

ctest -R <generator>

It is possible to increase verbosity of the test like this:

ctest -VV -R <generator>

CMakeLists for the test compilation can be debugged using messages: message(STATUS "my message text ${LIBLIST}")

It is recommended to avoid scripts in the ctests system

Once the test is defined in lcgcmake/generators/CMakeLists, it will be run in the nightlies.

Tests can have labels: LABELS PhysicsCheck. This is not used now. It is possible to specify maximal time: TIMEOUT 3000 Default time unit is seconds.

CTESTS NAMING CONVENTION:

Each test name contains

generator's name
generator's version
type of test
"native" tests are test that use examples, sources, ... provided by package developers
"genser" tests are test that use source codes made by GENSER team
some tests may have a short name accompanying to its function, mainly these tests check common functionality of Installation or provide specific actions.
definition of the test. It should be related with test's actions and/or parameters.

Formula of test name: generator-version.(native|genser)-definition[-rivet for rivet tests]

Physics rivet tests convention:

Reference histograms are kept in lcgcmake/generators/reference

At the beginning of each reference yoda file there is a line like below, with a keyword "Reference_info":

 # Reference_info : obtained 09.03.2015 with alpgen 2.1.4 and agile 1.4.1 on x86_64-slc6-gcc48-opt, experimental
 # Process Z + 2jets, electrons (steering file inputalpgen), warm 3x100000, 5000000,
 # hadronized and read by rivet 10000 events

Tools available for the histograms comparison:

${CMAKE_SOURCE_DIR}/cmake/scripts/yoda2root.cmake -DAIDA2ROOT=${AIDA2ROOT}, example is
```
${CMAKE_SOURCE_DIR}/generators/reference/aida2root.py
  -DPython_cmd=${Python_cmd}
  -Dyoda_home=${yoda_home}
  -DINPUT=filepath/filename.yoda
```
creates filename.root at filepath. In addition creates filename.aida
${CMAKE_SOURCE_DIR}/generators/reference/reference.py. The main mode of work:
```
reference.py file1.root file2.root "Path=name1" limit=0.05
```
or
```
reference.py file1.root file2.root "Path=name1,name2" "limit=0.05,0.05"
```
Compare histogram name1 in the two root files, Kolmogorov test, conf. limit 5%

If the path is "XS", the script will calculate chi2 and use Tmath::Prob to calculate probability.

Initially there was another mode of work:
```
reference.py file1.yoda file2.yoda "Path=XS" limit=0.05
```
But it is disabled now (will give an error). To use it, the code is to be further developed, adding extraction of scatter plot errors from yoda.

How to make validation analyses and publish results

# Use -DVALIDATION=ON in CMake command (and not use LCG_IGNORE)
  cmake -DVALIDATION=ON ...
# build 'validation' target. All required packages and tests will be built automatically
  make validation                 # build validation tests for all avaiable packages
  make validation.<package>         # -//- for all versions of package
  make validation.<package>-<version> # -//- for selected version of package
# publish results
  make validation.publish         # publish results to genser.cern.ch web interface. This step requires sftnight AFS token (will be asked if it isn't present)