ATLAS Computing | |
NICOS manual for ATLAS use case | |
US ATLAS Computing | |
This document describes how to use the NICOS (NIghtly COntrol MANagement) tool for ATLAS nightly builds. The similar information and additional concrete recipes are posted on NICOS wiki page.
NICOS (NIghtly COntrol MANagement) facilitates nightly builds of software projects. Nightly builds are common in software project involving multiple developers. NICOS is the system originally developed for ATLAS nightly builds. It was evolved in the independent software package that can be applied to more than one experiment or application.
NICOS performs nightly build processes in distinctive steps. Each step allows plug-ins and can be configured for particular use case. The compilation itself is one of initial steps followed by tests, analysis of errors, and creating web pages with build summaries. Developers are notified about problems with their packages via e-mail.
NICOS supports the organization when the releases are built on a local disk (usually fast and reliable) and then copied to a distributed file system (e.g. AFS). The builds can be incremental, i.e. when only new versions of packages are compiled, or from scratch. While NICOS is originally developed as the nightly build control system it can also build stable and local "personal" releases (in latter case the release contents is determined by a developer).
NICOS allows to build the chain of nightly projects. In this case the projects are built in the certain order determined by declared projects dependencies.
NICOS core does not contain ATLAS specific features. For convenience of ATLAS software infrastructure management the scripts with self-evident (and long) names for typical tasks are added to NICOS distribution. These scripts require very few parameters and allow to run NICOS without reading manuals.
The most recent version of NICOS is installed in
/afs/cern.ch/atlas/software/dist/nightlies/nicos
.
ATLAS nightlies use this installation.
ATLAS software releases (and ATLAS CVS repository)
contain NICOS in AtlasTest/NightlyTestTools/Nicos
.
ATLAS plug-ins and extensions are collected in atlas
and the scripts for typical atlas tasks in
atlasjobs
subdirectories.
NICOS executable is nicos_job
(or similar name), a Bourne
shell script.
Before running NICOS the following three variables must be defined:
NICOS_HOME
is the location of NICOS
installation (at CERN
/afs/cern.ch/atlas/software/dist/nightlies/nicos
)
NICOS_WORK_AREA
is an area for
temporary NICOS files. It can be any empty directory
(if the directory does not exist then NICOS tries to
create it)
NICOS_CONFIG_AREA
is an area where
configuration files are located. For ATLAS nightly builds the
configuration areas are located in:
/afs/cern.ch/atlas/software/dist/nightlies/nicos_config_area<suffix>
NICOS uses three configuration files (see descriptions and examples below).
nicos_cache
(fixed name)
must be located in ${NICOS_CONFIG_AREA}
.
It contains all definitions needed for nightly
builds
nicos_cache
. In case of ATLAS nightly
builds NICOS runs a plug-in that retrieves the tags
from ATLAS Tag Collector. However the tags
in NICOS text file override the tags from Tag Collector
nicos_cache
. In case of ATLAS nightly
builds NICOS uses ATN tool that automatically finds in the release tree and
runs integration tests. It reads the NICOS test database for information on
additional or canceled tests
Once the above-mentioned variables are defined and nicos_cache
is configured the nicos job can be started with
${NICOS_HOME}/nicos_job [OPTIONS]
The options are
This file in XML-like format contains all necessary configuration parameters. Each build step is associated with Markup tag that is followed by environment definitions and commands for particular step. Markup tag can contain some name or base directory definitions. Below are the examples from ATLAS nightly builds configuration files with commentaries.
NICOS_COMMON_CONFIG_AREA
variable). If a shared configuration is
specified then NICOS
<PROJECT SETTING name=ABCD relname=atlrel lifetime=2 sf=SlcOpt inc=yes> NICOS_PROJECT_HOME=/build/atnight/localbuilds/nightlies/relIn Markup tag several names are defined:
atlrel
).
The suffix with the release number is appended, for instance for
seven day cycle the releases names are atlrel_0
-
atlrel_6
, where the number is a day of week
lifetime
indicates number of releases in a cycle
(1 - 365). For instance, if it is equal to 7 then
seven nightly releases exist ( "0" release is built
on Sunday, "1" is built by Tuesday etc. )
sf
is a suffix added to the names of standard
NICOS areas. This is useful when several nightlies share one
code (e.g. optimized and debug ATLAS builds), so their log files
areas get different suffixes. Stored in variable $NICOS_SUFFIX
inc
indicates that the builds are incremental
(i.e. use the results of previous nightlies)
The tag content example:
NICOS_PROJECT_HOME=/build/atnight/localbuilds/nightlies/rel # Area where nightlies are built NICOS_COPY_HOME=/afs/cern.ch/atlas/software/dist/nightlies/rel # Area in distributed file system to which nightlies are copied NICOS_STAMPS_HOME=/afs/cern.ch/atlas/software/dist/nightlies/stamps_global # Area for NICOS time stamps used for synchronization of dependent projects NICOS_BUILD_FROM_SCRATCH="yes" # All files are wiped out from the release area before the build if "yes", # The sources are updated and release area is not cleaned if "no" NICOS_ATLAS_RELEASE="11.1.0" # This variable is added for convenience (not genuine NICOS variable) NICOS_TITLE_COMMENT="tags for ${NICOS_ATLAS_RELEASE}" # This line is added in the title of NICOS web pages NICOS_HOUR=15 # This variable indicates that the builds started after 15:00 # get release name with the suffix for the next day. For example # if release starts on Monday at 21:00 then its name gets suffix _2 NICOS_WEBPAGE="http://atlas.web.cern.ch/*****" NICOS_COMMON_WEBPAGE="http://atlas.web.cern.ch/*****" NICOS_WEBDIR=/afs/cern.ch/atlas/software/dist/nightlies/nicos_web_area${NICOS_SUFFIX} NICOS_COMMON_WEBDIR=/afs/cern.ch/atlas/software/dist/nightlies/global # These variables store URL's and locations of directories with # NICOS html files. Common web area holds the summary page for # all available nightlies NICOS_DBFILE=${NICOS_CONFIG_AREA}/tags.db # NICOS database file with names of packages, tags, and addresses of # software developers (default shown) NICOS_TEST_DBFILE=${NICOS_CONFIG_AREA}/test.db # NICOS test database file with names of tests, suites, locations, # and addresses of software developers (default shown) NICOS_DOMINANT_PARALLEL="SlcDbg" NICOS_PROJECT_TYPE="parallel" # This indicates that another nightly project with suffix "SlcDbg" # uses the same build area and is responsible for checkout (as # "DOMINANT"). Therefore the "SlcOpt" build will be suspended # until the checkout in "SlcDbg" is not completed. NICOS_COORDINATOR="undrus@bnl.gov" NICOS_RELEASE_COORDINATOR="undrus@bnl.gov" # NICOS send some notifications to these addresses. PATH=/afs/cern.ch/sw/java/i386_redhat73/jdk/sun-1.5.0beta/bin:${PATH} LD_LIBRARY_PATH=/usr/local/gcc-alt-3.2/lib:${LD_LIBRARY_PATH} CVSROOT=:pserver:aundrus@atlas-sw.cern.ch:/atlascvs SITEROOT=/afs/cern.ch CMTBASE=/afs/cern.ch/sw/contrib/CMT CMTVERS=v1r14p20031027 CMTCONFIG=i686-rh73-gcc32-opt CMTROOT=${CMTBASE}/${CMTVERS} CMTSITE=CERN CMTCVSOFFSET=offline #Variables for CMT setup.
command=source
is indicated then the variables
defined at this step are valid for subsequent steps.
<RELEASE TOOL SETUP dir=. command=source> source ${CMTROOT}/mgr/setup.sh CMTPATH=${NICOS_RELHOME}if [ "$NICOS_PROJECT_TYPE" != "parallel" ]; then ${NICOS_HOME}/atlas/atlas_read_tagcollector ${NICOS_ATLAS_RELEASE} rm -rf ${NICOS_RELHOME}/InstallArea fi
file
that is text database file.
cvs
.
pagsh
.
<CHECKOUT tagstorage=file cvsdir=offline command=cmt shell=/usr/bin/pagsh.krb>
command=source
is indicated then the variables
defined at this step are valid for subsequent steps.
<PROJECT SETUP dir=. command=source>if [ "$NICOS_PROJECT_TYPE" != "parallel" ]; then cd $NICOS_RELHOME/AtlasRelease/*/cmt ${NICOS_HOME}/atlas/atlas_generate_req cd ${NICOS_PROJECT_HOME}/${NICOS_PROJECTSETUP_DIR} source ${NICOS_HOME}/atlas/setup_cmt.sh
NICOS_BUILDLOG_SEPARATOR
be defined.
It is required that the release build tool prints
a separating line between builds of different packages.
This line should contain the name of package to be
built (preferably with all containers,
for instance Utilities/FileCatalog
if package FileCatalog is in the container Utilities).
In Markup tag the work directory (relative to the release directory
$NICOS_RELHOME) is defined. Also the desirable shell
can be indicated.
<PROJECT BUILD dir=AtlasRelease/*/cmt shell=/bin/zsh> cmt broadcast - 'rm -f ../i686*/*.so ; rm -rf ../run' cmt broadcast - cmt config source setup.sh cmt broadcast - make NICOS_FULLNAME_INSEPARATOR="yes" NICOS_BUILDLOG_SEPARATOR="Now trying"
rsync
. In the Markup tag the
work directory (relative to the release directory $NICOS_RELHOME),
desirable shell and copy type is defined. The possible
copy types are:
<COPY AFTER BUILD dir=AtlasRelease/*/cmt shell=/bin/sh type=scratch> #the additional commands can be added in the content, for example, #for fixing soft links in the copied release (as rsync does not #modify soft links automatically).
<QA TESTS dir=AtlasRelease/*/cmt> cp -a ${NICOS_RELHOME}/Tools/Scripts/Scripts-*/share/checkreq.sh ~/. cmt broadcast - ~/checkreq.sh NICOS_FULLNAME_INSEPARATOR="yes" NICOS_BUILDLOG_SEPARATOR="Now trying"
<UNIT TESTS dir=.>
<INTEGRATED TESTS dir=qmtest tool=qmtest> > ATN_HOME=/afs/cern.ch/atlas/software/dist/nightlies/atn_11 NICOS_INTTESTS_FILES=".sh$" NICOS_INTTESTS_LOCATION="AtlasTest/TestTools/*/scripts/nightlies" NIGHTLYAREA=${NICOS_RELHOME} cd $NICOS_RELHOME/AtlasRelease/*/cmt source setup.shIn Markup tag directory is defined from which this step is performed. If tool is
qmtest
then the Python based QMTest tool
is used for running tests.
rsync
. In the Markup tag the
work directory (relative to the release directory $NICOS_RELHOME),
desirable shell and copy type is defined. If the copy after
build is performed then the recommended type is "update"
.
<COPY AFTER TESTS dir=AtlasRelease/*/cmt shell=/bin/sh type=update> #the additional commands can be added in the content, for example, #for fixing soft links in the copied release (as rsync does not #modify soft links automatically).
NICOS_TEST_SUCCESS_PATTERN
: this pattern
must be present in output of successful integration test.
NICOS_TEST_FAILURE_PATTERN
: this pattern
must be absent in output of successful integration test.
NICOS_TEST_WARNING_PATTERN
: this pattern
triggers warning level of alarm
NICOS_MAIL
: if "yes" then automatic e-mail
notifications about problems are sent (to addresses from
NICOS database file).
<ERROR ANALYSIS> NICOS_TEST_SUCCESS_PATTERN=" " NICOS_TEST_FAILURE_PATTERN="FAILURE " NICOS_TEST_WARNING_PATTERN="test warning" NICOS_QA_SUCCESS_PATTERN=" " NICOS_QA_FAILURE_PATTERN="FAILURE " NICOS_QA_WARNING_PATTERN="checkreq warning" NICOS_MAIL="no" NICOS_MAIL_UNIT_TESTS="no" NICOS_MAIL_QA_TESTS="no"
<WEB PAGE>
NICOS database file is a text file. Each line contains name of package, tag of package, and e-mail addresses of developers (could be multiple) separated by unlimited number of spaces. For example:
FileCatalog FileCatalog-0-0-1 developer1@cern.ch developer2@bnl.gov Utilities/FileCatalog Utilites-0-0-1 developer1@cern.chNote that the second line describes a package inside container, so checkout is performed for
Utilities/FileCatalog
.
There are several options for tags:
FileCatalog
is specified as a tag
then the latest tag with name starting with FileCatalog
will be selected (e.g. FileCatalog-89-99-99
).
recent
: the recent CVS submission is selected
(not necessarily tagged).
development
: the latest tagged CVS version is
selected, the format of tag name is not restricted.
official
: the latest tag in the format
<package name>-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]
or
<package name>-[0-9]-[0-9]-[0-9]
is selected.