README.md

Collection of HEP workloads for benchmarking purposes.

Workloads are packaged in self contained Docker images, including the needed libraries from CVMFS.

Snapshotting of CVMFS is obtained using the [CVMFS Shrinkwrap utility](https://cvmfs.readthedocs.io/en/stable/cpt-shrinkwrap.html)

## Summary of currently supported HEP workloads

| Experiment |  Name  | Description | Experiment license| Latest Container | Readiness | Pipeline status | 
| -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Alice    | [gen-sim](alice/gen-sim/)  | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/alice/gen-sim/alice-gen-sim/DESCRIPTION) | [GNU GPL v3](github.com/AliceO2Group/AliceO2/blob/dev/COPYING) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/alice-gen-sim-bmk:latest) | w.i.p. | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-alice-gen-sim/pipeline.svg) | 
| Atlas    | [gen](atlas/gen)      | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/atlas/gen/atlas-gen/DESCRIPTION) | [Apache v2](https://gitlab.cern.ch/atlas/athena/blob/master/LICENSE) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/atlas-gen-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-atlas-gen/pipeline.svg) |
| Atlas    | [sim](atlas/sim)      | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/atlas/sim/atlas-sim/DESCRIPTION) | [Apache v2](https://gitlab.cern.ch/atlas/athena/blob/master/LICENSE) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/atlas-sim-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-atlas-sim/pipeline.svg) |
| Atlas    | [digi-reco](atlas/digi-reco) | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/atlas/digi-reco/atlas-digi-reco/DESCRIPTION)  | [Apache v2](https://gitlab.cern.ch/atlas/athena/blob/master/LICENSE) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/atlas-digi-reco-bmk:latest) | w.i.p. | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-atlas-digi-reco/pipeline.svg) |
| CMS      | [gen-sim](cms/gen-sim)  | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/cms/gen-sim/cms-gen-sim/DESCRIPTION)  | [Apache v2](https://github.com/cms-sw/cmssw/blob/master/LICENSE) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-cms-gen-sim/pipeline.svg) |
| CMS      | [digi](cms/digi)     | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/cms/digi/cms-digi/DESCRIPTION)   | [Apache v2](https://github.com/cms-sw/cmssw/blob/master/LICENSE) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-digi-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-cms-digi/pipeline.svg) |
| CMS      | [reco](lhcb/gen-sim)     | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/cms/reco/cms-reco/DESCRIPTION)  | [Apache v2](https://github.com/cms-sw/cmssw/blob/master/LICENSE) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-reco-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-cms-reco/pipeline.svg) |
| LHCb     | [gen-sim](lhcb/gen-sim)  | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/lhcb/gen-sim/lhcb-gen-sim/DESCRIPTION)  | [GNU GPL v3](indico.cern.ch/event/727095/contributions/2992611/attachments/1646177/2631053/20180509-HSFLicensing-LHCbStatus.pdf) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/lhcb-gen-sim-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-lhcb-gen-sim/pipeline.svg) |
| Belle2     | [gen-sim-reco](belle2/gen-sim-reco)  | [link](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/-/blob/master/belle2/gen-sim-reco/belle2-gen-sim-reco/DESCRIPTION)  | [GNU GPL v3](indico.cern.ch/event/727095/contributions/2992611/attachments/1646177/2631053/20180509-HSFLicensing-belle2Status.pdf) | [docker](docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/belle2-gen-sim-reco-bmk:latest) | Y | ![pipeline status](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/badges/qa-build-belle2-gen-sim-reco/pipeline.svg) |


## HEP application in a standalone container
A standalone container running an HEP workload consists of
   1. A cvmfs area exposing the software libraries
   1. A set of input data (root files and condition data)
   1. An orchestrator script

### The orchestrator script
The orchestrator script (example here for [KV](atlas/kv/atlas-kv/atlas-kv-bmk.sh) ) takes care of configuring the application
environment, run the application, parse the produced output and create the score results.
The script has few utility functions to start a configurable number of parallel (and independent) copies of the same application, so that all the cores of the machine under test receive a workload. Typically the number of copies depends on the number of available cores and the number of threads each copy will spawn.

The orchestrator script resides in a directory (example here for [KV](atlas/kv/atlas-kv/) ) that expects all other components (namely /cvmfs and input files) are already available. How to make those components available depends on the approaches followed.

### Run the HEP application in a Docker container
One of the approaches to run the application is to build a Docker a standalone container starting from the SLC6 base image, 
include the directory of the orchestration script, the input files and /cvmfs.
This is what achieved by using this [Dockerfile](common/Dockerfile.template), where the local cvmfs area (empty in this repo) can be populated with the snapshot fo CVMFS obtained by using the [CVMFS Shrinkwrap utility](https://cvmfs.readthedocs.io/en/stable/cpt-shrinkwrap.html).

Alternatively cvmfs can be made available on the host, and bind mounted to the docker container using -v /cvmfs:/cvmfs:shared

### Snapshot the CVMFS repository using the CVMFS Shrinkwrap utility
This utilities assumes that 
   * a recent version of CVMFS (still pre-release) is adopted
   * an appropriate configuration of the shrinkwrap utility is present
   * that the HEP application runs once, in order to open the cvmfs file that will be then extracted to build the snapshot.

When this is done, after running the shrinkwrap utility, a local archive of the cvmfs snapshot is build.
The process goes through the creation of a trace file, that traces what has been accesses, followed by the copy of those files to the local archive.

In order to automate this procedure a bash script is available in this repo [main.sh](/build-executor/main.sh)
The script takes care of
   * Installing the appropriate version of cvmfs (if needed)
   * Configuring the cvmfs application for the tracer
   * Trigger the execution of the HEP application
   * Create the cvmfs local archive
   * Create the standalone Docker container with the HEP application and the local copy of the cvmfs archive

The gitlab CI of this project is configured to run all those steps (see [.gitlabci.yml](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/blob/master/.gitlab-ci.yml))
In order to execute all those steps in an isolate environment, Docker containers are used.

#### Example: Run interactively the CVMFS Shrinkwrap procedure to build a standalone LHCb container

 1. Install docker ([doc from official site](https://docs.docker.com/install/linux/docker-ce/centos/))
    * `yum remove docker         docker-common         docker-selinux         docker-engine`   
        `yum install -y yum-utils device-mapper-persistent-data lvm2`  
        `yum-config-manager  --add-repo   https://download.docker.com/linux/centos/docker-ce.repo`  
        `yum install -y docker-ce`  
        `systemctl  start docker`   
        `systemctl enable docker`   
 
 2. Clone the repository  
    * `git clone https://:@gitlab.cern.ch:8443/hep-benchmarks/hep-workloads.git`
    * ``HEPWL=`readlink -f hep-workloads` ``

 3. Run the automatic procedure   
    * ``$HEPWL/build-executor/run_build.sh -s $HEPWL/lhcb/gen-sim/lhcb-bmk.spec``  
    * run ``$HEPWL/build-executor/run_build.sh -h`` to know all options

4. In case only cvmfs needs to be exposed, run ``run_build.sh`` with the option ``-e``
    * ``$HEPWL/build-executor/run_build.sh -s $HEPWL/cms/gen-sim/cms-gen-sim.spec -e sleep``
       * This will start a privileged container that will mount the cvmfs repositories listed in `` $HEPWL/cms/gen-sim/cms-gen-sim.spec`` and will stay alive so that a second container can bind mount the exposed cvmfs mount point
          * ``docker run -v /tmp/root/cvmfs_hep/MOUNTPOINT:/cvmfs:shared ...``
     
# How to run a built HEP workload container

Given a HEP workload docker image ($IMAGE) from the [registry](https://gitlab.cern.ch/hep-benchmarks/hep-workloads/container_registry), get the benchamark to run through
  * ``docker run --rm -v /tmp/results:/results $IMAGE``
  * In order to access the command line options
     * ``docker run --rm -v /tmp/results:/results $IMAGE -h``

The HEP workload docker images are also compatible with Singularity.  It's possible to run the containers with Singularity by prepending the image with a ``docker://`` URI:
  * ``singularity run -C -B /tmp:/tmp -B /tmp/results:/results docker://$IMAGE``
      * The ``/tmp/results`` directory must exist.
Note that the ``overlay`` or ``underlay`` options must be enabled in your singularity.conf file.  On systems with older Linux kernels that do not support OverlayFS, such as in Scientific Linux 6, underlay must be used.  Also note that full containment ('-C') is recommended when invoking Singularity, in order to avoid potential conflicts with your environment settings and the configuraton files in your home directory, which would otherwise be inherited by the container.

## Example of a fast benchmark: Run Athena KV
  * Run Athena v17.8.0.9 (legacy version of KV)
      * docker run --rm -v /some_path:/results gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/atlas-kv-bmk:ci1.2
      * singularity run -C -B /tmp:/tmp -B /some_path:/results docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest
      * command line options: -h (for help), -n xx (number of copies, default is the number of available cores), -d (debug verbosity)
      * NB: any combination of the docker run options described above for CMS works also here

## Example: Run CMS ttbar GEN-SIM

Any combination of the following options
    
   * command line options: -h (for help), -t xx (number of threads, default: 4), -e xx (number of events per thread, default: 100 ), -d (debug verbosity)
      * The script will spawn a number of parallel processes equal to (number of available cores) / (number of defined threads per proces) 
   
   * In order to retrieve json benchmark results and logs from a local directory, mount a host directory (/some_path) as container volume /results 
      * docker run --rm -v /some_path:/results gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest
      * singularity run -C -B /tmp:/tmp -B /some_path:/results docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest
 
   * In order to fix number of events per thread (default is 20) and number of threads per cmssw (default is 4)
      * docker run --rm -v /some_path:/results gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest -t 10 -e 50
      * singularity run -C -B /tmp:/tmp -B /some_path:/results docker://gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest -t 10 -e 50

   * In order to fix number of running cores 
      * docker run --rm --cpuset-cpus=0-7  gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest
      * This is not directly possible with Singularity

   * In order to inspect information about the docker container, including the label with the description of the HEP workload included
      * docker inspect gitlab-registry.cern.ch/hep-benchmarks/hep-workloads/cms-gen-sim-bmk:latest
      * This is not directly possible with Singularity