Merge branch 'migration' into 'master'

Preparing migration to CAT

Closes #11

See merge request DasAnalysisSystem/gitlab-profile!10

README.md

-# General instruction
+# Das Analysis System
 
-Darwin is a general project involving modular tools for physics analysis. This repository provides the necessary tools to install all pieces of software.
+Das Analysis System is a general project involving modular tools for physics analysis with high-level objects such as jets, leptons, and photons. The principle is to perform physics directly from the shell in atomic steps, applying one correction at a time and treating all systematic uncertainties simultaneously.
 
-In general, it is recommended that you install the software on a fast disk, but that you process the heavy n-tuples on a dedicated area (e.g. NFS at DESY, EOS at CERN).
+The GitLab group is divided in several repositories:
+- The `Installer` repo contains the necessary scripts to install the suite.
+- The `Core` repo corresponds to a CMSSW module and is where most of the code for the analysis of CMS data is to be found.
+- The `Tables` repo contains the calibration of the high-level objects.
+- The `Darwin` repo is a mirror of a [general toolkit](https://protodarwin.docs.cern.ch) for physics analysis.
+- `PlottingHelper` is a useful [library](https://github.com/zleba/PlottingHelper) originally made by Radek Zlebcík (Charles University) to help make plots with ROOT.
+
+The `Campaigns` subgroup contains actual repos corresponding to different analyses with configs, plotting macros, and possibly CRAB outputs, or anything relevant to the reproducibility of an analysis. A template repo is provided with suggestions and guidelines.
 
 ## Installation
 
-First source the minimal environment for CMSSW (not necessary at CERN):
+### From scratch
+
+In general, it is recommended that you install the software on a fast disk to ensure fast compilation (e.g. AFS), but that you process the heavy n-tuples on a dedicated area (e.g. NFS at DESY, EOS at CERN).
+
+First clone the `Installer`. Then source the minimal environment for CMSSW, CRAB, and RUCIO:
 ```
+cd DasAnalysisSystem
 source ./setup
 ```
-It is currently working at CERN, at DESY, and for GitLab CI. Feel free to make a merge request to include your favourite facility.
+It is currently working at CERN (CH), at DESY (DE), at IIHE (BE), and for GitLab CI. Feel free to make a merge request to include your favourite facility.
 
-The installation is then trivial:
+Finally, the installation is trivial:
 ```
 make
 ```
+This takes a few minutes, then you're all set.
 
-### Remarks
+#### Remarks
 
-1. You may overwrite (at your own risks) the CMSSW release by running `make CMSSW=CMSSW_X_Y_Z` where you tune `X`, `Y`, and `Z` to the values of the release that you want to test. This may be necessary to run over certain data sets.
-2. `git-cms-init` is not run, since it is a priori not useful for the framework to run, take 100MB of space, and makes the installation slower. If it may happen to be useful, you have to add it in the Makefile. It must be run right after setting up the release, when it is locally still empty.
+1. You may overwrite (at your own risks) the CMSSW release by running `make CMSSW=CMSSW_X_Y_Z` where you tune `X`, `Y`, and `Z` to the values of the release that you want to test. This may be necessary to run over certain data sets; however, we cannot guarantee that the code will compile nor provide the same result as in the default release.
+2. `git-cms-init` is not run, since it is a priori not useful for the framework to run, takes 100MB of space, and makes the installation slower. If it may happen to be useful, you have to add it in the Makefile. It must be run right after setting up the release, when it is locally still empty.
 3. To use SSH instead of HTTPS, run `make GITLAB=ssh://git@gitlab.cern.ch:7999`.
+4. The `PlottingHelper` is not cloned by default as it is only useful for plotting purposes.
 
-## Setting up the environment
+### From an existing CMSSW release or from an existing installation.
+
+In the following, we explain the installation of the framework step by step. A few packages external to CMSSW are necessary (e.g. TUnfold), which you will need to tell CMSSW how to find them with `scram setup`.
+
+First source your CMSSW release and close to a directory where you want to install the external packages.
+1. Install TUnfold:
+```
+wget https://www.desy.de/~sschmitt/TUnfold/TUnfold_V17.9.tgz
+mkdir -p TUnfold TUnfold/lib
+mv TUnfold_V17.9.tgz TUnfold
+cd TUnfold
+tar xvzf TUnfold_V17.9.tgz
+make lib TUNFOLDVERSION='V17' -j
+mv -f libunfold.so lib/libtunfold.so
+mv -f TUnfoldV17Dict_rdict.pcm lib/
+(cd $CMSSW_BASE && scram setup $OLDPWD/../tunfold.xml)
+```
+2. Tell CMSSW where to find libgit2:
+```
+(cd $CMSSW_BASE && scram setup $OLDPWD/libgit2.xml)
+```
+3. Install ProtoDarwin:
+```
+git clone $(GITLAB)/paconnor/ProtoDarwin.git
+cmake3 -B build -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/build/install
+cmake3 --build build -j`nproc`
+cmake3 --build build --target install
+(cd $CMSSW_BASE && scram setup $OLDPWD/../protodarwin.xml)
+```
+4. Clone the `Tables` wherever you like and the `Core` in CMSSW, and compile as usual with `scram b -j`.
 
-First source the minimal environment for CMSSW as above.  Then just source the CMSSW release as usual with `cmsenv`.
+#### Remark
 
-To run CRAB jobs, you'll also need to set up a valid [grid certificate](https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookStartingGrid#ObtainingCert). If you already have done so, this will enable it:
+If you want to rerun jet clustering (e.g. to have AK8 jets at a low transverse momentum), you need to clone the [JetToolbox](https://github.com/cms-jet/JetToolbox.git):
+```
+git clone https://github.com/cms-jet/JetToolbox.git $CMSSW_BASE/JMEAnalysis/JetToolbox -b jetToolbox_120X
+```
+That version needs a minor patch:
+```
+cd $CMSSW_BASE/src/JMEAnalysis/JetToolbox
+patch -p1 $OLDPWD/jetToolbox_120X.patch
+```
 
+## Setting up the environment
+
+First source the minimal environment for CMSSW with `source ./setup`, then just source the CMSSW release as usual with `cmsenv`.
+
+To run CRAB jobs, you will also need to set up a valid [grid certificate](https://twiki.cern.ch/twiki/bin/view/CMSPublic/WorkBookStartingGrid#ObtainingCert). If you already have done so, this will enable it:
+```
 voms-proxy-init --rfc --voms cms -valid 192:00
+```
+
+To make [RUCIO](https://twiki.cern.ch/twiki/bin/viewauth/CMS/Rucio) requests, the `setup` file tried to guess your RUCIO username from your local username (unless was already set up). This is not guaranteed to work and you may have to define your RUCIO username ahead for sourcing the DAS environment.
 
 ### Good practices
 
 Fight against the increase of entropy by
 - pulling regularly,
-- commiting regularly,
-- making regular merge requests.
+- committing regularly,
+- document your code,
+- making frequent merge requests.
 
 Happy analysis!

jetToolbox_120X.patch

+diff --git a/python/jetToolbox_cff.py b/python/jetToolbox_cff.py
+index f545dea..6c3e422 100644
+--- a/python/jetToolbox_cff.py
++++ b/python/jetToolbox_cff.py
+@@ -164,9 +164,9 @@ def jetToolbox( proc, jetType, jetSequence, outputFile,
+                                             ]
+ 
+     if not bTagDiscriminators:
+-        if jetSize == 0.4: bTagDiscriminators = defaultBTagDiscriminators
+-        elif jetSize == 0.8: bTagDiscriminators = defaultBoostedBTagDiscriminators
+-        else:
++        #if jetSize == 0.4: bTagDiscriminators = defaultBTagDiscriminators
++        #elif jetSize == 0.8: bTagDiscriminators = defaultBoostedBTagDiscriminators
++        #else:
+             bTagDiscriminators = None
+             print('|---- jetToolBox: btag discriminators are recommended for AK4 and AK8 jets. For a different cone size, please especify the discriminators. Turning OFF all the default btag discriminators.')
+     if not subjetBTagDiscriminators:
+@@ -1305,7 +1305,7 @@ def jetToolbox( proc, jetType, jetSequence, outputFile,
+ 
+     #################################################################################
+     ###### Computing jetID
+-    from PhysicsTools.NanoAOD.jets_cff import looseJetId, tightJetId, tightJetIdLepVeto
++    from PhysicsTools.NanoAOD.jetsAK4_CHS_cff import looseJetId, tightJetId, tightJetIdLepVeto
+ 
+     mod["looseJetId"] = mod["PATJets"]+"looseJetId"
+     _addProcessAndTask( proc, mod["looseJetId"], looseJetId.clone(src = cms.InputTag(mod["PATJets"]) ))

setup

@@ -28,4 +28,9 @@ esac
 
 source /cvmfs/cms.cern.ch/cmsset_default.sh
 source /cvmfs/cms.cern.ch/rucio/setup-py3.sh
-export RUCIO_ACCOUNT=$USER
+
+if [ -n "$RUCIO_ACCOUNT" ]
+then
+    echo "\$RUCIO_ACCOUNT not found in environment. Setting up with \$USER."
+    export RUCIO_ACCOUNT=$USER
+fi