Instructions for registering new jobOptions (MC contacts)

Before uploading new jO files to the repository, you need to make sure that you have tested the jO locally with Gen_tf.py and have included the log.generate produced by Gen_tf.py in the DSID directories you want to upload.

Important: if log.generate files are not provided, merge requests will in general not be accepted!

Once you have made sure that your jO run, the request has been discussed in an ATLMCPROD JIRA ticket and the relevant physics group is happy with the physics output, you can follow the next steps in order to upload the jO to the repository.

The repository is copied to /cvmfs/atlas.cern.ch/repo/sw/Generators/MCJobOptions every 4 hours starting at 00.00 CERN time.

Steps for uploading new jO

1. Execute the automatic commit script from the mcjoboptions directory

./scripts/commit_new_dsid.sh -d=DSID1,DSID2,DSIDn-DSIDk -n

where DSID1,DSID2 is a comma-separated list of the new DSID directories to be added to the commit. DSID ranges are also supported using a dash, e.g. -d=421001-421003 will submit DSIDs 421001, 421002, 421003. It is also possible to combine individual DSIDs with a DSID range as indicated in the example above.

The script will loop through the directories and locate the log.generate and check if there are no errors from the logParser. If errors are found, you will not be allowed to commit to the repository and you will first have to fix the errors in your local jO file. If no errors are found, the script will create a file called log.generate.short in every DSID directory you want to upload.

The script will also print out the names of the files which will be added to the commit. Check the list and make sure that it is as expected. If not contact the package maintainers.

2. Run

./scripts/commit_new_dsid.sh -d=DSID1,DSID2,...

This will upload all the necessary files to git in a new branch called dsid_USER_DSID1.

3. A CI pipeline will be launched with this commit, as described in the pipelines section below. Make sure that the pipelines run and are successful! If they don't run please contact the package maintainers. If they fail, look into the pipeline output to see why they fail and update your MR accordingly. If it's unclear why they fail, contact the package maintainers.

4. Create a merge request here . The source branch should be the one created in step (2) above (e.g. dsid_USER_DSID1) and the target branch should be atlas-physics/pmg/mcjoboptions/master

5. Select the new_jO template in the title drop menu and fill in the description

Skipping CI jobs

It is possible to skip certain CI jobs by manipulating the git commit messages, as described below in the Pipelines section.
This is in general discouraged and should only be performed by expert users. MR with skipped CI jobs will in general not be accepted!!!

To skip certain CI jobs run the commit script adding the -m="[skip X],[skip Y]" option, where X,Y=ci,modfiles,size,athena,logparser, for example to commit the directory 421001 skipping the checks for modified files and the athena running do

./scripts/commit_new_dsid.sh -d=DSID1,DSID2,... -m="[skip modfiles],[skip athena]"

Accepting a MR with new jobOptions

Before accepting a MR make sure that all the items below have been completed successfully:

• All the items in the todo list of the MR have been comleted successfully
• The pipeline has run and the status is green
• CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging)
• Look at the output of the run_athena pipeline and make sure that it has run for all DSIDS added to the commit. If it's not the case, confirmation from the PMG conveners is necessary before merging
• Check that no log.generate.short files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.

Instructions for developers

To push new changes to the repository follow the next steps:

1. Check out the master branch and before changing any file from the master create a new branch with git checkout -b myBranch

2. Push the branch with git push -o ci.skip origin myBranch. The -o option works with git 2.0 or higher. If you are on lxplus simply do setupATLAS; lsetup git to set up a recent git version.

3. Add your changes with git add ...

4. Finally commit using an appropriate git message, e.g. git commit -m "appropriate message [skip modfiles]" if you have modified pre-existing files. Don't forget to push your changes using git push origin myBranch.

5. Create a merge request here . The source branch should be the one created in step (1) above (e.g. myBranch) and the target branch should be atlas-physics/pmg/mcjoboptions/master. Explain what your MR does.

6. Make sure to check the options "Delete source branch when merge request is accepted" and "Squash commits when merge request is accepted."

7. Make sure that when squashing commits an appropriate message is picked up with the correct CI tag (by default the squashed commit message will coincide with the title of the MR, so if you want to skip certain CI jobs, you might need to add these in the squashed commit message by hand, if the title of the MR does not already contain the tag)

Pipelines

This repository uses CI pipelines that run the following jobs. A brief description of the pipelines is given below. Skipping pipelines is an expert operation restricted to the package maintainers. No MR will be approved if pipelines have been skipped!

1. check_jo_consistency:

• checks: naming convention for jO files, length of jO file name
• skippable: NO

2. check_unique_physicsShort:

• checks: that there is no other jO file with the same physics short in the repository
• skippable: NO

3. check_modified_files:

• checks: that no pre-existing files have been modified or deleted and that no files have been added in pre-existing DSID directories
• skippable: with [skip modfiles]

4. check_added_files:

• checks: that the files added are jO files (named like mc.*.py), integration grids (named like *.GRID.tar.gz) or directories that contain control files for generators (these have to be named after the generator)
• skippable: NO

5. check_grid_atlcvmfs:

• checks: that the GRID files have been put on /eos/atlas or /eos/user in a directory that is readable by atlcvmfs
• skippable: NO

6. check_grid_size:

• checks: that the GRID file size is less than 100 MB
• skippable: with [skip size]

7. run_athena:

• checks: that Gen_tf.py can run successfully with the uploaded jO files
• skippable: with [skip athena]

8. check_logParser:

• checks: that a log.generate file exists for every new DSID directory added with a commit and that the log.generate contains no errors
• skippable: with [skip logparser]. The job is also not run when [skip athena] is used