README.md 12.2 KB
Newer Older
1
2
ATLAS MC JobOptions
====================
3
4
5

For any technical issues contact [atlas-phys-mcprod-jo@cern.ch](mailto:atlas-phys-mcprod-jo@cern.ch)

6
# Instructions for registering new jobOptions (MC contacts)
Frank Siegert's avatar
Frank Siegert committed
7

8
9
10
For detailed step-by-step instructions on how to request a Monte Carlo sample, 
please see [McSampleRequestProcedure](https://twiki.cern.ch/twiki/bin/viewauth/AtlasProtected/McSampleRequestProcedure).

11
Before uploading new jO files to the repository, you need to make sure that you have tested the jO locally with `Gen_tf.py` and have included the `log.generate` produced by `Gen_tf.py` in the DSID directories you want to upload.  
12

13
**Important: if `log.generate` files are not provided, merge requests will in general not be accepted!**  
14

15
16
17
18
Once you have made sure that your jO run, the request has been discussed in an `ATLMCPROD` JIRA ticket and the relevant physics group is happy with the physics output, you can follow the next steps in order to
upload the jO to the repository.  

The repository is copied to `/cvmfs/atlas.cern.ch/repo/sw/Generators/MCJobOptions` every 4 hours starting at 00.00 CERN time.
19

20
## Using the automatic commit script
21

22
The procedure describe below makes use of the automatic commit script `scripts/commit_new_dsid.sh`, which contains several options. To get a list of options and their explanation run  
23
```
24
25
26
27
28
29
30
31
32
33
./scripts/commit_new_dsid.sh -h
```

## Steps for booking DSIDs and uploading new jO

1.  For each jO you want to upload, prepare a dummy DSID directory outside the `500xxx-999xxx` range, (e.g. `100xxx/100000`) and add all the necessary files (jO, gridpacks, etc) in the directory. To upload the directories
to git it is **strongly recommended** that you use the automatic commit script `commit_new_dsid.sh` as described below.

2.  **Dry run of automatic commit script** . From the `mcjoboptions` directory run the script using the `--dry-run` flag:
```
Spyros Argyropoulos's avatar
Spyros Argyropoulos committed
34
./scripts/commit_new_dsid.sh -m="Commit message" --dry-run /path/with/DSIDs/*
35
```  
Spyros Argyropoulos's avatar
Spyros Argyropoulos committed
36
37
38
39
or  
```
./scripts/commit_new_dsid.sh -d=DSID1,DSID2,DSIDn-DSIDk -m="Commit message" --dry-run
```
40
41
42
43
44
There are three ways to specify which DSID directories to commit:  
1.  using a comma-separated list of DSID directories with the `-d` option:  e.g. `-d=100000,100001`  
2.  using a continuous range of DSID directories with the `-d` option: e.g. `-d=100000-100005`
3.  directly providinng the paths to the directories where the jO files are stored, e.g. `./scripts/commit_new_dsid.sh /path/with/DSIDs/*`.  
The last option should expand to a **list of directories** containing the jO files, not the files themselves.
45

46
The script will loop through the directories and 
47
48
49
*  check if the jO files in the DSID directories specified with the `-d` flag contain any naming errors. If they do, it will print out the ERRORs. The user should fix them and rerun the command above.
*  the script will identify the generator used in each jO and will search for the smallest continuous block of DSIDs that is not already used in any of the remote branches. It will then print out where each dummy DSID will be copied
(e.g. `Will move 999xxx/999995 to 800xxx/800045`)
50
51
52
53
*  locate the `log.generate` and check if there are no errors from the `logParser`. 
  If errors are found, you will not be allowed to commit to the repository and you will first have to fix the errors in your local jO file.  
  If no errors are found, the script will create a file called `log.generate.short` in every DSID directory you want to upload.  

54
55
56
**Before proceeding to the next step**:
*  **make sure that the list of DSIDs that the script suggested make sense to you**. If you are not sure contact the mailing list.
*  **check the list of files which will be added to the commit and make sure that it is as expected.** If not contact the mailing list.
57

58
3. **Assignment of DSIDs**: Assuming the dry run (step 2) was successful, run the commit script using the `-n` or `--nogit` flag:  
59
```
60
./scripts/commit_new_dsid.sh -d=DSID1,DSID2,DSIDn-DSIDk -m="Commit message" -n
61
```  
62
63
64
65
66
67
This will go through the same steps as step (2) above with the difference that now the **dummy DSIDs will actually be copied to their final location**. At the end the script should print out where each initial DSID
has been copied and it will suggest the command to use in order to commit the DSIDs to the repository, e.g.  
```
The following DSIDs have been assigned:
    999xxx/999995 -> 800xxx/800045
    999xxx/999996 -> 800xxx/800046
68
Run: ./scripts/commit_new_dsid.sh -d=800045-800046 -m="Commit message" to push them to git
69
70
71
72
```

4.  **Upload to git**: run the script as suggested by the previous command  
```
73
./scripts/commit_new_dsid.sh -d=800045-800046 -m="Commit message"
74
75
```  
This will upload all the necessary files to git in a new branch called `dsid_USER_800045`. If you want to specify a custom branch name you can use `-b=branchName` when running the script.  
76

77
5.  A CI pipeline will be launched with this commit, as described in the pipelines section below. **Make sure that the pipelines run and are successful!** If they don't run please contact the package maintainers. If they fail, look into the pipeline output to see why they fail and update your MR accordingly. If it's unclear why they fail, contact the package maintainers.  
78

79
6.  Create a merge request [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/merge_requests) . The source branch should be the one created in step (2) above (e.g. `dsid_USER_DSID1`) and the target branch should be `atlas-physics/pmg/mcjoboptions/master`  
80

81
7.  Select the `new_jO` template in the title drop menu and fill in the description
82
83
84
85
86
87

## Skipping CI jobs

It is possible to skip certain CI jobs by manipulating the git commit messages, as described below in the Pipelines section.  
**This is in general discouraged and should only be performed by expert users. MR with skipped CI jobs will in general not be accepted!!!**  

88
To skip certain CI jobs run the commit script adding the `-m="[skip X],[skip Y]"` option, where `X,Y=all,modfiles,athena,logparser`, for example to commit the directory 800045 skipping the checks for modified files and the athena running do   
89
```
90
./scripts/commit_new_dsid.sh -d=800045 -m="Adding 800045 [skip modfiles] [skip athena]"
91
92
```

93
94
95
**Nota bene: Using the default gitlab rules to skip all pipelines, like `[skip ci]`, has the same effect as a failed pipeline, i.e. it leads to a merge request which cannot be merged according to the new gitlab policy for pipelines!**. 
The only way to skip all pipelines is to add `[skip all]` in the commit message, however this is reserved for expert users only and associated MRs will need approval from the PMG conveners.

96
# Accepting a MR with new jobOptions (jO coordinators)
97
98
99

Before accepting a MR make sure that all the items below have been completed successfully:  

100
* All the items in the todo list of the MR have been completed successfully
101
* The pipeline has run and the status is green
102
103
* CI jobs have not been skipped (if jobs have been skipped, confirmation from the PMG conveners is necessary before merging). 
* Look at the output of the `run_athena` pipeline and make sure that it has run for one of the DSIDs added to the commit. If it's not the case, confirmation from the PMG conveners is necessary before merging
104
* Check that no `log.generate.short` files are included in the commit to be merged to master. If such files are present, it indicates that something went wrong in the pipeline. Check the job logs and contact the package maintainers if needed.
105
106
107
108
109

# Instructions for developers

To push new changes to the repository follow the next steps:

110
1.  Check out the master branch and **before changing any file from the master** create a new branch with `git fetch --all && git checkout -b myBranch origin/master`  
111

112
2.  Add your changes with `git add ...`  
113

114
3.  Finally commit using an appropriate git message, e.g.  `git commit -m "appropriate message [skip modfiles]"` if you have modified pre-existing files residing in the DSID directories. Don't forget to push your changes using `git push origin myBranch`.  
115

116
4.  Create a merge request [here](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/merge_requests) . The source branch should be the one created in step (1) above (e.g. `myBranch`) and the target branch should be `atlas-physics/pmg/mcjoboptions/master`. Explain what your MR does.  
117

Spyros Argyropoulos's avatar
Spyros Argyropoulos committed
118
5.  **Make sure to check the option "Delete source branch when merge request is accepted" is selected**  
119

Spyros Argyropoulos's avatar
Spyros Argyropoulos committed
120
6.  **Make sure that an appropriate message is picked up with the correct CI tag** (by default the squashed commit message will coincide with the title of the MR, so if you want to skip certain CI jobs, you might need to add these in the squashed commit message by hand, if the title of the MR does not already contain the tag) 
121

122
123
124
125
126
127
128
129
130
131
132
133
134
## New password for `mcgensvc` account

When a new log-in password for `mcgensvc` is set, you need to update the value `K8S_SECRET_SERVICE_PASSWORD` in Settings > CI/CD > Variables.

## New grid certificate for `mcgensvc` account

When a new grid certificate is requested for `mcgensvc`:

1.  Encode the value of `ucercert.pem` with `base64 usercert.pem` and update the value `GRID_USERCERT` in Settings > CI/CD > Variables.
2.  Encode the value of `userkey.pem` with `base64 userkey.pem` and update the value `GRID_USERKEY` in Settings > CI/CD > Variables.
3.  The password that was used in the generation of `userkey.pem` should be copied to `GRID_PW` in Settings > CI/CD > Variables.
4.  Potentially the `export RUCIO_ACCOUNT` might need to be changed in `scripts/run_athena.sh` to match the person who requested the certificate.

135
# Pipelines
136

137
This repository uses CI pipelines that run the following jobs. A brief description of the pipelines is given below. Skipping pipelines is an expert operation restricted to the package maintainers. **No MR will be approved if pipelines have been skipped!**
138

139
1.  `check_jo_consistency`: 
140
*  **checks**: naming convention for jO files, length of jO file name, DSID ranges
141
142
*  **skippable**: NO  

143
2.    `check_unique_physicsShort`:
144
145
146
* **checks**: that there is no other jO file with the same physics short in the repository
* **skippable**: NO

147
148
149
150
151
3.    `check_unique_file`:
* **checks**: that there is no other python file with exactly the same name in the repository. If the job fails the requester should check if the files have the same content and if so try to rename the files if possible.
* **skippable**: NO

4.    `check_modified_files`:
152
* **checks**: that no pre-existing files have been modified or deleted (excluding files listed in `scripts/modifiable.txt`) and that no files have been added in pre-existing DSID directories
153
154
* **skippable**: with `[skip modfiles]`

155
5.  `check_added_files:`
156
157
158
* **checks**: that the files added are jO files (named like `mc.*.py`), integration grids (named like `*.GRID.tar.gz`) or directories that contain control files for generators (these have to be named after the generator)
* **skippable**: NO

159
6.    `check_tarball`:
160
* **checks**: that the GRID files have been put on `/eos/atlas` or `/eos/user` in a directory that is readable by `atlcvmfs`, `mcgensvc` and if the files are on `cvmfs` that they are readable by all users. It also checks that the number of files in the gridpack are less than 80k.
161
162
* **skippable**: NO

163
7.    `check_grid_size`:
164
* **checks**: that the GRID file size is less than 100 MB
165
* **skippable**:NO (but the job is allowed to fail)
166

167
8. `setup_athena`:
168
169
* **checks**: if new jO are added in a commit with a `log.generate.short` file  in the respective DSID directory, this job will create the configuration for a dynamic child pipeline that will consist of the `run_athena_DSID` and 
`check_logParser_DSID` jobs described below. A maximum of 5 jobs are created. If there are no new jO committed, a `dummy` job will be created in the child pipeline, which will always succeed.
170
171
* **skippable**: with `[skip athena]`

172
9. `run_athena_DSID`:
173
174
175
* **checks**: that `Gen_tf.py` can run successfully with the `DSID` file. `Gen_tf.py` will run **only if `log.generate.short` is provided with the commit**. 
* **skippable**: with `[skip athena]`

176
10.    `check_logParser_DSID`:
177
* **checks**: that the `run_athena_DSID` job produced no error
178
179
* **skippable**: with `[skip logparser]`. The job is also not run when `[skip athena]` is used

180
11.    `remove_logs`:
181
182
* **checks**: if `log.generate.short` files are present in the branch it will remove them
* **skippable**: NO
183
184
185