README.md 6.64 KB
Newer Older
Simon Spannagel's avatar
Simon Spannagel committed
1
2
3
4
5
6
7
8
9
10
11
12
13
## Jobsub

### Overview

`jobsub` is a tool for the convenient run-specific modification of Corryvreckan configuration files and their execution through the `corry` executable. It is derived from the original `jobsub` written for EUTelescope by Hanno Perrey, Lund University.

### Usage

The following help text is printed when invoking `jobsub` with the `-h` argument:

```result
usage: jobsub.py [-h] [--option NAME=VALUE] [-c FILE] [-csv FILE]
                 [--log-file FILE] [-l LEVEL] [-s] [--dry-run]
14
		 [--batch FILE] [--subdir]
Simon Spannagel's avatar
Simon Spannagel committed
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
                 jobtask [runs [runs ...]]

A tool for the convenient run-specific modification of Marlin steering files
and their execution through the Marlin processor

positional arguments:
  runs                  The runs to be analyzed; can be a list of single runs
                        and/or a range, e.g. 1056-1060.

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -c FILE, --conf-file FILE, --config FILE
                        Configuration file with all Corryvreckan algorithms
                        defined
  --option NAME=VALUE, -o NAME=VALUE
                        Specify further options such as 'beamenergy=5.3'. This
                        switch be specified several times for multiple options
                        or can parse a comma-separated list of options. This
                        switch overrides any config file options and also
                        overwrites hard-coded settings on the Corryvreckan
                        configration file.
37
38
39
  -htc FILE, --htcondor-file FILE, --batch FILE
                        Specify condor_submit parameter file for HTCondor submission. Run
                        HTCondor submission via condor_submit instead of calling
Simon Spannagel's avatar
Simon Spannagel committed
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
                        Corryvreckan directly
  -csv FILE, --csv-file FILE
                        Load additional run-specific variables from table
                        (text file in csv format)
  --log-file FILE       Save submission log to specified file
  -v LEVEL, --verbosity LEVEL
                        Sets the verbosity of log messages during job
                        submission where LEVEL is either debug, info, warning
                        or error
  -s, --silent          Suppress non-error (stdout) Corryvreckan output to
                        console
  --dry-run             Write configuration files but skip actual Corryvreckan
                        execution
  --subdir              Execute every job in its own subdirectory instead of
                        all in the base path
  --plain               Output written to stdout/stderr and log file in
                        prefix-less format i.e. without time stamping
  --zfill N             Fill run number with zeros up to the defined number of
                        digits
```

61
62
The environment variables need to be set using ```source etc/setup_lxplus.sh```.
When using a submission file, `getenv = True` should be used (see example.sub).
Jens Kroeger's avatar
Jens Kroeger committed
63

Simon Spannagel's avatar
Simon Spannagel committed
64
65
66
67
68
69
70
71
72
### Preparation of Configuration File Templates

Configuration file templates are valid Corryvreckan configuration files in TOML format, where single values are replaced by variables in the form `@SomeVariable@`.
A more detailed description of the configuration file format can be found elsewhere in the user manual.
The section of a configuration file template with variable geometry file and DUT name could e.g. look like

```toml
[Corryvreckan]
detectors_file = "@telescopeGeometry@"
73
histogramFile = "histograms_@RunNumber@.root"
Simon Spannagel's avatar
Simon Spannagel committed
74
75
76
77
78
79
80
81
82
83
84

number_of_events = 5000000

log_level = WARNING
```

When `jobsub` is executed, these placeholders are replaced with user-defined values that can be specified through command-line arguments or a table with a row for each run number processed, and a final configuration file is produced for each run separately, e.g.

```toml
[Corryvreckan]
detectors_file = "my_telescope_Nov2017_1.conf"
85
histogram_file = "histograms_run999.root"
Simon Spannagel's avatar
Simon Spannagel committed
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130

number_of_events = 5000000

log_level = WARNING
```

There is only one predefined placeholder, `@RunNumber@`, which will be substituted with the current run number. Run numbers are not padded with leading zeros unless the `--zfill` option is provided.

### Using Configuration Variables

As described in the previous paragraph, variables in the configuration file template are replaced with values at run time.
Two sources of values are currently supported, and are described in the following.

#### Command Line
   Variable substitutions can be specified using the `--option` or `-o` command line switches, e.g.

   ```bash
   jobsub.py --option beamenergy=5.3 -c alignment.conf 1234
   ```

   This switch be specified several times for multiple options or can parse a comma-separated list of options. This switch overrides any config file options.

#### Table (comma-separated text file)
   - format: e.g.
     - export from Open/LibreOffice with default settings (UTF-8,comma-separated, text-field delimiter: ")
     - emacs org-mode table (see http://orgmode.org/manual/Tables.html)
     - use Atom's *tablr* extension
   - commented lines (starting with #) are ignored
   - first row (after comments) has to provide column headers which identify the variables in the steering template to replace (case-insensitive)
   - requires one column labeled "RunNumber"
   - only considers placeholders left in the steering template after processing command-line arguments and config file options
   -
##### Example
    The CSV file could have the following form:

    ```csv
    RunNumber, BeamEnergy, telescopeGeometry
          4115,          1, telescope_june2017_1.conf
          4116,          2, telescope_june2017_1.conf
          4117,          3, telescope_june2017_1.conf
          4118,          4, telescope_june2017_1.conf
          4119,          5, telescope_june2017_1.conf
    ```

    Using this table, the variables `@BeamEnergy@` and `@telescopeGeometry@` in the templates would be replaced by the values corresponding to the current run number.
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
    
### Example Usage with a Batch File:

Example command line usage:
```bash
./jobsub.py -c /path/to/example.conf -v DEBUG --batch /path/to/example.sub --subdir <run_number>
```

The batch file needs to look like `example.sub`:
```
output                  = log/corryvreckan.$(ClusterId).$(ProcId).out
error                   = log/corryvreckan.$(ClusterId).$(ProcId).err
log                     = log/corryvreckan.$(ClusterId).log
should_transfer_files   = Yes
when_to_transfer_output = ON_EXIT
transfer_output_files   = output/histograms_$(RunNumber).root
getenv                  = True
queue
```
where it is important that the output file name corresponds to the one set in the configuration file.