README.md 14 KB
Newer Older
1
2
# Notebooks for Circuit Analysis of HWC Tests and Events during Operation
Although, as the project name indicates, our primary goal is the development of signal monitoring applications, we realized that the analysis modules developed so far can be pieced together into HWC test and operation analysis notebooks.
Michal Maciejewski's avatar
Michal Maciejewski committed
3

4
The monitoring applications are organized system by system. Each analysis has been developed in a general way to account for all circuits in which the system was present. 
Michal Maciejewski's avatar
Michal Maciejewski committed
5
Thus, by taking a perpendicular view of the analysis table, a circuit analysis for this stance was possible.
6

7
8
9
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/monitoring-vs-hwc.png">

In particular, those notebooks are suited for HWC tests and operational analysis (FPA):
10
11
12
13
14

- can be adjusted on-the-fly for new requirements while performing a test; 
- can immediately generate a report for storage and distribution among a team of domain experts; 
- provide a sequential way of testing each system in a given order.

15
## User Guide
Michal Maciejewski's avatar
Michal Maciejewski committed
16
The execution of notebooks is carried out with SWAN service (http://swan.cern.ch) and requires four steps:
17

Michal Maciejewski's avatar
Michal Maciejewski committed
18
19
20
21
1. Getting NXCALS Access (once only)
2. Logging to SWAN
3. Setting up an appropriate environment script (done at each login)
4. Running an appropriate notebook
22

23
### 1. NXCALS Access
Michal Maciejewski's avatar
Michal Maciejewski committed
24
25
The analysis notebooks query certain signals with NXCALS. The NXCALS database requires an assignment of dedicated access rights for a user. 
Please follow a procedure below to request the NXCALS access.
26

Michal Maciejewski's avatar
Michal Maciejewski committed
27
1. Go to http://nxcals-docs.web.cern.ch/current/user-guide/data-access/nxcals-access-request/ for most recent procedure
Michal Maciejewski's avatar
Michal Maciejewski committed
28
2. Send an e-mail to mailto:acc-logging-support@cern.ch with the following pieces of information:
29

Michal Maciejewski's avatar
Michal Maciejewski committed
30
31
32
33
 - your NICE username
 - system: WinCCOA, CMW
 - NXCALS environment: PRO
 
Michal Maciejewski's avatar
Michal Maciejewski committed
34
Optionally please can mention that the NXCALS database will be accessed through SWAN.
Michal Maciejewski's avatar
Michal Maciejewski committed
35
36
Once the access is granted, you can use NXCALS with SWAN.

37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
### 2. EOS lhcsm Access

EOS lhcsm access are granted via e-groups.

Allow for storage of reports.
- cernbox-project-mp3-writers
- cernbox-project-mp3-readers

To modify the package (env), scripts and notebooks.
- cernbox-project-lhcsm-writers
- cernbox-project-lhcsm-readers

**Note it could take some time for the changes to propagate.**

### 3. Logging to SWAN
Michal Maciejewski's avatar
Michal Maciejewski committed
52
The following steps should be followed in order to log-in to SWAN
53

Michal Maciejewski's avatar
Michal Maciejewski committed
54
55
1. Go to http://swan.cern.ch
2. Login with your NICE account
Michal Maciejewski's avatar
Michal Maciejewski committed
56
  - SWAN is tightly integrated with CERNBox service (in fact, files created in SWAN are accessible in CERNBox). In case you have not yet used CERNBox, the following error message will be displayed indicating that your CERNBox account has not been activated yet. In order to activate your CERNBox account, please login on the website: http://cernbox.cern.ch. Afterwards, please login to SWAN service again. In case the error persists, please contact the SWAN support (see section **Help and Feedback** at the bottom of the guide).
Michal Maciejewski's avatar
Michal Maciejewski committed
57
 
58
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-inactive-cernbox-error.png" width=50%>
59

60
### 4. Setting an Environment Script
Michal Maciejewski's avatar
Michal Maciejewski committed
61
In order to execute the HWC notebooks, one requires `lhc-sm-api` package and HWC notebooks. To this end, we created a dedicated environment script to prepare the SWAN project space.
62

63
64
65
66
67
68
69
The script sets a path to a virtual environment with the necessary packages (for more details, cf. https://lhc-sm-api.web.cern.ch/lhc-sm-api/user_install.html#preinstalled-packages) as well as makes a copy of HWC notebooks. 

The SWAN Notebooks can be loaded in 3 different environments:
- PRO
- DEV
- DEV CONTRIBUTOR

70
#### 4.1 PRO
71
72
73
This is the PRO and stable version of the notebooks, users and experts should use this script.

At every log-in to SWAN, please provide the following environment script:
Michal Maciejewski's avatar
Michal Maciejewski committed
74
`/eos/project/l/lhcsm/public/packages_notebooks.sh`
75

76
77
78
**Note that in order to ensure compatibility between package and notebook versions, the `hwc` folder is deleted each time the script is executed.**
**Note: Notebooks changes are not persisted when using this environment script.**

79
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan_environment_script.png" width=25%>
80

Michal Maciejewski's avatar
Michal Maciejewski committed
81
82
83
84
85
Note the following settings while configuring environment:
- Software stack: `NXCals Python3`
- Platform: `CentOS 7 (gcc7)` - default
- Environment script: `/eos/project/l/lhcsm/public/packages_notebooks.sh`
- Number of cores: `4`
Michal Maciejewski's avatar
Michal Maciejewski committed
86
- Memory: `16 GB`
Michal Maciejewski's avatar
Michal Maciejewski committed
87
- Spark cluster: `BE NXCALS (NXCals)`
88

89
90
This script creates a new `hwc` folder in SWAN.

91
#### 4.2 DEV
92
93
94
95
96
97
98
99
100
101
This is the DEV version of the notebooks, this script is used for new feature validations and verification.

At every log-in to SWAN, please provide the following environment script:
`/eos/project/l/lhcsm/public/packages_notebooks_dev.sh`

This script creates a new `hwc_dev` folder in SWAN.

**Note: that in order to ensure compatibility between package and notebook versions, the `hwc_dev` folder is deleted each time the script is executed.**
**Note: Notebooks changes are not persisted when using this environment script.**

102
#### 4.3 DEV CONTRIBUTOR
103
104
105
106
107
This is a DEV version of the notebooks, this script is used to implement new features and new notebooks.
see [CONTRIBUTING.md](CONTRIBUTING.md)

This script uses the `hwc_working` folder in SWAN. If the folder already exists, then it just loads the DEV environment.

108
109
### 5. Running Notebook
#### 5.1. Open notebook 
110

111
To do so simply open the folder and then select a circuit. Afterwards click name of a notebook to open a new page. The top of the notebook is presented in Figure below.
112

113
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-rb-fpa-analysis-intro.png" width=50%>
114
115


116
#### 5.2. Connect to the NXCALS Spark Cluster
117
118
Once a notebook is opened, please click a star button as shown in Figure below in order to open the Spark cluster configuration in a panel on the right side of an active notebook.

119
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-open-spark-cluster-configuration.png" width=50%>
120
121

Figure below shows a three-step procedure of Spark cluster connection. The first step involves providing the NICE account password. The second step allows setting additional settings for the connection. In order to connect with NXCALS please make sure to enable the following options:
122

123
- Include NXCALS options - to connect to the cluster
Michal Maciejewski's avatar
Michal Maciejewski committed
124
- Include SparkMetrics options - to enable metrics used for analysing NXCALS queries 
125
126
127

The last step is a confirmation of a successful connection to the cluster.

128
129
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-spark-cluster-connection.png" width=75%>

130
#### 5.3. Analysis Notebook Execution
131
132
133
134
135
136
137
138
A notebook is composed by cells.  A cell contains either a markdown text with description or python code to execute. Cells with markdown text have white background and can contain text, tables, figures, and hyperlinks. Cells with code have gray background and are executed by clicking a run icon in the top bar highlighted in Figure below. Alternatively, one can put a cursor in a cell with code an press a keyboard shortcut Ctrl+Enter.

<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-execute-cell.png" width=50%>

A state of a cell is indicated by square brackets located on the left to a cell. Execution of a cell is indicated by a star in the square brackets. Once cell execution is completed the star changes into a number representing the order of cell execution. A cell can execute for too long due to connection problems, issues with a database query, kernel problems. In this case, two actions are recommended:

1.  Select from the top menu: `Kernel` -> `Interrupt` and execute the problematic cell again (either a run button (cf. Figure above) or Ctrl+Enter).
2.  In case the first option does not help, select from the top menu `Kernel` -> `Restart & Clear Output`. Then all cells prior to the problematic one have to be executed again (multiple cell selection is possible by clicking on the left of a cell to select it and afterwards selecting others with pressed Shift button).  After this operation one needs to reconnect to the NXCALS Spark cluster.
139

140
#### 5.4. Analysis Assumptions:
141

142
143
1. We consider standard analysis scenarios, i.e., all signals can be queried. Depending on what signal is missing, an analysis can raise a warning and continue or an error and abort the analysis.
2. It is recommended to execute each cell one after another. However, since the signals are queried prior to an analysis, any order of execution is allowed. In case an analysis cell is aborted, the following ones may not be executed (e.g. I_MEAS not present).
144

145
#### 5.5. FPA Notebooks
146

147
##### Analysis Workflow
148

149
An FPA analysis workflow consists of four steps: (i) finding an FGC Post Mortem timestamp; (ii) executing analysis cells on the cluster (iii); (iv) storing output files on EOS; see Figure below.
150

151
152
153
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/fpa-analysis-workflow.png" width=75%>

##### Notebook Structure
154

155
The RB FPA Analysis notebook is organized into 10 chapters (Note that for the remaining circuits, some analyses may not be present.):
156

157
0. Initialise the working environment
158
  Loads external packages as well as lhcsmapi classes required to perform analysis and plot results.
159
1. Select FGC Post Mortem Entry
160
  After executing this cell, a FGC Post Mortem GUI with default settings is displayed.
Michal Maciejewski's avatar
Michal Maciejewski committed
161

162
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-rb-fpa-analysis-fgc-pm-browser-empty.png" width=75%>
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179

The GUI consists of 8 widgets described in Table below.

|Widget|Description|
|------|-----------|
|Circuit name|Circuit name|
|Start|Start date and time|
|End|End date and time|
|Analysis|Automatic (each cell executed without user input); Manual (some analysis steps take expert comment)|
|Done by|NICE login of a person executing the analysis|
|Find FGC PM entries|Button triggering a search of FGC PM entries|
|Query progress bar|Displays progress of querying days in between indicated datesFGC PM EntriesList of FGC PM timestamps|

**Please note that in order to execute any of the following cells, there should be at least one entry in the FGC PM Entries list. The list is populated after clicking [Find FGC PM entries button].**

Figure below shows the GUI after clicking button [Find FGC PM entries] with the default settings. Note that the list only contains FGC PM timestamps surrounded by QPS timestamps (1 minute before and 5 minutes after an FGC PM timestamp).

180
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-rb-fpa-analysis-fgc-pm-browser.png" width=75%>
181
182
183
184
185
186
187
188

2. Query All Signals Prior to Analysis  
In order to avoid delays between analyses, the necessary signals are queried prior to performing the analysis.
3. Timestamps  
Table of timestamps main systems representing the sequence of events for a given analysis.
4. Schematic  
Interactive schematic of the RB circuit composed of: power converter, two energy extraction systems, current leads, magnets, and nQPS crates. Hovering a mouse over a center of a box representing a system provides additional pieces of information. Location of quenched magnets is highlighted. Slider below the schematic enables its scrolling.

189
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/rb-schematic.png" width=75%>
190
191
192
193
194
195

5. PIC  
Check of PIC timestamps
6. Power Converter  
Analysis of the main power converter as well as earth currents.  
7. Energy Extraction Analysis of the energy extraction voltage and temperature
196
197
198
8. Quench Protection System Analysis of the quench detection system, quench heaters, diode, voltage feelers, diode leads resistance, current leads voltage (resistive and HTS).
9. Plot of Energy Extraction after 3 h from an FPA
10. Final Report  
199
Saving of the CSV results table and HTML report to EOS folder.
200
The RQ analysis notebook follows the same structure except for the lack of schematic. Typically, there is only a single main quadrupole magnet quenching and the schematic does not provide more information as compared to the timestamps table in point 3. For the remaining circuits the analysis cells reflect the presence of particular hardware.
201

202
##### Notebook Output
203

Michal Maciejewski's avatar
Michal Maciejewski committed
204
The notebook creates two output files in the folder (path with Windows convention)
205
```
206
\\cernbox-smb\eos\project\m\mp3\$circuit_type$\$circuit_name$\FPA
207
208
209
```
e.g., 
```
210
\\cernbox-smb\eos\project\m\lhcsm\operation\RB\RB.A12\FPA
211
212
```

213
214
- HTML report file with the snapshot of the entire notebook - [circuit-name]_FPA-[fgc-timestamp]-[analysis-execution-date].html;
- CSV file with MP3 results table with a subset analysis results - [circuit-name]_FPA-[fgc-timestamp]-[analysis-execution-date].csv;
215

Michal Maciejewski's avatar
Michal Maciejewski committed
216

217
#### 5.6. HWC Notebooks
218
##### Analysis Workflow
219
220
221

A HWC analysis workflow consists of four steps: (i) finding of start and end time of an HWC test (ii) executing analysis cells on the cluster (iii); (iv) storing output files on EOS; see Figure below.

222
223
224
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/hwc-analysis-workflow.png" width=75%>

##### Notebook Structure
225

226
Each notebook is composed of initial part with a circuit schematic, test current profile, and table summarising test criteria. This part is followed by package import instructions, display and the browser of historical HWC tests (until and including HWC 2018); see Figure below.
227

228
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-hwc-browser.png" width=75%>
229

230
231
232
There is a manual integration with AccTesting that requires a copy&paste of a text parameterising a notebook

<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-manual-acctesting-integration.png">
233
234
235

The remainder of each notebook depends on the particular test to be performed. At the end of an HWC notebook there are instructions for saving the output files.

236
##### Notebook Output
237

238
The notebook creates an output in the folder (path with Windows convention)
239
```
240
\\cernbox-smb\eos\project\m\mp3\$circuit_type$\$circuit_name$\$hwc_test$\
241
242
243
```
e.g., 
```
244
\\cernbox-smb\eos\project\m\mp3\RB\RB.A12\PNO.b2\:
245
246
```

247
- HTML report file with the snapshot of the entire notebook - [circuit-name]_[test_name]-[test-start]-[analysis-execution-date].html;