README.md 13.7 KB
Newer Older
1
2
# Notebooks for Circuit Analysis of HWC Tests and Events during Operation
Although, as the project name indicates, our primary goal is the development of signal monitoring applications, we realized that the analysis modules developed so far can be pieced together into HWC test and operation analysis notebooks.
Michal Maciejewski's avatar
Michal Maciejewski committed
3

4
The monitoring applications are organized system by system. Each analysis has been developed in a general way to account for all circuits in which the system was present. 
Michal Maciejewski's avatar
Michal Maciejewski committed
5
Thus, by taking a perpendicular view of the analysis table, a circuit analysis for this stance was possible.
6

7
8
9
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/monitoring-vs-hwc.png">

In particular, those notebooks are suited for HWC tests and operational analysis (FPA):
10
11
12
13
14

- can be adjusted on-the-fly for new requirements while performing a test; 
- can immediately generate a report for storage and distribution among a team of domain experts; 
- provide a sequential way of testing each system in a given order.

15
## User Guide
Michal Maciejewski's avatar
Michal Maciejewski committed
16
The execution of notebooks is carried out with SWAN service (http://swan.cern.ch) and requires four steps:
17

Michal Maciejewski's avatar
Michal Maciejewski committed
18
19
20
21
1. Getting NXCALS Access (once only)
2. Logging to SWAN
3. Setting up an appropriate environment script (done at each login)
4. Running an appropriate notebook
22

23
### 1. NXCALS Access
Michal Maciejewski's avatar
Michal Maciejewski committed
24
25
The analysis notebooks query certain signals with NXCALS. The NXCALS database requires an assignment of dedicated access rights for a user. 
Please follow a procedure below to request the NXCALS access.
26

Michal Maciejewski's avatar
Michal Maciejewski committed
27
1. Go to http://nxcals-docs.web.cern.ch/current/user-guide/data-access/nxcals-access-request/ for most recent procedure
Michal Maciejewski's avatar
Michal Maciejewski committed
28
2. Send an e-mail to mailto:acc-logging-support@cern.ch with the following pieces of information:
29

Michal Maciejewski's avatar
Michal Maciejewski committed
30
31
32
33
 - your NICE username
 - system: WinCCOA, CMW
 - NXCALS environment: PRO
 
Michal Maciejewski's avatar
Michal Maciejewski committed
34
Optionally please can mention that the NXCALS database will be accessed through SWAN.
Michal Maciejewski's avatar
Michal Maciejewski committed
35
36
Once the access is granted, you can use NXCALS with SWAN.

37
### 2. Logging to SWAN
Michal Maciejewski's avatar
Michal Maciejewski committed
38
The following steps should be followed in order to log-in to SWAN
39

Michal Maciejewski's avatar
Michal Maciejewski committed
40
41
1. Go to http://swan.cern.ch
2. Login with your NICE account
Michal Maciejewski's avatar
Michal Maciejewski committed
42
  - SWAN is tightly integrated with CERNBox service (in fact, files created in SWAN are accessible in CERNBox). In case you have not yet used CERNBox, the following error message will be displayed indicating that your CERNBox account has not been activated yet. In order to activate your CERNBox account, please login on the website: http://cernbox.cern.ch. Afterwards, please login to SWAN service again. In case the error persists, please contact the SWAN support (see section **Help and Feedback** at the bottom of the guide).
Michal Maciejewski's avatar
Michal Maciejewski committed
43
 
44
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-inactive-cernbox-error.png" width=50%>
45

46
### 3. Setting an Environment Script
Michal Maciejewski's avatar
Michal Maciejewski committed
47
In order to execute the HWC notebooks, one requires `lhc-sm-api` package and HWC notebooks. To this end, we created a dedicated environment script to prepare the SWAN project space.
48

49
50
51
52
53
54
55
56
57
58
59
The script sets a path to a virtual environment with the necessary packages (for more details, cf. https://lhc-sm-api.web.cern.ch/lhc-sm-api/user_install.html#preinstalled-packages) as well as makes a copy of HWC notebooks. 

The SWAN Notebooks can be loaded in 3 different environments:
- PRO
- DEV
- DEV CONTRIBUTOR

#### 3.1 PRO
This is the PRO and stable version of the notebooks, users and experts should use this script.

At every log-in to SWAN, please provide the following environment script:
Michal Maciejewski's avatar
Michal Maciejewski committed
60
`/eos/project/l/lhcsm/public/packages_notebooks.sh`
61

62
63
64
**Note that in order to ensure compatibility between package and notebook versions, the `hwc` folder is deleted each time the script is executed.**
**Note: Notebooks changes are not persisted when using this environment script.**

65
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan_environment_script.png" width=25%>
66

Michal Maciejewski's avatar
Michal Maciejewski committed
67
68
69
70
71
Note the following settings while configuring environment:
- Software stack: `NXCals Python3`
- Platform: `CentOS 7 (gcc7)` - default
- Environment script: `/eos/project/l/lhcsm/public/packages_notebooks.sh`
- Number of cores: `4`
Michal Maciejewski's avatar
Michal Maciejewski committed
72
- Memory: `16 GB`
Michal Maciejewski's avatar
Michal Maciejewski committed
73
- Spark cluster: `BE NXCALS (NXCals)`
74

75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
This script creates a new `hwc` folder in SWAN.

#### 3.2 DEV
This is the DEV version of the notebooks, this script is used for new feature validations and verification.

At every log-in to SWAN, please provide the following environment script:
`/eos/project/l/lhcsm/public/packages_notebooks_dev.sh`

This script creates a new `hwc_dev` folder in SWAN.

**Note: that in order to ensure compatibility between package and notebook versions, the `hwc_dev` folder is deleted each time the script is executed.**
**Note: Notebooks changes are not persisted when using this environment script.**

#### 3.3 DEV CONTRIBUTOR
This is a DEV version of the notebooks, this script is used to implement new features and new notebooks.
see [CONTRIBUTING.md](CONTRIBUTING.md)

This script uses the `hwc_working` folder in SWAN. If the folder already exists, then it just loads the DEV environment.

94
95
### 4. Running Notebook
#### 4.1. Open notebook 
96

97
To do so simply open the folder and then select a circuit. Afterwards click name of a notebook to open a new page. The top of the notebook is presented in Figure below.
98

99
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-rb-fpa-analysis-intro.png" width=50%>
100
101


102
#### 4.2. Connect to the NXCALS Spark Cluster
103
104
Once a notebook is opened, please click a star button as shown in Figure below in order to open the Spark cluster configuration in a panel on the right side of an active notebook.

105
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-open-spark-cluster-configuration.png" width=50%>
106
107

Figure below shows a three-step procedure of Spark cluster connection. The first step involves providing the NICE account password. The second step allows setting additional settings for the connection. In order to connect with NXCALS please make sure to enable the following options:
108

109
- Include NXCALS options - to connect to the cluster
Michal Maciejewski's avatar
Michal Maciejewski committed
110
- Include SparkMetrics options - to enable metrics used for analysing NXCALS queries 
111
112
113

The last step is a confirmation of a successful connection to the cluster.

114
115
116
117
118
119
120
121
122
123
124
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-spark-cluster-connection.png" width=75%>

#### 4.3. Analysis Notebook Execution
A notebook is composed by cells.  A cell contains either a markdown text with description or python code to execute. Cells with markdown text have white background and can contain text, tables, figures, and hyperlinks. Cells with code have gray background and are executed by clicking a run icon in the top bar highlighted in Figure below. Alternatively, one can put a cursor in a cell with code an press a keyboard shortcut Ctrl+Enter.

<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-execute-cell.png" width=50%>

A state of a cell is indicated by square brackets located on the left to a cell. Execution of a cell is indicated by a star in the square brackets. Once cell execution is completed the star changes into a number representing the order of cell execution. A cell can execute for too long due to connection problems, issues with a database query, kernel problems. In this case, two actions are recommended:

1.  Select from the top menu: `Kernel` -> `Interrupt` and execute the problematic cell again (either a run button (cf. Figure above) or Ctrl+Enter).
2.  In case the first option does not help, select from the top menu `Kernel` -> `Restart & Clear Output`. Then all cells prior to the problematic one have to be executed again (multiple cell selection is possible by clicking on the left of a cell to select it and afterwards selecting others with pressed Shift button).  After this operation one needs to reconnect to the NXCALS Spark cluster.
125

126
#### 4.4. Analysis Assumptions:
127

128
129
1. We consider standard analysis scenarios, i.e., all signals can be queried. Depending on what signal is missing, an analysis can raise a warning and continue or an error and abort the analysis.
2. It is recommended to execute each cell one after another. However, since the signals are queried prior to an analysis, any order of execution is allowed. In case an analysis cell is aborted, the following ones may not be executed (e.g. I_MEAS not present).
130

131
#### 4.5. FPA Notebooks
132

133
##### Analysis Workflow
134

135
An FPA analysis workflow consists of four steps: (i) finding an FGC Post Mortem timestamp; (ii) executing analysis cells on the cluster (iii); (iv) storing output files on EOS; see Figure below.
136

137
138
139
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/fpa-analysis-workflow.png" width=75%>

##### Notebook Structure
140

141
The RB FPA Analysis notebook is organized into 10 chapters (Note that for the remaining circuits, some analyses may not be present.):
142

143
0. Initialise the working environment
144
  Loads external packages as well as lhcsmapi classes required to perform analysis and plot results.
145
1. Select FGC Post Mortem Entry
146
  After executing this cell, a FGC Post Mortem GUI with default settings is displayed.
Michal Maciejewski's avatar
Michal Maciejewski committed
147

148
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-rb-fpa-analysis-fgc-pm-browser-empty.png" width=75%>
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165

The GUI consists of 8 widgets described in Table below.

|Widget|Description|
|------|-----------|
|Circuit name|Circuit name|
|Start|Start date and time|
|End|End date and time|
|Analysis|Automatic (each cell executed without user input); Manual (some analysis steps take expert comment)|
|Done by|NICE login of a person executing the analysis|
|Find FGC PM entries|Button triggering a search of FGC PM entries|
|Query progress bar|Displays progress of querying days in between indicated datesFGC PM EntriesList of FGC PM timestamps|

**Please note that in order to execute any of the following cells, there should be at least one entry in the FGC PM Entries list. The list is populated after clicking [Find FGC PM entries button].**

Figure below shows the GUI after clicking button [Find FGC PM entries] with the default settings. Note that the list only contains FGC PM timestamps surrounded by QPS timestamps (1 minute before and 5 minutes after an FGC PM timestamp).

166
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-rb-fpa-analysis-fgc-pm-browser.png" width=75%>
167
168
169
170
171
172
173
174

2. Query All Signals Prior to Analysis  
In order to avoid delays between analyses, the necessary signals are queried prior to performing the analysis.
3. Timestamps  
Table of timestamps main systems representing the sequence of events for a given analysis.
4. Schematic  
Interactive schematic of the RB circuit composed of: power converter, two energy extraction systems, current leads, magnets, and nQPS crates. Hovering a mouse over a center of a box representing a system provides additional pieces of information. Location of quenched magnets is highlighted. Slider below the schematic enables its scrolling.

175
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/rb-schematic.png" width=75%>
176
177
178
179
180
181

5. PIC  
Check of PIC timestamps
6. Power Converter  
Analysis of the main power converter as well as earth currents.  
7. Energy Extraction Analysis of the energy extraction voltage and temperature
182
183
184
8. Quench Protection System Analysis of the quench detection system, quench heaters, diode, voltage feelers, diode leads resistance, current leads voltage (resistive and HTS).
9. Plot of Energy Extraction after 3 h from an FPA
10. Final Report  
185
Saving of the CSV results table and HTML report to EOS folder.
186
The RQ analysis notebook follows the same structure except for the lack of schematic. Typically, there is only a single main quadrupole magnet quenching and the schematic does not provide more information as compared to the timestamps table in point 3. For the remaining circuits the analysis cells reflect the presence of particular hardware.
187

188
##### Notebook Output
189

Michal Maciejewski's avatar
Michal Maciejewski committed
190
The notebook creates two output files in the folder (path with Windows convention)
191
```
192
\\cernbox-smb\eos\project\m\mp3\$circuit_type$\$circuit_name$\FPA
193
194
195
```
e.g., 
```
196
\\cernbox-smb\eos\project\m\lhcsm\operation\RB\RB.A12\FPA
197
198
```

199
200
- HTML report file with the snapshot of the entire notebook - [circuit-name]_FPA-[fgc-timestamp]-[analysis-execution-date].html;
- CSV file with MP3 results table with a subset analysis results - [circuit-name]_FPA-[fgc-timestamp]-[analysis-execution-date].csv;
201

Michal Maciejewski's avatar
Michal Maciejewski committed
202

203
204
#### 4.6. HWC Notebooks
##### Analysis Workflow
205
206
207

A HWC analysis workflow consists of four steps: (i) finding of start and end time of an HWC test (ii) executing analysis cells on the cluster (iii); (iv) storing output files on EOS; see Figure below.

208
209
210
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/hwc-analysis-workflow.png" width=75%>

##### Notebook Structure
211

212
Each notebook is composed of initial part with a circuit schematic, test current profile, and table summarising test criteria. This part is followed by package import instructions, display and the browser of historical HWC tests (until and including HWC 2018); see Figure below.
213

214
<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/raw/master/figures/swan-hwc-browser.png" width=75%>
215

216
217
218
There is a manual integration with AccTesting that requires a copy&paste of a text parameterising a notebook

<img src="https://gitlab.cern.ch/LHCData/lhc-sm-hwc/-/raw/master/figures/swan-manual-acctesting-integration.png">
219
220
221

The remainder of each notebook depends on the particular test to be performed. At the end of an HWC notebook there are instructions for saving the output files.

222
##### Notebook Output
223

224
The notebook creates an output in the folder (path with Windows convention)
225
```
226
\\cernbox-smb\eos\project\m\mp3\$circuit_type$\$circuit_name$\$hwc_test$\
227
228
229
```
e.g., 
```
230
\\cernbox-smb\eos\project\m\mp3\RB\RB.A12\PNO.b2\:
231
232
```

233
- HTML report file with the snapshot of the entire notebook - [circuit-name]_[test_name]-[test-start]-[analysis-execution-date].html;