Commit 52a93d2c authored by Aleksandra Mnich's avatar Aleksandra Mnich
Browse files

Merge branch 'dev' into 'master'

New release

See merge request !29
parents 575157ba c7a85c18
Pipeline #3163225 canceled with stages
in 11 minutes
variables:
KRB5CCNAME: "/tmp/krb5cc"
NAME: "gitlab-registry.cern.ch/db/spark-pipelines/sparknotebook"
test_folder: test
stages:
- test
- deploy_dev
- deploy
- execute-nxcals-notebook
deploy_dev_eos:
stage: deploy_dev
......@@ -20,6 +21,7 @@ deploy_dev_eos:
- mkdir public/
- cp -r qh public
- cp -r nqps public
- cp -r pgc public
- cp -r rb public
- cp -r rq public
- cp -r it public
......@@ -52,6 +54,7 @@ deploy_production_eos:
- mkdir public/
- cp -r qh public
- cp -r nqps public
- cp -r pgc public
- cp -r rb public
- cp -r rq public
- cp -r it public
......@@ -70,40 +73,31 @@ deploy_production_eos:
only:
- tags
execute_nxcals_notebook:
stage: execute-nxcals-notebook
test_papermill_nxcals:
image: python:3.8
stage: test
artifacts:
untracked: true
expire_in: 1 day
script:
# kinit to obtain kerberos credentials
- echo "$EOS_ACCOUNT_PASSWORD" | kinit -c $KRB5CCNAME "$EOS_ACCOUNT_USERNAME@CERN.CH"
# NXCALS configuration
- export LCG_VIEW=/cvmfs/sft.cern.ch/lcg/views/LCG_95apython3_nxcals/x86_64-centos7-gcc7-opt
- source $LCG_VIEW/setup.sh
- source /cvmfs/sft.cern.ch/lcg/etc/hadoop-confext/hadoop-swan-setconf.sh hadoop-nxcals
- export PYTHONPATH=/usr/local/lib/python3.6/site-packages/:$PYTHONPATH
- pip list
- pip install -t /tmp/ tzlocal
- pip install -t /tmp/ lhcsmapi
- pip install -t /tmp/ plotly
- pip install -t /tmp/ tqdm
- pip3 install -t /tmp/ -r ./papermill_nxcals/requirements.txt
- export PYTHONPATH=$PYTHONPATH:/tmp/
- export OUTPUT_DIR=/eos/project/m/mp3/public
- export NXCALS_JARS=$(ls $LCG_VIEW/nxcals/nxcals_java/* | xargs | sed -e 's/ /:/g')
- keytool -import -alias cerngridCA -file /etc/pki/tls/certs/CERN_Grid_Certification_Authority.crt -keystore /tmp/truststore.jks -storepass password -noprompt
- keytool -import -alias cernRootCA2 -file /etc/pki/tls/certs/CERN_Root_Certification_Authority_2.crt -keystore /tmp/truststore.jks -storepass password -noprompt
# Execute the notebook, convert to pdf and copy the pdf to EOS folder
- echo $CLUSTER
- echo $NOTEBOOK_NAME
- echo $CIRCUIT_NAME
- echo $START_TIME
- echo $END_TIME
- python3 execute_notebook.py $NOTEBOOK_NAME $CIRCUIT_NAME $START_TIME $END_TIME
image: "gitlab-registry.cern.ch/db/spark-pipelines/sparknotebook:v1.0-python36"
only:
refs:
- schedules
- web
- triggers
variables:
- $CLUSTER == "hadoop-nxcals"
- python3 -m pytest ./papermill_nxcals/${test_folder} --junitxml=report.xml
tags:
- docker
notebooks_exec:
image: gitlab-registry.cern.ch/db/swan-spark-notebooks:latest
stage: test
artifacts:
when: on_failure
paths:
- report.xml
- results
script:
- bash CI/run_notebooks.sh
tags:
- docker
only:
- pipelines
- web
\ No newline at end of file
#!/bin/bash
### kinit to obtain kerberos credentials
echo "$EOS_ACCOUNT_PASSWORD" | kinit -c $KRB5CCNAME "$EOS_ACCOUNT_USERNAME@CERN.CH"
### NXCALS configuration
export LCG_VIEW=/cvmfs/sft.cern.ch/lcg/views/LCG_100_nxcals/x86_64-centos7-gcc9-opt
source $LCG_VIEW/setup.sh
alias python="/cvmfs/sft.cern.ch/lcg/views/LCG_100_nxcals/x86_64-centos7-gcc9-opt/bin/python"
alias python3="/cvmfs/sft.cern.ch/lcg/views/LCG_100_nxcals/x86_64-centos7-gcc9-opt/bin/python3"
alias pip="/cvmfs/sft.cern.ch/lcg/views/LCG_100_nxcals/x86_64-centos7-gcc9-opt/bin/pip3.8"
alias pytest="/cvmfs/sft.cern.ch/lcg/views/LCG_100_nxcals/x86_64-centos7-gcc9-opt/bin/pytest"
source /cvmfs/sft.cern.ch/lcg/etc/hadoop-confext/hadoop-swan-setconf.sh hadoop-nxcals
export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip:$PYTHONPATH
export NXCALS_JARS=$(ls $LCG_VIEW/nxcals/nxcals_java/* | xargs | sed -e 's/ /:/g')
### install dependencies
pip list
export PYTHONPATH=/tmp/:$PYTHONPATH
pip install -t /tmp/ papermill
pip install -t /tmp/ ./papermill_nxcals/
git clone --single-branch --branch "${API_BRANCH:-$CI_COMMIT_BRANCH}" https://:@gitlab.cern.ch:8443/LHCData/lhc-sm-api.git
pip install -t /tmp/ ./lhc-sm-api/
### copy HWC_Summary.csv
mkdir -p /eos/project/l/lhcsm/hwc/
xrdcp root://eosuser.cern.ch//eos/project/l/lhcsm/hwc/HWC_Summary.csv /eos/project/l/lhcsm/hwc/HWC_Summary.csv
### create dir for executed notebooks
mkdir ./results
### run notebooks
python -m pytest test --junitxml=report.xml
\ No newline at end of file
__version__ = "1.5.64"
__version__ = "1.5.65"
......@@ -196,17 +196,18 @@
" results_table = query.create_report_analysis_template(timestamp_fgc=timestamp_fgc, init_file_path='../__init__.py', author=author)\n",
" analysis = IpdCircuitAnalysis(query.circuit_type, results_table)\n",
" \n",
" timestamp_dct = {'FGC': timestamp_fgc, 'PIC': timestamp_pic, \n",
" 'QDS_A':source_timestamp_qds_df.loc[0, 'timestamp'] if len(source_timestamp_qds_df) > 0 else float('nan'), \n",
" 'QDS_B':source_timestamp_qds_df.loc[1, 'timestamp'] if len(source_timestamp_qds_df) > 1 else float('nan')}\n"
" 'QDS_B':source_timestamp_qds_df.loc[1, 'timestamp'] if len(source_timestamp_qds_df) > 1 else float('nan')}\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false
 
 
"source": [
"# 3. Timestamps\n",
"The analysis for MP3 consists of checking the existence of PM file and of consistency of the PM timestamps (PC, QPS). The criterion of passing this test described in detail in 600APIC2.\n",
"In short the following criteria should be checked:\n",
"- The PC timestamp (51_self) is QPS timestamp +-20 ms. \n",
......@@ -270,13 +271,10 @@
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"deletable": false
"deletable": false
},
"outputs": [],
},
"outputs": [],
"source": [
"t_quench = analysis.find_start_end_quench_detection(u_res_df)[0]\n",
"t_quench_correction = analysis.plot_i_meas_pc_with_cursor(circuit_name, timestamp_fgc, [i_meas_df, i_a_df, i_ref_df], t_quench)"
......
......@@ -110,11 +110,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -315,25 +315,25 @@
"- If pure inductive signal and di/dt < 0:\n",
" - U_1_B1 = L di/dt < 0\n",
" - U_2_B1 = -L di/dt < 0\n",
" \n",
"- PM file\n",
]
" - Buffer range 501 to 1500, event at point 1000\n",
" - Time range: -2 to 2 s\n",
" - Frequency: 250 Hz (dt = 4 ms)"
]
"execution_count": null,
 
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
]
 
"from lhcsmapi.gui.pc.fgc_pm_event_select.IpdFgcPmEventSelectBaseModule import IpdFgcPmEventSelectBaseModule\n",
"IpdFgcPmEventSelectBaseModule('IPD').display_qps_circuit_schematic(circuit_name)"
]
"metadata": {
 
{
"cell_type": "markdown",
"metadata": {
"deletable": false
},
......
......@@ -117,11 +117,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -292,25 +292,25 @@
"# 5. Quench Protection System\n",
"\n",
"The signal names used for quench detection are shown in the figure below (*please run a cell below to display a QPS circuit schematic corresponding to the circuit name under analysis*).\n",
"\n",
"**Quench Detector Type** \n",
"**Current Leads:**\n",
"DQQDC - current leads quench detector \n",
"DQAMG - controller attached to global protection \n",
"\n",
"**Current Leads:**\n",
"- PM file\n",
"- Typical resistance for U_RES: 7 uOhm\n",
"- Threshold for U_HTS: 3 mV, 1s\n",
"- Polarity convention: Arrows show how signals are measured. If I > 0, LD1: U_RES > 0, LD2: U_RES < 0\n",
"- PM file\n",
" - Buffer range 0 to 250, event at point 50\n",
" - Time range: -10 to 40 s\n",
"- See polarity convention in the circuit schematics\n",
" - Frequency: 5 Hz (dt = 200 ms)\n",
" \n",
"**Magnet:**\n",
"- See polarity convention in the circuit schematics\n",
"- Signals are measured with -2.5 V offset and with the gain factor = 0.0012\n",
"- U_RES_B1 = U_1_B1 + U_2_B1\n",
"- Threshold on U_RES_B1: 100 mV, 10 ms\n",
"- U_RES_B2, U_1_B2, U_2_B2 and U_INDUCT_B2 are given for diagnostics only\n",
"- Signals are measured with -2.5 V offset and with the gain factor = 0.0012\n",
"- *Attention: B1 signals and B2 singals can be shifted by 4 ms from each other*\n",
"- If pure inductive signal and di/dt < 0:\n",
......
......@@ -118,11 +118,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -421,25 +421,25 @@
]
},
{
"cell_type": "code",
"execution_count": null,
"from lhcsmapi.gui.pc.fgc_pm_event_select.IpdFgcPmEventSelectBaseModule import IpdFgcPmEventSelectBaseModule\n",
 
"outputs": [],
"source": [
"from lhcsmapi.gui.pc.fgc_pm_event_select.IpdFgcPmEventSelectBaseModule import IpdFgcPmEventSelectBaseModule\n",
{
"IpdFgcPmEventSelectBaseModule('IPD').display_qps_circuit_schematic(circuit_name)"
]
},
{
"cell_type": "markdown",
"metadata": {
"## 5.1. Resistive Voltage\n",
 
},
"source": [
"## 5.1. Resistive Voltage\n",
"- Origin of a quench, i.e., name of the first magnet for which the U_RES voltage exceeded the 100 mV threshold.\n",
 
"*ANALYSIS*: \n",
"- Initial voltage slope of U_RES signal. The slope is calculated as a ratio of the voltage change from 50 to 200 mV and the corresponding time change.\n",
"- Origin of a quench, i.e., name of the first magnet for which the U_RES voltage exceeded the 100 mV threshold.\n",
"\n",
"*GRAPHS*: \n",
......
......@@ -119,11 +119,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -340,25 +340,25 @@
" - Buffer range 0 to 250, event at point 50\n",
" - Time range: -10 to 40 s\n",
" - Frequency: 5 Hz (dt = 200 ms)\n",
" \n",
"**Magnet:**\n",
"- U_RES_B2, U_1_B2, U_2_B2 and U_INDUCT_B2 are given for diagnostics only\n",
"- See polarity convention in the circuit schematics\n",
"- U_RES_B1 = U_1_B1 + U_2_B1\n",
"- Threshold on U_RES_B1: 100 mV, 10 ms\n",
"- U_RES_B2, U_1_B2, U_2_B2 and U_INDUCT_B2 are given for diagnostics only\n",
" - U_1_B1 = L di/dt < 0\n",
"- Signals are measured with -2.5 V offset and with the gain factor = 0.0012\n",
"- *Attention: B1 signals and B2 singals can be shifted by 4 ms from each other*\n",
"- If pure inductive signal and di/dt < 0:\n",
" - U_1_B1 = L di/dt < 0\n",
" - U_2_B1 = -L di/dt < 0\n",
" \n",
" - Frequency: 250 Hz (dt = 4 ms)"
 
" - Buffer range 501 to 1500, event at point 1000\n",
" - Time range: -2 to 2 s\n",
" - Frequency: 250 Hz (dt = 4 ms)"
"cell_type": "code",
 
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
......
......@@ -121,11 +121,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -349,25 +349,25 @@
"cell_type": "markdown",
"metadata": {
"deletable": false
},
"source": [
"\n",
"## 5.1. Resistive Voltage\n",
"\n",
"*GRAPHS*: \n",
"\n",
{
"- t = 0 s corresponds to the start of the test"
]
},
{
"cell_type": "code",
"execution_count": null,
"outputs": [],
 
"deletable": false
},
"outputs": [],
},
 
"analysis.plot_u_res(circuit_name, timestamp_fgc, u_res_df, i_meas_df)"
]
},
{
"cell_type": "markdown",
......
......@@ -126,11 +126,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -351,25 +351,25 @@
{
"cell_type": "markdown",
"metadata": {
"deletable": false
},
"*GRAPHS*: \n",
 
"## 5.1. Resistive Voltage\n",
"\n",
"*GRAPHS*: \n",
},
 
"- t = 0 s corresponds to the start of the test"
]
},
{
"cell_type": "code",
},
 
"metadata": {
"deletable": false
},
]
 
"source": [
"analysis.plot_u_res(circuit_name, timestamp_fgc, u_res_df, i_meas_df)"
]
},
{
......
......@@ -202,14 +202,17 @@
]
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false
},
"source": [
},
"source": [
"# 3. Timestamps\n",
"The analysis for MP3 consists of checking the existence of PM file and of consistency of the PM timestamps (PC, QPS). \n",
"In short the following criteria should be checked:\n",
"- The PC timestamp (51_self) is QPS timestamp +-20 ms. \n",
"- The difference between QPS board A and B timestamp = 1ms.\n",
"\n",
"If one or more of these conditions are not fulfilled, then an in-depth analysis has to be performed by the QPS team."
]
},
......@@ -218,14 +221,17 @@
"execution_count": null,
"metadata": {
"deletable": false
},
"outputs": [],
"source": [
"ipq_analysis.create_timestamp_table(timestamp_dct)"
]
},
"outputs": [],
},
{
"cell_type": "markdown",
"metadata": {
"deletable": false
},
"source": [
"# 4. PC\n",
"## 4.1. Main Current\n",
......
......@@ -104,11 +104,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -340,25 +340,25 @@
"execution_count": null,
"metadata": {
"deletable": false
},
"outputs": [],
"\n",
 
"ipq_analysis.plot_i_meas_pc(circuit_name+'B2', timestamp_fgc, [i_meas_b2_df, i_a_b2_df, i_ref_b2_df],\n",
" xlim=(-0.1, 0.2), ylim=(i_meas_b2_df.max().values[0]-1500, i_meas_b2_df.max().values[0]+500))\n",
"\n",
{
"ipq_analysis.calculate_current_slope(i_meas_b2_df.rename(columns={i_meas_b2_df.columns[0]: 'STATUS.I_MEAS'}), col_name=['Ramp Rate B2', 'Plateau Duration B2'])"
]
},
{
"cell_type": "markdown",
"metadata": {
"## 4.2. Earth Current\n",
 
},
"source": [
"## 4.2. Earth Current\n",
"\n",
 
"*ANALYSIS*: \n",
"- calculation of the maximum absolute value of the earth current\n",
"\n",
"*GRAPHS*: \n",
"- t = 0 s corresponds to the FGC timestamp\n",
......
......@@ -94,11 +94,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -269,25 +269,25 @@
"deletable": false
},
"outputs": [],
"source": [
"import matplotlib as mpl\n",
"ipq_analysis.plot_i_meas_pc(circuit_name, timestamp_fgc, [i_meas_b1_df.rename(columns={i_meas_b1_df.columns[0]: i_meas_b1_df.columns[0]+'_B1'}), \n",
"mpl.rcParams['savefig.dpi'] = 80\n",
"mpl.rcParams['figure.dpi'] = 80\n",
"%matplotlib notebook\n",
"ipq_analysis.plot_i_meas_pc(circuit_name, timestamp_fgc, [i_meas_b1_df.rename(columns={i_meas_b1_df.columns[0]: i_meas_b1_df.columns[0]+'_B1'}), \n",
]
" i_meas_b2_df.rename(columns={i_meas_b2_df.columns[0]: i_meas_b2_df.columns[0]+'_B2'}), \n",
" i_ref_b1_df.rename(columns={i_ref_b1_df.columns[0]: i_ref_b1_df.columns[0]+'_B1'}), \n",
" i_ref_b2_df.rename(columns={i_ref_b1_df.columns[0]: i_ref_b2_df.columns[0]+'_B2'})])"
]
},
{
"deletable": false,
 
"execution_count": null,
"metadata": {
"deletable": false,
"source": [
 
},
"outputs": [],
"source": [
"tau_ipq = ipq_analysis.calculate_i_meas_tau([i_meas_b1_df, i_meas_b2_df], duration_decay=(0, 50))\n",
"\n",
......
......@@ -99,11 +99,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -334,25 +334,25 @@
" - Frequency: 250Hz (dt = 4ms)"
]
},
{
"cell_type": "code",
"source": [
 
"metadata": {},
"outputs": [],
"source": [
},
"from lhcsmapi.gui.pc.fgc_pm_event_select.IpqFgcPmEventSelectBaseModule import IpqFgcPmEventSelectBaseModule\n",
"IpqFgcPmEventSelectBaseModule('IPQ').display_qps_circuit_schematic(circuit_name)"
]
},
{
"cell_type": "markdown",
"source": [
 
"deletable": false
},
"source": [
"- Check if R_max < 5 nOhm **Due to limited voltage resolution of logged signals this check is only possible with ELQA measurements**\n",
"## 5.1. Splice Resistance\n",
"\n",
"*CRITERIA*\n",
"- Check if R_max < 5 nOhm **Due to limited voltage resolution of logged signals this check is only possible with ELQA measurements**\n",
"\n",
"*GRAPHS*: \n",
......
......@@ -98,11 +98,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",
"1. Copy code from AccTesting and paste into an empty cell below\n",
......@@ -458,25 +458,25 @@
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
},
"from lhcsmapi.gui.pc.fgc_pm_event_select.IpqFgcPmEventSelectBaseModule import IpqFgcPmEventSelectBaseModule\n",
"IpqFgcPmEventSelectBaseModule('IPQ').display_qps_circuit_schematic(circuit_name)"
]
},
"deletable": false
 
"cell_type": "markdown",
"metadata": {
"deletable": false
},
"source": [
"- Initial voltage slope of U_RES signal. The slope is calculated as a ratio of the voltage change from 50 to 200 mV and the corresponding time change.\n",
"## 5.1. Resistive Voltage\n",
"\n",
"*ANALYSIS*: \n",
"- Initial voltage slope of U_RES signal. The slope is calculated as a ratio of the voltage change from 50 to 200 mV and the corresponding time change.\n",
"\n",
"- Origin of a quench, i.e., name of the first magnet for which the U_RES voltage exceeded the 100 mV threshold.\n",
"\n",
"*GRAPHS*: \n",
"\n",
"- t = 0 s corresponds to the FGC timestamp\n",
"- detected slope of U_RES is depicted with an orange line in plots with U_RES, U_1, U_2"
......
......@@ -97,11 +97,11 @@
"print('Analysis performed by %s' % HwcSearchModuleMediator.get_user())"
]
},
{
"cell_type": "markdown",
"metadata": {
 
"deletable": false
},
"source": [
"# 1. User Input\n",