cmsgemos-analysis issueshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues2023-06-23T12:53:54+02:00https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/55Identify poor vfat data in S-curve analysis2023-06-23T12:53:54+02:00Daniel EstradaIdentify poor vfat data in S-curve analysis## Description
The S-curve analysis version developed on !47 can identify when a VFAT just has zeros and label it as an empty vfat. But, there are some cases where vfat is not completely powered off but neither has good detections (like ...## Description
The S-curve analysis version developed on !47 can identify when a VFAT just has zeros and label it as an empty vfat. But, there are some cases where vfat is not completely powered off but neither has good detections (like in the images). It should be necessary to identify and label them (maybe to mask them). Probably those cases are responsible to produce the warning message `OptimizeWarning: Covariance of the parameters could not be estimated` which at the moment is being ignored.
## Possible fixes
I think that some callback produced by the warning message could be used to identify problematic VFATs or channels. Another option could be, to think if it would be possible to use the new histogram added in the S-curves ROOT file producer (the number of events per point) to catch such cases.
<!-- If you can, link to the line of code that might be responsible for the problem -->
## screenshots
![5](/uploads/e93843e05a12fb5341d29157b28f22d4/5.png)Daniel EstradaDaniel Estradahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/54Parallelize S-curves analysis accross intput files2023-04-06T15:43:31+02:00Laurent PetreParallelize S-curves analysis accross intput files## Description
<!-- Describe the bug encountered or feature you want to add -->
While reviewing !47, it appears that the proposed S-curves analysis sequence does not parallelize across multiple input files (see https://gitlab.cern.ch/cm...## Description
<!-- Describe the bug encountered or feature you want to add -->
While reviewing !47, it appears that the proposed S-curves analysis sequence does not parallelize across multiple input files (see https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/47#note_6580289). Such an enhancement could be implemented with the goal to fasten up, even more, the analysis routine!
<!-- ## Steps to reproduce -->
<!--- How one can reproduce the issue - this is very important -->
<!-- ## Possible fixes -->
<!-- If you can, link to the line of code that might be responsible for the problem -->
<!-- ## Relevant logs and/or screenshots -->
<!-- Paste any relevant logs - please use code blocks (```) to format -->
<!-- console output, logs, and code as it's very hard to read otherwise -->Daniel EstradaDaniel Estradahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/53Use ROOT histograms for the threshold scan analysis2023-07-31T16:06:59+02:00Laurent PetreUse ROOT histograms for the threshold scan analysis## Description
<!-- Describe the bug encountered or feature you want to add -->
As the title says and as discussed in private, the threshold scan analysis should take ROOT histograms (and thus ROOT files) as input instead of the current...## Description
<!-- Describe the bug encountered or feature you want to add -->
As the title says and as discussed in private, the threshold scan analysis should take ROOT histograms (and thus ROOT files) as input instead of the current CSV-like file. This is similar to what is being done for the S-curves in https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/47.
Getting a reasonable input dataset is strictly dependent on https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/312 - without a proper fix, threshold scans cannot be taken at P5 - and partially dependent on https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/305 - the MR does work but should be finalized and merged in.
<!-- ## Steps to reproduce -->
<!--- How one can reproduce the issue - this is very important -->
<!-- ## Possible fixes -->
<!-- If you can, link to the line of code that might be responsible for the problem -->
<!-- ## Relevant logs and/or screenshots -->
<!-- Paste any relevant logs - please use code blocks (```) to format -->
<!-- console output, logs, and code as it's very hard to read otherwise -->Daniel EstradaDaniel Estradahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/49Better identify failed DAC scans2022-06-15T14:23:09+02:00Laurent PetreBetter identify failed DAC scans## Description
<!-- Describe the bug encountered or feature you want to add -->
In the current DAC scan analysis routine, no fitting of the points is performed. While not strictly required for the DAC scan analysis itself, fitting the r...## Description
<!-- Describe the bug encountered or feature you want to add -->
In the current DAC scan analysis routine, no fitting of the points is performed. While not strictly required for the DAC scan analysis itself, fitting the raw data can help to spot problematic DAC scans.
<!-- ## Steps to reproduce -->
<!--- How one can reproduce the issue - this is very important -->
## Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem -->
Fit the DAC scans and use the success or failure and the quality of fit to spot problematic DAC scans.
<!-- ## Relevant logs and/or screenshots -->
<!-- Paste any relevant logs - please use code blocks (```) to format -->
<!-- console output, logs, and code as it's very hard to read otherwise -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/48Convert the analysis suite storage to ROOT files2022-06-15T14:06:53+02:00Laurent PetreConvert the analysis suite storage to ROOT files## Description
<!-- Describe the bug encountered or feature you want to add -->
As discussed in https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/issues/184#note_5417332 (a follow-up from https://gitlab.cern.ch/cmsgemonline/gem-daq...## Description
<!-- Describe the bug encountered or feature you want to add -->
As discussed in https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/issues/184#note_5417332 (a follow-up from https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/6), we want to start using the **ROOT file format** (but not the ROOT framework) for the storage of our data, including the calibration scans.
This issue is here to keep track of the implementation.
<!-- ## Steps to reproduce -->
<!--- How one can reproduce the issue - this is very important -->
## Possible fixes
<!-- If you can, link to the line of code that might be responsible for the problem -->
There is the desire not to include and use the ROOT framework, including `PyROOT`, but modern and safer libraries such as `pandas` and `uproot`. Loading/saving a Pandas Dataframe from/to a ROOT file is a matter of a few lines of Python only.
* [ ] Create a command that "imports" the calibration scan output into the analysis realm, i.e. aggregate N `.dat` CSV-like files into a unique ROOT file
* [ ] Use ROOT files in place of the CSV-like files in the various existing analysis routines
<!-- ## Relevant logs and/or screenshots -->
<!-- Paste any relevant logs - please use code blocks (```) to format -->
<!-- console output, logs, and code as it's very hard to read otherwise -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/40Check if input and auxiliary files exist2022-02-22T18:41:47+01:00Camilla GalloniCheck if input and auxiliary files exist## Description
<!-- Describe the bug encountered or feature you want to add -->
At the moment there is no error/warning thrown by gemos if one of the files that are optionally givable doesn't exist during the create-config step.
The use...## Description
<!-- Describe the bug encountered or feature you want to add -->
At the moment there is no error/warning thrown by gemos if one of the files that are optionally givable doesn't exist during the create-config step.
The user should be notified.
<!-- ## Steps to reproduce -->
<!--- How one can reproduce the issue - this is very important -->
gemos create-config vfat -m vfat-map-2022021801.dat -c vfat-calibration-db-2022021801.dat --default default-vfat.cfg -d /cmsnfsgemdata/gemdata/data/dac-scan-out-2022022201/dac-values.dat -t /cmsnfsgemdata/gemdata/data/sbit-rate-scan-out-100hz-2022022201/threshold-values.dat vfat/2022022202 vfat/2022022203
<!-- ## Possible fixes -->
<!-- If you can, link to the line of code that might be responsible for the problem -->
<!-- ## Relevant logs and/or screenshots -->
<!-- Paste any relevant logs - please use code blocks (```) to format -->
<!-- console output, logs, and code as it's very hard to read otherwise -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/33Give an uncertainty on the thresholds2021-08-30T13:58:23+02:00Laurent PetreGive an uncertainty on the thresholds## Description
<!-- Describe the bug encountered or feature you want to add -->
It is known in GE1/1 that the noise level changes when the VFAT run mode is toggled. As a consequence, the threshold to be applied to the front-end for a gi...## Description
<!-- Describe the bug encountered or feature you want to add -->
It is known in GE1/1 that the noise level changes when the VFAT run mode is toggled. As a consequence, the threshold to be applied to the front-end for a given noise level changes between runs. This can be used as the uncertainty on the threshold. Such parameter should be provided as the output of the S-bit rate scan analysis.
<!-- ## Steps to reproduce -->
<!--- How one can reproduce the issue - this is very important -->
<!-- ## Possible fixes -->
<!-- If you can, link to the line of code that might be responsible for the problem -->
<!-- ## Relevant logs and/or screenshots -->
<!-- Paste any relevant logs - please use code blocks (```) to format -->
<!-- console output, logs, and code as it's very hard to read otherwise -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/32Structure the analysis scripts2021-08-24T19:02:02+02:00Laurent PetreStructure the analysis scripts### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
At the moment, we are using a disparate set of analysis scripts with different data formats and behaviors. We should uniformize tho...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
At the moment, we are using a disparate set of analysis scripts with different data formats and behaviors. We should uniformize those in order to simplify their usage and their port to the analysis suite.
The proposal is the following:
* All ~~scripts~~ executables do only one task and well (KISS philosophy). A typical example is the S-bit rate analysis that can be factored in the following steps:
1. Raw results -> output thresholds (with an option to define the allowed noise levels)
2. Plotting of the raw results (with an option to add the derived thresholds in the plot)
3. Convert the output thresholds into VFAT configuration files
These 3 ~~scripts~~ executables being able to run independently and linked together via Bash commands/scripts.
* The Python scripts have the following structure to help in the port to the analysis framework when the time will come:
```python
def my_helper_function_1():
pass
def my_helper_function_2():
pass
def my_main_function():
pass
def my_tool_1():
# Create an argument parser
# Call the functions with the right parameters
def my_tool_2():
# Create an argument parser
# Call the functions with the right parameters
# EDIT
# if __name__ == "__main__":
# # Create an argument parser
# # Call the functions with the right parameters
```
The Python functions are converted into executable programs with the right import in the `pyproject.toml` file. This is an intermidate solution until a better and more uniform CLI system is provided.
* The input and output paths are given as arguments to the scripts and are not inferred based on the location of the scripts themselves or other files. This allows to easily run on the same data with different parameters.
* The manipulated files are, at the moment, CSV files, ideally GZIP compressed. They should be manipulated with `pandas` in case the dataframe storage format would change (HDF5?).
* We need to agree on a delimiter. A reduced list of potential characters is `;`, `:`, `|`.
* **`;` has been chosen by @cgalloni in `cmsgemos`, use it in absence of other proposals**
* A line in the data file must be self-consistent. In the S-bit rate example, a line in the output file is enough to figure out to which VFAT in the whole system a threshold must be applied. In this case, it means that the `fed`, `slot`, `optohybrid` and `vfat` must all be defined.
@aaravind @cgallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/30Use case: channel loss monitoring2020-11-10T20:27:27+01:00Laurent PetreUse case: channel loss monitoring### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
The S-curves allow extracting the noise of a given VFAT channel. When a VFAT channel is damaged after a propagating discharge, its ...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
The S-curves allow extracting the noise of a given VFAT channel. When a VFAT channel is damaged after a propagating discharge, its noise level changes significantly, it either drops close to 0 fC or increases. One must be able to detect changes in noise across runs and automatically detect input channel damages.
Typically, a S-curve scan is taken daily and compared with the one from the previous `n` days. Then, summary plots are provided for extend periods of time that can span over years.
### What is the expected correct behavior?
<!--- What you should see instead -->
It must be possible to automatically compare the noise levels between S-curves and detect damaged channels. The evolution of the number of dead channels over long periods of time must be easily accessible to the end user.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/29Use case: compare analysis results between releases2020-11-10T20:20:35+01:00Laurent PetreUse case: compare analysis results between releases### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
With software improvements, data will be re-analyzed, for example, to provide more accurate results. One must be able to easily com...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
With software improvements, data will be re-analyzed, for example, to provide more accurate results. One must be able to easily compare the results of different analysises to understand the software improvements as well as changes in the interpretation of past data.
### What is the expected correct behavior?
<!--- What you should see instead -->
The old analysis outputs should be stored (not overwritten) and comparable through relevant tools.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/28Use case: noise comparison2020-11-10T20:27:28+01:00Laurent PetreUse case: noise comparison### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
Optimizing the detector configuration to reduce the electronics noise is crucial to maximize the detection efficency. If the S-curv...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
Optimizing the detector configuration to reduce the electronics noise is crucial to maximize the detection efficency. If the S-curves can be used to extract the electronics noise of a given VFAT channel, we need a tool that allows to compare different configurations. The comparison must be doable for `n` chambers with `m` sets of S-curves.
For instance, in the legacy software, the following comparison plots can be produced for every chamber.
![image](/uploads/abbe80a0882c6e0dc2feb6a5bf20e5cf/image.png)
### What is the expected correct behavior?
<!--- What you should see instead -->
It must be possible to compare S-curves (but probably also other scans, such as S-bit rate scans). For convenience, it should be possible to assosciate one (or more) symbolic name to every run (i.e. dataset).https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/27Tensorboard vs Matplotlib for data visualization2020-10-01T14:39:40+02:00Pranjal SharmaTensorboard vs Matplotlib for data visualization### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
Tensorboard is a data visualization tool by Google mostly used for machine learning but may have some interesting use cases for us. ...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
Tensorboard is a data visualization tool by Google mostly used for machine learning but may have some interesting use cases for us. But is it worth adding as a new dependency?
### Description
<!--- What you should see instead -->
Currently, we only use Matplotlib for data visualization since our analysis is mostly limited to static (histograms). But in the future, we may need more visualization power where Tensorboard's ability to scale and smooth, overlay and dynamically refresh may come in handy. It currently supports scalars, graphs (tensors), audio and image data whereas Matplotlib has a more comprehensive plotting arsenal as long as one sticks to static analysis.
One interesting use case would be to use both in conjunction, utilizing Tensorboard's dashboard as a UI with Matplotlib plots.
<!--- Paste any relevant logs - please use code blocks (```) to format -->
<!--- console output, logs, and code as it's very hard to read otherwise -->Pranjal SharmaPranjal Sharmahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/26Data are unpacked as float2020-09-28T14:16:25+02:00Louis MoureauxData are unpacked as float### Summary
<!--- Summarize the bug encountered concisely -->
The DataFrame generated by the unpacker out of `/afs/cern.ch/work/c/cgalloni/public/latency_scan/run000000_Testing_CERN904_2020-06-05_chunk_0.dat` has the following `float` c...### Summary
<!--- Summarize the bug encountered concisely -->
The DataFrame generated by the unpacker out of `/afs/cern.ch/work/c/cgalloni/public/latency_scan/run000000_Testing_CERN904_2020-06-05_chunk_0.dat` has the following `float` columns:
```
('RP:EXT TRG', 'AMC', 'HEADER', 2, None, None)
('RP:IS C', 'AMC', 'HEADER', 2, None, None)
('RP:CAL DAC', 'AMC', 'HEADER', 2, None, None)
('RP:PULSE STRETCH', 'AMC', 'HEADER', 2, None, None)
('RP:LATENCY', 'AMC', 'HEADER', 2, None, None)
```
They contain, among others, `NaN`s.
### Steps to reproduce
<!--- How one can reproduce the issue - this is very important -->
```py
from gdh import unpacker
import numpy as np
df = unpacker.run('/afs/cern.ch/work/c/cgalloni/public/latency_scan/run000000_Testing_CERN904_2020-06-05_chunk_0.dat','sdram')
for c in df.columns:
if np.dtype(df[c]) == np.float64:
print(c)
```
### What is the expected correct behavior?
<!--- What you should see instead -->
All columns are integers.
### Environment
- Version used: latest `develop`
- Operation System: CentOS 7
<!--- ### Possible fixes -->
<!--- If you can, link to the line of code that might be responsible for the problem -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/24Latency scans2023-04-19T13:54:42+02:00Louis MoureauxLatency scans### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
This is a split of #5 for ~~internal~~ latency scans. The intent is to come to a specification that can be implemented.
### Initial...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
This is a split of #5 for ~~internal~~ latency scans. The intent is to come to a specification that can be implemented.
### Initial description
This is what I inferred from @anlevin's description in #5. It may be incorrect.
The internal latency scan determines which entry in the VFAT3 internal buffer should be fetched when a L1A is received. Data is generated by pulsing the VFAT (one channel only? all channels?) at some rate (what's the value?). Every pulse is set to last 4 bunch crossings (why?). L1A's are sent back for every L1 trigger sent by the VFAT, and the data stored in the VFAT buffer at the value of the `Latency` register (what's the real name?) are sent back to the AMC and stored to disk.
The procedure is repeated for every possible value of the `Latency` register. The correct value of the `Latency` can be inferred from a number-of-hits-seen vs latency histogram like this:
![image](/uploads/6bcd28bc72f9b3cc287ae6dec0ffbd4c/image.png)
The plateau is 4 bins wide because every bin corresponds to possible value of the `Latency`, which is specified as a number of bunch crossings, and the pulse is kept high for 4 bunch crossings. The first bin with a large number of hits corresponds to the "best" latency setting.
### Corner cases
The plateau above is not sharp. This creates an ambiguity for the best value of the latency: it can be the bin at 31 or the bin at 32.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/23Document the unpacker output2020-09-24T16:38:21+02:00Louis MoureauxDocument the unpacker output### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The output of the unpacker should be documented so it can actually be used. There is currently no documentation.
...### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The output of the unpacker should be documented so it can actually be used. There is currently no documentation.
The documentation should at least describe the overall structure, list the most important fields and provide a short explanation. Links to other manuals with a more in depth description would be useful.
### What is the expected correct behavior?
<!--- What you should see instead -->
A documentation page exists which describes the contents of the DataFrame produced by the unpacker.
### Relevant logs and/or screenshots
<!--- Paste any relevant logs - please use code blocks (```) to format -->
<!--- console output, logs, and code as it's very hard to read otherwise -->
This is the list of columns generated when parsing the example file in the repository:
```
[('AMC N', 'AMC', 'HEADER', 5, None, None),
('AMC:5:AMC N', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:AMC SIZE', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:B', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:BLK SEQ N', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:BOARD ID', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:C', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:E', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:L', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:M', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:P', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:PAD', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:S', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:V', 'AMC13', 'PAYLOAD', None, None, None),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('BCL', 'AMC', 'TRAILER', 5, None, None),
('BLK N', 'AMC13', 'TRAILER', None, None, None),
('BOARD ID', 'AMC', 'HEADER', 5, None, None),
('BOE N', 'CDF', 'HEADER', None, None, None),
('BP', 'AMC', 'TRAILER', 5, None, None),
('BUF STATUS', 'AMC', 'HEADER', 5, None, None),
('BX ID', 'AMC13', 'TRAILER', None, None, None),
('BX ID', 'CDF', 'HEADER', None, None, None),
('BX MM AV', 'GEB', 'HEADER', 5, 0, None),
('BX MM VV', 'GEB', 'HEADER', 5, 0, None),
('BX N', 'AMC', 'HEADER', 5, None, None),
('C', 'CDF', 'TRAILER', None, None, None),
('CAL CHAN', 'GEB', 'HEADER', 5, 0, None),
('CAL TYPE', 'AMC13', 'HEADER', None, None, None),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 0),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 1),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 2),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 3),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 8),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 9),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 10),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 11),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 16),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 17),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 18),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 19),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 0),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 1),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 2),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 3),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 8),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 9),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 10),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 11),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 16),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 17),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 18),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 19),
('CL', 'AMC', 'TRAILER', 5, None, None),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('CRC16', 'CDF', 'TRAILER', None, None, None),
('CRC32', 'AMC', 'TRAILER', 5, None, None),
('CRC32', 'AMC13', 'TRAILER', None, None, None),
('DATA LGTH', 'AMC', 'HEADER', 5, None, None),
('DATA LGTH', 'AMC', 'TRAILER', 5, None, None),
('DAV CNT', 'AMC', 'HEADER', 5, None, None),
('DAV LIST', 'AMC', 'HEADER', 5, None, None),
('DR', 'AMC', 'TRAILER', 5, None, None),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('EOE N', 'CDF', 'TRAILER', None, None, None),
('EVT F', 'GEB', 'HEADER', 5, 0, None),
('EVT LGTH', 'CDF', 'TRAILER', None, None, None),
('EVT NF', 'GEB', 'HEADER', 5, 0, None),
('EVT STATUS', 'CDF', 'TRAILER', None, None, None),
('EVT SZ OFW', 'GEB', 'HEADER', 5, 0, None),
('EVT SZ WARN', 'GEB', 'HEADER', 5, 0, None),
('EVT TYPE', 'CDF', 'HEADER', None, None, None),
('EVT TYPE', 'CDF', 'TRAILER', None, None, None),
('EVT UFW', 'GEB', 'TRAILER', 5, 0, None),
('F', 'CDF', 'TRAILER', None, None, None),
('FOV', 'CDF', 'HEADER', None, None, None),
('FV', 'AMC', 'HEADER', 5, None, None),
('H', 'CDF', 'HEADER', None, None, None),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 0),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 1),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 2),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 3),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 8),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 9),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 10),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 11),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 16),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 17),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 18),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 19),
('IN F', 'GEB', 'HEADER', 5, 0, None),
('IN NF', 'GEB', 'HEADER', 5, 0, None),
('IN UFW', 'GEB', 'TRAILER', 5, 0, None),
('INPUT ID', 'GEB', 'HEADER', 5, 0, None),
('L1A F', 'GEB', 'HEADER', 5, 0, None),
('L1A ID', 'AMC', 'HEADER', 5, None, None),
('L1A ID', 'AMC', 'TRAILER', 5, None, None),
('L1A ID', 'AMC13', 'TRAILER', None, None, None),
('L1A ID', 'CDF', 'HEADER', None, None, None),
('L1A NF', 'GEB', 'HEADER', 5, 0, None),
('LINK TO', 'AMC', 'TRAILER', 5, None, None),
('ML', 'AMC', 'TRAILER', 5, None, None),
('N AMC', 'AMC13', 'HEADER', None, None, None),
('N VFAT WORDS', 'GEB', 'HEADER', 5, 0, None),
('N VFAT WORDS', 'GEB', 'TRAILER', 5, 0, None),
('NO VFAT MARK', 'GEB', 'HEADER', 5, 0, None),
('OOS', 'AMC', 'TRAILER', 5, None, None),
('OOS AV', 'GEB', 'HEADER', 5, 0, None),
('OOS VV', 'GEB', 'HEADER', 5, 0, None),
('OR N', 'AMC', 'HEADER', 5, None, None),
('ORBIT N', 'AMC13', 'HEADER', None, None, None),
('PAD', 'AMC', 'HEADER', 5, None, None),
('PAD', 'AMC13', 'HEADER', None, None, None),
('PAD', 'AMC13', 'TRAILER', None, None, None),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 0),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 1),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 2),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 3),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 8),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 9),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 10),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 11),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 16),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 17),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 18),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 19),
('PAD1', 'AMC', 'TRAILER', 5, None, None),
('PAD1', 'GEB', 'HEADER', 5, 0, None),
('PAD1', 'GEB', 'TRAILER', 5, 0, None),
('PAD2', 'AMC', 'TRAILER', 5, None, None),
('PAD2', 'GEB', 'HEADER', 5, 0, None),
('PAD2', 'GEB', 'TRAILER', 5, 0, None),
('PAD3', 'AMC', 'TRAILER', 5, None, None),
('PAYLOAD TYPE', 'AMC', 'HEADER', 5, None, None),
('PAYLOAD VAR', 'AMC', 'HEADER', 5, None, None),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 0),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 1),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 2),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 3),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 8),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 9),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 10),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 11),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 16),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 17),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 18),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 19),
('R', 'CDF', 'TRAILER', None, None, None),
('RES(0)', 'AMC13', 'HEADER', None, None, None),
('RP:LATENCY', 'AMC', 'HEADER', 5, None, None),
('RP:PAD', 'AMC', 'HEADER', 5, None, None),
('RP:PULSE STRETCH', 'AMC', 'HEADER', 5, None, None),
('RUN TYPE', 'AMC', 'HEADER', 5, None, None),
('S', 'CDF', 'HEADER', None, None, None),
('S', 'CDF', 'TRAILER', None, None, None),
('SRC ID', 'CDF', 'HEADER', None, None, None),
('STUCK DATA', 'GEB', 'TRAILER', 5, 0, None),
('T', 'CDF', 'TRAILER', None, None, None),
('TTS', 'AMC', 'HEADER', 5, None, None),
('TTS', 'CDF', 'TRAILER', None, None, None),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('X', 'CDF', 'HEADER', None, None, None),
('X', 'CDF', 'TRAILER', None, None, None),
('uFOV', 'AMC13', 'HEADER', None, None, None)]
```
<!--- ### Possible fixes -->
<!--- If you can, provide a code snippet which may implement the requested feature -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/22Overall architecture2020-09-17T17:27:33+02:00Louis MoureauxOverall architecture### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
We need to settle on the architecture of the analysis suite.
On the input side, we will need to handle several GB-level files for s...### Summary
<!--- Summarize what you would like to discuss -->
<!--- and what is the actual behavior -->
We need to settle on the architecture of the analysis suite.
On the input side, we will need to handle several GB-level files for scans using tracking data (latency, S-curves...), or one (?) small file generated by [`cmsgemos`](gitlab.cern.ch/cmsgemonline/cmsgemos) for non-tracking data scans (can't find an example on top of my head).
On the output side, we will output either plots or text files to be digested by `cmsgemos`. The largest such file would contain the thresholds for all 128 channels of every VFAT. That's about half a million integers, which would produce a ~2MB JSON file.
### Proposed solution
It would be valuable to divide the analysis in two steps:
1. Histograms are filled from the contents of the tracking data files. They are saved to disk.
2. A post-processing step is applied to histograms to extract the parameters of physics interest. Ideally the dirty bits of reading histograms from disks are abstracted away.
This two-steps process would facilitate iterations during development, because the histogram files could be used many times without unpacking everything again. Care should be taken that the analysis applied in step 2 uses the correct type of histograms...
There should probably also be a convention on where to put plots (and corresponding helper functions), since we'll likely be able to produce thousands of them.
This architecture is what I drafted in [my slides](https://indico.cern.ch/event/955893/contributions/4016985/attachments/2104359/3539104/Moureaux_WelcomeAndIntro.pdf) today:
![image](/uploads/54f56555b17dfb9d889b1494a73c7d6d/image.png)
### What is the expected correct behavior?
<!--- What you should see instead -->
We have a well-defined architecture that we can move forward implementing!https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/21Unpack several files in sequence2020-09-27T00:49:34+02:00Louis MoureauxUnpack several files in sequence### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The unpacker is designed to read one file at a time, but scans will produce more than one file. Handling any numb...### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The unpacker is designed to read one file at a time, but scans will produce more than one file. Handling any number of files should be as transparent as possible.
### What is the expected correct behavior?
<!--- What you should see instead -->
Any operation can be performed with only one or with several input files.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/18Asymmetric error bar calculation2020-06-03T17:04:39+02:00Andrew Michael LevinAsymmetric error bar calculationFollowing up on https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/issues/15#note_3503273, we should try to use statistically meaningful error bars when handling scurve and possibly other types of data. In ROOT there is a dedicate...Following up on https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/issues/15#note_3503273, we should try to use statistically meaningful error bars when handling scurve and possibly other types of data. In ROOT there is a dedicated class
https://root.cern.ch/doc/master/classTEfficiency.html
with many options (which are probably not relevant for our purpose).
From a quick search of scipy
https://docs.scipy.org/doc/scipy/reference/search.html?q=efficiency&check_keywords=yes&area=default
I don't see anything similar, but it could of course be there without "efficiency" in the name.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/17Running over old data in CI2022-06-14T15:49:03+02:00Andrew Michael LevinRunning over old data in CIFollowing up on these posts
https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/merge_requests/5#note_3505827
https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/merge_requests/5#note_3505829
I would like to discuss how we ...Following up on these posts
https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/merge_requests/5#note_3505827
https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/merge_requests/5#note_3505829
I would like to discuss how we can add old data to a common place that the CI job will have access to. Obviously, we want to run over small examples, like a single VFAT. I think the main options are eos, afs, git lfs, mounting our own filesystem, or just storing the data directly in our gitlab repository.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/14Fitting in new framework2022-06-14T15:46:42+02:00Andrew Michael LevinFitting in new frameworkFor performing the scurve and DAC scan fitting in the new framework, scipy is the default. If there other suggestions please write them here. I will try to make some performance comparisons between scipy and ROOT.For performing the scurve and DAC scan fitting in the new framework, scipy is the default. If there other suggestions please write them here. I will try to make some performance comparisons between scipy and ROOT.