cmsgemos-analysis issueshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues2023-06-23T12:53:54+02:00https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/55Identify poor vfat data in S-curve analysis2023-06-23T12:53:54+02:00Daniel EstradaIdentify poor vfat data in S-curve analysis## Description
The S-curve analysis version developed on !47 can identify when a VFAT just has zeros and label it as an empty vfat. But, there are some cases where vfat is not completely powered off but neither has good detections (like ...## Description
The S-curve analysis version developed on !47 can identify when a VFAT just has zeros and label it as an empty vfat. But, there are some cases where vfat is not completely powered off but neither has good detections (like in the images). It should be necessary to identify and label them (maybe to mask them). Probably those cases are responsible to produce the warning message `OptimizeWarning: Covariance of the parameters could not be estimated` which at the moment is being ignored.
## Possible fixes
I think that some callback produced by the warning message could be used to identify problematic VFATs or channels. Another option could be, to think if it would be possible to use the new histogram added in the S-curves ROOT file producer (the number of events per point) to catch such cases.
<!-- If you can, link to the line of code that might be responsible for the problem -->
## screenshots
![5](/uploads/e93843e05a12fb5341d29157b28f22d4/5.png)Daniel EstradaDaniel Estradahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/23Document the unpacker output2020-09-24T16:38:21+02:00Louis MoureauxDocument the unpacker output### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The output of the unpacker should be documented so it can actually be used. There is currently no documentation.
...### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The output of the unpacker should be documented so it can actually be used. There is currently no documentation.
The documentation should at least describe the overall structure, list the most important fields and provide a short explanation. Links to other manuals with a more in depth description would be useful.
### What is the expected correct behavior?
<!--- What you should see instead -->
A documentation page exists which describes the contents of the DataFrame produced by the unpacker.
### Relevant logs and/or screenshots
<!--- Paste any relevant logs - please use code blocks (```) to format -->
<!--- console output, logs, and code as it's very hard to read otherwise -->
This is the list of columns generated when parsing the example file in the repository:
```
[('AMC N', 'AMC', 'HEADER', 5, None, None),
('AMC:5:AMC N', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:AMC SIZE', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:B', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:BLK SEQ N', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:BOARD ID', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:C', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:E', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:L', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:M', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:P', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:PAD', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:S', 'AMC13', 'PAYLOAD', None, None, None),
('AMC:5:V', 'AMC13', 'PAYLOAD', None, None, None),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('BC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('BCL', 'AMC', 'TRAILER', 5, None, None),
('BLK N', 'AMC13', 'TRAILER', None, None, None),
('BOARD ID', 'AMC', 'HEADER', 5, None, None),
('BOE N', 'CDF', 'HEADER', None, None, None),
('BP', 'AMC', 'TRAILER', 5, None, None),
('BUF STATUS', 'AMC', 'HEADER', 5, None, None),
('BX ID', 'AMC13', 'TRAILER', None, None, None),
('BX ID', 'CDF', 'HEADER', None, None, None),
('BX MM AV', 'GEB', 'HEADER', 5, 0, None),
('BX MM VV', 'GEB', 'HEADER', 5, 0, None),
('BX N', 'AMC', 'HEADER', 5, None, None),
('C', 'CDF', 'TRAILER', None, None, None),
('CAL CHAN', 'GEB', 'HEADER', 5, 0, None),
('CAL TYPE', 'AMC13', 'HEADER', None, None, None),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 0),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 1),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 2),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 3),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 8),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 9),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 10),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 11),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 16),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 17),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 18),
('CHANNEL DATA L', 'VFAT', 'PAYLOAD', 5, 0, 19),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 0),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 1),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 2),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 3),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 8),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 9),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 10),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 11),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 16),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 17),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 18),
('CHANNEL DATA M', 'VFAT', 'PAYLOAD', 5, 0, 19),
('CL', 'AMC', 'TRAILER', 5, None, None),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('CRC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('CRC16', 'CDF', 'TRAILER', None, None, None),
('CRC32', 'AMC', 'TRAILER', 5, None, None),
('CRC32', 'AMC13', 'TRAILER', None, None, None),
('DATA LGTH', 'AMC', 'HEADER', 5, None, None),
('DATA LGTH', 'AMC', 'TRAILER', 5, None, None),
('DAV CNT', 'AMC', 'HEADER', 5, None, None),
('DAV LIST', 'AMC', 'HEADER', 5, None, None),
('DR', 'AMC', 'TRAILER', 5, None, None),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('EC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('EOE N', 'CDF', 'TRAILER', None, None, None),
('EVT F', 'GEB', 'HEADER', 5, 0, None),
('EVT LGTH', 'CDF', 'TRAILER', None, None, None),
('EVT NF', 'GEB', 'HEADER', 5, 0, None),
('EVT STATUS', 'CDF', 'TRAILER', None, None, None),
('EVT SZ OFW', 'GEB', 'HEADER', 5, 0, None),
('EVT SZ WARN', 'GEB', 'HEADER', 5, 0, None),
('EVT TYPE', 'CDF', 'HEADER', None, None, None),
('EVT TYPE', 'CDF', 'TRAILER', None, None, None),
('EVT UFW', 'GEB', 'TRAILER', 5, 0, None),
('F', 'CDF', 'TRAILER', None, None, None),
('FOV', 'CDF', 'HEADER', None, None, None),
('FV', 'AMC', 'HEADER', 5, None, None),
('H', 'CDF', 'HEADER', None, None, None),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 0),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 1),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 2),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 3),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 8),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 9),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 10),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 11),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 16),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 17),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 18),
('HEADER', 'VFAT', 'PAYLOAD', 5, 0, 19),
('IN F', 'GEB', 'HEADER', 5, 0, None),
('IN NF', 'GEB', 'HEADER', 5, 0, None),
('IN UFW', 'GEB', 'TRAILER', 5, 0, None),
('INPUT ID', 'GEB', 'HEADER', 5, 0, None),
('L1A F', 'GEB', 'HEADER', 5, 0, None),
('L1A ID', 'AMC', 'HEADER', 5, None, None),
('L1A ID', 'AMC', 'TRAILER', 5, None, None),
('L1A ID', 'AMC13', 'TRAILER', None, None, None),
('L1A ID', 'CDF', 'HEADER', None, None, None),
('L1A NF', 'GEB', 'HEADER', 5, 0, None),
('LINK TO', 'AMC', 'TRAILER', 5, None, None),
('ML', 'AMC', 'TRAILER', 5, None, None),
('N AMC', 'AMC13', 'HEADER', None, None, None),
('N VFAT WORDS', 'GEB', 'HEADER', 5, 0, None),
('N VFAT WORDS', 'GEB', 'TRAILER', 5, 0, None),
('NO VFAT MARK', 'GEB', 'HEADER', 5, 0, None),
('OOS', 'AMC', 'TRAILER', 5, None, None),
('OOS AV', 'GEB', 'HEADER', 5, 0, None),
('OOS VV', 'GEB', 'HEADER', 5, 0, None),
('OR N', 'AMC', 'HEADER', 5, None, None),
('ORBIT N', 'AMC13', 'HEADER', None, None, None),
('PAD', 'AMC', 'HEADER', 5, None, None),
('PAD', 'AMC13', 'HEADER', None, None, None),
('PAD', 'AMC13', 'TRAILER', None, None, None),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 0),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 1),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 2),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 3),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 8),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 9),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 10),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 11),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 16),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 17),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 18),
('PAD', 'VFAT', 'PAYLOAD', 5, 0, 19),
('PAD1', 'AMC', 'TRAILER', 5, None, None),
('PAD1', 'GEB', 'HEADER', 5, 0, None),
('PAD1', 'GEB', 'TRAILER', 5, 0, None),
('PAD2', 'AMC', 'TRAILER', 5, None, None),
('PAD2', 'GEB', 'HEADER', 5, 0, None),
('PAD2', 'GEB', 'TRAILER', 5, 0, None),
('PAD3', 'AMC', 'TRAILER', 5, None, None),
('PAYLOAD TYPE', 'AMC', 'HEADER', 5, None, None),
('PAYLOAD VAR', 'AMC', 'HEADER', 5, None, None),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 0),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 1),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 2),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 3),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 8),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 9),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 10),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 11),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 16),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 17),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 18),
('POS', 'VFAT', 'PAYLOAD', 5, 0, 19),
('R', 'CDF', 'TRAILER', None, None, None),
('RES(0)', 'AMC13', 'HEADER', None, None, None),
('RP:LATENCY', 'AMC', 'HEADER', 5, None, None),
('RP:PAD', 'AMC', 'HEADER', 5, None, None),
('RP:PULSE STRETCH', 'AMC', 'HEADER', 5, None, None),
('RUN TYPE', 'AMC', 'HEADER', 5, None, None),
('S', 'CDF', 'HEADER', None, None, None),
('S', 'CDF', 'TRAILER', None, None, None),
('SRC ID', 'CDF', 'HEADER', None, None, None),
('STUCK DATA', 'GEB', 'TRAILER', 5, 0, None),
('T', 'CDF', 'TRAILER', None, None, None),
('TTS', 'AMC', 'HEADER', 5, None, None),
('TTS', 'CDF', 'TRAILER', None, None, None),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 0),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 1),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 2),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 3),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 8),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 9),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 10),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 11),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 16),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 17),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 18),
('VC', 'VFAT', 'PAYLOAD', 5, 0, 19),
('X', 'CDF', 'HEADER', None, None, None),
('X', 'CDF', 'TRAILER', None, None, None),
('uFOV', 'AMC13', 'HEADER', None, None, None)]
```
<!--- ### Possible fixes -->
<!--- If you can, provide a code snippet which may implement the requested feature -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/21Unpack several files in sequence2020-09-27T00:49:34+02:00Louis MoureauxUnpack several files in sequence### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The unpacker is designed to read one file at a time, but scans will produce more than one file. Handling any numb...### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
The unpacker is designed to read one file at a time, but scans will produce more than one file. Handling any number of files should be as transparent as possible.
### What is the expected correct behavior?
<!--- What you should see instead -->
Any operation can be performed with only one or with several input files.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/9Setup efficient unpacker logging2020-05-18T14:52:19+02:00Mykhailo DalchenkoSetup efficient unpacker logging### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
Unpacker uses built-in python `logger` module.
While this module is very handy in setting up different levels of v...### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
Unpacker uses built-in python `logger` module.
While this module is very handy in setting up different levels of verbosity and support different handlers (console, file etc.),
it comes with a significant overhead even for silenced calls (i.e. if one has multiple `debug` statements in the code, parsing them may slow down the code very significantly even when logging level is set to `info`).
### What is the expected correct behavior?
<!--- What you should see instead -->
Need to provide proper console and file handlers for logger and optimize log calls.
Pehaps provide logging functionality on per-event basis to reduce the number of `logger.debug` statements.
### Relevant logs and/or screenshots
<!--- Paste any relevant logs - please use code blocks (```) to format -->
<!--- console output, logs, and code as it's very hard to read otherwise -->
<!--- ### Possible fixes -->
<!--- If you can, provide a code snippet which may implement the requested feature -->https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/8Implement CRC checks on unpacked data2020-05-18T13:36:01+02:00Mykhailo DalchenkoImplement CRC checks on unpacked data### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
Currently unpacker doesn't provide any data integrity checks.
CRCs are provided at AMC13, AMC and VFAT levels and ...### Summary
<!--- Summarize the new feature you would like to add concisely -->
<!--- and describe the actual behavior -->
Currently unpacker doesn't provide any data integrity checks.
CRCs are provided at AMC13, AMC and VFAT levels and have to be implemented in the software too.
### Relevant logs and/or screenshots
<!--- Paste any relevant logs - please use code blocks (```) to format -->
<!--- console output, logs, and code as it's very hard to read otherwise -->
<!--- ### Possible fixes -->
<!--- If you can, provide a code snippet which may implement the requested feature -->