cmsgemos-analysis merge requestshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests2022-06-15T14:10:30+02:00https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/1Feature/initial repository setup2022-06-15T14:10:30+02:00Mykhailo DalchenkoFeature/initial repository setupCloses #1
Details are provided in the underlying issue and in commit messagesCloses #1
Details are provided in the underlying issue and in commit messagesLaurent PetreCamilla GalloniLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/2Feature/unpacker skeleton2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/unpacker skeleton## Description
Detailed description with some implementation details has been presented at the [GEM DAQ SW meeting](https://indico.cern.ch/event/918874/contributions/3861739/attachments/2038025/3412756/test_unpacker.pdf)
The general stru...## Description
Detailed description with some implementation details has been presented at the [GEM DAQ SW meeting](https://indico.cern.ch/event/918874/contributions/3861739/attachments/2038025/3412756/test_unpacker.pdf)
The general structure remains the same, however important improvements were made:
- fixed bug with arbitrary hardware block ordering
- pandas.DataFrame conversion is not made once for all events instead of event-by-event (@lmoureau)
- `bitstring` package was replaced by `cbitstruct` (@lpetre)
- input data is read as `numpy.ndarray` (`fromfile` method)
- formatting strings (dictionaries) are now compiled
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #3, partially addresses #4 and #2
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested by running on verified test data set, consisting of 9306 events
### Test on `gem904daq01`
```shell
CPU info:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
C++ code:
time ./gemtreewriter/bin/linux/x86_64_centos7/unpacker /tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat sdram
[GEMUnpacker]: ---> Main()
real 0m0.704s
user 0m0.556s
sys 0m0.151s
python code:
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('/tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat','sdram')"
***
INFO:gdh.unpacker:Finished unpacking. Processed 9306 events.
1 loops, best of 3: 3.36 sec per loop
Jupiter notebook:
CPU times: user 3.15 s, sys: 78.1 ms, total: 3.23 s
Wall time: 3.22 s
```
### Test on MBP 13" Fall 2014
```
CPU info:
machdep.cpu.max_basic: 13
machdep.cpu.max_ext: 2147483656
machdep.cpu.vendor: GenuineIntel
machdep.cpu.brand_string: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
machdep.cpu.family: 6
machdep.cpu.model: 69
machdep.cpu.extmodel: 4
machdep.cpu.extfamily: 0
machdep.cpu.stepping: 1
python code:
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('/tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat','sdram')"
***
INFO:gdh.unpacker:Finished unpacking. Processed 9306 events.
1 loop, best of 5: 3.9 sec per loop
Jupiter notebook:
CPU times: user 3.66 s, sys: 122 ms, total: 3.78 s
Wall time: 3.84 s
```
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [x] All new and existing tests passed.Louis MoureauxLaurent PetreLouis Moureauxhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/3Feature/tuples as column names2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/tuples as column names## Description
<!--- Describe your changes in detail -->
This MR contains two commits, we have to decide which one we want to follow:
- one with partial implementation of column names tuplization: `('FIELD NAME', slot, link, position)`...## Description
<!--- Describe your changes in detail -->
This MR contains two commits, we have to decide which one we want to follow:
- one with partial implementation of column names tuplization: `('FIELD NAME', slot, link, position)`
- second with extension of the tuple including block id and sub block: `('FIELD NAME', 'BLOCK ID', 'SUB BLOCK', slot, link, position)`
Since the performance drop is not significant (and anyways it is an improvement over the f-strings), I'd go for the second option as this would greatly simplify the analysis. The end goal for analysis would be to convert the `pandas.DataFrame` returned by unpacker to `pandas.MultiIndex` and then it can easily be sliced or cross-sectioned to select required subset of data (e.g. select all VFATs, or all VFATs from AMC N GEB M etc.)
Note: the documentation will require an update. I do not include it here for two reasons:
- First we need to decide on the tuple complexity
- There will be separate MR on the documentation build
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #11
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested by unpacking a test dataset. Performance result on MBP 13" Fall 2014:
```shell
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('./doc/notebooks/test_data/test_sdram.dat','sdram')"
1 loop, best of 5: 3.39 sec per loop
```
For comparison, previous result was 3.9s
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Pieter EveraertsLouis MoureauxLaurent PetrePieter Everaertshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/4Feature/documentation build2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/documentation build## Description
<!--- Describe your changes in detail -->
Implements the documentation build system based on `sphinx` with read-the-docs theme.
Changelog:
- Provide `sphinx` configuration with required extensions
- Include markdown CONT...## Description
<!--- Describe your changes in detail -->
Implements the documentation build system based on `sphinx` with read-the-docs theme.
Changelog:
- Provide `sphinx` configuration with required extensions
- Include markdown CONTRIBUTING and DEVELOPERS files
- Note: toc tree and some links are messed up. Fix TBD
- problem: it is not possible to have working representation both for GitLab and for
`recommonmark` due to anchoring and referencing differencies. Currently keep `GitLab` anchoring.
- Include Jupyter notebook as user story
- Provide API doc and cleanup comments in the code
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #10
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
The documentation has been built in local virtual environment.
Uploaded for testing and looking around [here](https://mdalchen.web.cern.ch/gem-online-analysis/index.html)
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Pieter EveraertsLouis MoureauxLaurent PetrePieter Everaertshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/5Refactoring2022-06-15T14:10:34+02:00Mykhailo DalchenkoRefactoring## Description
<!--- Describe your changes in detail -->
After setting up basic CI with linting, need in refactoring became evident.
The code is brought in accordance with PEP8 standard and few improvements were also made,
notably lazy ...## Description
<!--- Describe your changes in detail -->
After setting up basic CI with linting, need in refactoring became evident.
The code is brought in accordance with PEP8 standard and few improvements were also made,
notably lazy logging is now used, so the logger should not bring too much overhead
(this should be visible with test bages). The CI file was pre-pushed to the develop in order to have convenient badges on the gitlab page. Sorry :blush:
If there're any comments to CI file, the can be addressed in the current MR,
however I'd consider it to be a pretty good basic template. As soon as eos website for service account will be created, the docs deployment job can be used.
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #12
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested both locally and with the CI. The pylint score changed from 4.06 to 9.82
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Louis MoureauxLaurent PetreCamilla GalloniLouis Moureauxhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/10Convert the repository to poetry2021-08-18T14:32:42+02:00Laurent PetreConvert the repository to poetry## Description
<!-- Describe your changes in detail -->
This MR converts the repository to `poetry` instead of `tito` for easier tracking of the dependencies and easier safe usage. The CI is also fixed and adapted. As an optimization, a...## Description
<!-- Describe your changes in detail -->
This MR converts the repository to `poetry` instead of `tito` for easier tracking of the dependencies and easier safe usage. The CI is also fixed and adapted. As an optimization, a custom CI Docker image is produced.
RPM creation will be added in a subsequent MR.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The repository can be used locally following the instruction in the developers' guide.
The CI is also fully working:
* A custom CI Docker image is produced on demand: https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/container_registry/10566
* The documentation is published (manual job for `master`): https://cmsgemos.web.cern.ch/analysis/doc/
* Python packages will be released in the GitLab PyPi repository (manual job for `master`): will appear in https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/packages
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/7S-bit rate analysis2022-02-22T16:02:46+01:00Anagha AravindS-bit rate analysis## Description
<!-- Describe your changes in detail -->
Analysis of S-bit rate vs THR_ARM_DAC scan.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, pl...## Description
<!-- Describe your changes in detail -->
Analysis of S-bit rate vs THR_ARM_DAC scan.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Anagha AravindAnagha Aravindhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/9DAC scan analysis2021-08-24T15:45:26+02:00Laurent PetreDAC scan analysis## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Anagha AravindAnagha Aravindhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/15Analysis suite template2021-08-26T18:07:51+02:00Laurent PetreAnalysis suite template## Description
<!-- Describe your changes in detail -->
This MR provides a template for the analysis suite under construction. The Python `gemanalysis` package has been created and is installed with its expected dependencies. The docume...## Description
<!-- Describe your changes in detail -->
This MR provides a template for the analysis suite under construction. The Python `gemanalysis` package has been created and is installed with its expected dependencies. The documentation also mentions PostgreSQL that is installed on the official GEM development machines.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Can be installed with the updated instructions.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/16Fix typo in DEVELOPERS.md instructions2021-08-27T18:51:25+02:00Camilla GalloniFix typo in DEVELOPERS.md instructions## Description
<!-- Describe your changes in detail -->
Merge request to fix a typo in the instructions.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change...## Description
<!-- Describe your changes in detail -->
Merge request to fix a typo in the instructions.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/17Create the script producing GBT configuration2021-09-13T17:11:23+02:00Laurent PetreCreate the script producing GBT configuration## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds a new script producing GBT configuration files to be uploaded to the back-end board based on the output of the GBT phase scan routine.
Depends on h...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds a new script producing GBT configuration files to be uploaded to the back-end board based on the output of the GBT phase scan routine.
Depends on https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/191.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
With `fed1478-slot02.cfg` as the output of the GBT phase scan routine, `gemos create-config gbt fed1478-slot02.cfg test-output` creates the expected files in the expected location:
```
test-output/
└── fed1478-slot2
├── config-oh0-gbt.cfg
├── config-oh1-gbt.cfg
└── config-oh2-gbt.cfg
```
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/20Retrieve VFAT calibration parameters from database2021-11-05T15:17:07+01:00Antonello PellecchiaRetrieve VFAT calibration parameters from database## Description
This MR adds to `gemos` a command that dumps to a CSV-like file the VFAT calibration parameters obtained from a query to the database.
Example of usage:
```
poetry run gemos poetry run gemos query vfat-calibration vfat-c...## Description
This MR adds to `gemos` a command that dumps to a CSV-like file the VFAT calibration parameters obtained from a query to the database.
Example of usage:
```
poetry run gemos poetry run gemos query vfat-calibration vfat-calibrations.dat
```
The file `vfat-calibrations.dat` is sanitize and contains the following information:
```
chip-id;vref-adc;iref;adc0-m;adc0-b;cal-dac-m;cal-dac-b
1230;3;29;1.9089;-324.593;-0.272049;67.5893
1231;3;39;1.89825;-318.341;-0.229325;56.4414
1232;3;29;1.98346;-325.632;-0.245428;61.0245
1233;3;33;1.87333;-307.779;-0.229754;56.3122
1234;3;35;1.90041;-317.408;-0.245102;60.4882
1235;3;27;1.94132;-319.831;-0.23473;57.6574
1236;3;33;1.9382;-320.212;-0.23664;58.5432
1237;3;35;1.96087;-326.629;-0.251636;62.2413
1238;3;31;1.92872;-329.099;-0.257939;63.8939
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #34.
## How Has This Been Tested?
Tested on some VFATs found lying around in the lab :)
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/21Add script to create VFAT configuration file from scan results2021-11-05T19:48:10+01:00Antonello PellecchiaAdd script to create VFAT configuration file from scan results## Description
Add an option to `gemos` to create VFAT configuration files. The command takes as input a folder with a well-known file structure* and produces as output a folder with the same file structure and updated parameters.
In t...## Description
Add an option to `gemos` to create VFAT configuration files. The command takes as input a folder with a well-known file structure* and produces as output a folder with the same file structure and updated parameters.
In the absence of an input file, a default configuration file can be used. This is typically used to bootstrap a new configuration based on the setup description.
The following parameters are currently supported:
* The VFAT calibration parameters retrieved from the database
* The results of the DAC scans
* The thresholds based on the results of the S-bit rate scans analysis routine
Usage example:
``` sh
gemos create-config vfat --default default.cfg -c calibration/fed0001-slot01.cfg -d dacScan/dac_Scans.cfg -t sBitRateScan/sbits_1.cfg -t sBitRateScan/sbits_2.cfg results/vfatConfig/
```
* The files will be organized in subfolders named `fed<xxxx>-slot<y>`.
## Related Issue
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #35.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested with results of scans performed on the VFATs available in the lab on a GE2/1 setup as well as the GE1/1 integration setup chambers.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/18Create script to analyze DAC scans2022-02-22T16:02:46+01:00Antonello PellecchiaCreate script to analyze DAC scans## Description
This MR adds an option to the `gemos` command that takes as an input the result of a DAC scan and creates an output file with the optimal DAC values to be set for each VFAT. Example of usage:
``` sh
poetry run gemos anal...## Description
This MR adds an option to the `gemos` command that takes as an input the result of a DAC scan and creates an output file with the optimal DAC values to be set for each VFAT. Example of usage:
``` sh
poetry run gemos analysis dac ~/daq/examples/gemdata/dacScan/fed1478-slot07.txt ~/daq/examples/results/dacScan/fed1478-slot07.cfg
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on the result of DAC scan performed on a GE2/1 chamber.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/23Optionally plot the DAC scans2022-02-22T16:13:50+01:00Camilla GalloniOptionally plot the DAC scans## Description
<!-- Describe your changes in detail -->
This aims at:
- improving the clarity of the options in gemos
- introduce the options not to plot the DAC scan results (now default)
- state in the warning for which VFATs the DAC s...## Description
<!-- Describe your changes in detail -->
This aims at:
- improving the clarity of the options in gemos
- introduce the options not to plot the DAC scan results (now default)
- state in the warning for which VFATs the DAC scan fit fails
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Coffin setup
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/19S-bit rate scan analysis2022-02-22T16:05:13+01:00Antonello PellecchiaS-bit rate scan analysis## Description
Add a command to `gemos` to analyze the results of the sbit rate scans. The command takes as input the files containing the result of the threshold scan (can be for different VFATs and different iterations) and the rate to...## Description
Add a command to `gemos` to analyze the results of the sbit rate scans. The command takes as input the files containing the result of the threshold scan (can be for different VFATs and different iterations) and the rate to be accepted to determine the VFAT threshold. The output file contains the thresholds in DAC units for each VFAT.
Example of usage:
```
poetry run gemos analyze sbit -r 100 /gemdata/sBitRateScan/fed1478-slot07.txt /gemdata/results/sBitRateScan/fed1478-slot07.cfg
```
Example of sbit scan plot:
![fed1478-slot7-oh1](/uploads/e2f84079948e55640a64c20e32234506/fed1478-slot7-oh1.png)
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
Tested on the results of a threshold scan taken on a GE2/1 module.
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/25Render DAC scans plot for VFATs with failing DAC by default2022-03-03T18:25:38+01:00Laurent PetreRender DAC scans plot for VFATs with failing DAC by default## Description
<!-- Describe your changes in detail -->
This MR allows plotting only VFAT with a failing DAC. The option `-p/--plotting` is modified to take any of `{always, never, warning}`
It was implemented in two steps:
* Refactor ...## Description
<!-- Describe your changes in detail -->
This MR allows plotting only VFAT with a failing DAC. The option `-p/--plotting` is modified to take any of `{always, never, warning}`
It was implemented in two steps:
* Refactor the DAC scans routine to split the analysis from the plotting
* Implement the conditional plotting option
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Fixes #38.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on the 72 VFAT of the GE1/1 integration setup. The chosen DAC values did not change. The plotting option behaves as expected.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/26Fix the DAC scan plotting of VFAT with warning2022-03-07T18:17:10+01:00Laurent PetreFix the DAC scan plotting of VFAT with warning## Description
<!-- Describe your changes in detail -->
This MR fixes bugs encountered running on DAC scans from p5 with multiple problematic VFAT and missing scans, namely:
* Fix a bug when the analysis was crashing if some VFAT from ...## Description
<!-- Describe your changes in detail -->
This MR fixes bugs encountered running on DAC scans from p5 with multiple problematic VFAT and missing scans, namely:
* Fix a bug when the analysis was crashing if some VFAT from the layout tree weren't included in the scan.
* Fix a bug leading to too many VFAT being plotted in "warning-only plotting" mode
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The code now runs on the most recent p5 dataset, plotting only the VFAT with warnings, without changing the result of the scan.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/27Add latency to create VFAT configuration files2022-03-09T14:52:56+01:00Camilla GalloniAdd latency to create VFAT configuration files## Description
<!-- Describe your changes in detail -->
Title says it all.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an iss...## Description
<!-- Describe your changes in detail -->
Title says it all.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Will be tested at p5.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/29Fix typo in VFAT config latency2022-03-10T13:08:21+01:00Camilla GalloniFix typo in VFAT config latency## Description
<!-- Describe your changes in detail -->
Fix silly typo
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue f...## Description
<!-- Describe your changes in detail -->
Fix silly typo
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Change already implemented in `gem-locdaq02.cms` at p5.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Camilla GalloniCamilla Galloni