cmsgemos-analysis merge requestshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests2022-06-15T14:10:30+02:00https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/1Feature/initial repository setup2022-06-15T14:10:30+02:00Mykhailo DalchenkoFeature/initial repository setupCloses #1
Details are provided in the underlying issue and in commit messagesCloses #1
Details are provided in the underlying issue and in commit messagesLaurent PetreCamilla GalloniLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/2Feature/unpacker skeleton2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/unpacker skeleton## Description
Detailed description with some implementation details has been presented at the [GEM DAQ SW meeting](https://indico.cern.ch/event/918874/contributions/3861739/attachments/2038025/3412756/test_unpacker.pdf)
The general stru...## Description
Detailed description with some implementation details has been presented at the [GEM DAQ SW meeting](https://indico.cern.ch/event/918874/contributions/3861739/attachments/2038025/3412756/test_unpacker.pdf)
The general structure remains the same, however important improvements were made:
- fixed bug with arbitrary hardware block ordering
- pandas.DataFrame conversion is not made once for all events instead of event-by-event (@lmoureau)
- `bitstring` package was replaced by `cbitstruct` (@lpetre)
- input data is read as `numpy.ndarray` (`fromfile` method)
- formatting strings (dictionaries) are now compiled
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #3, partially addresses #4 and #2
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested by running on verified test data set, consisting of 9306 events
### Test on `gem904daq01`
```shell
CPU info:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
C++ code:
time ./gemtreewriter/bin/linux/x86_64_centos7/unpacker /tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat sdram
[GEMUnpacker]: ---> Main()
real 0m0.704s
user 0m0.556s
sys 0m0.151s
python code:
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('/tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat','sdram')"
***
INFO:gdh.unpacker:Finished unpacking. Processed 9306 events.
1 loops, best of 3: 3.36 sec per loop
Jupiter notebook:
CPU times: user 3.15 s, sys: 78.1 ms, total: 3.23 s
Wall time: 3.22 s
```
### Test on MBP 13" Fall 2014
```
CPU info:
machdep.cpu.max_basic: 13
machdep.cpu.max_ext: 2147483656
machdep.cpu.vendor: GenuineIntel
machdep.cpu.brand_string: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
machdep.cpu.family: 6
machdep.cpu.model: 69
machdep.cpu.extmodel: 4
machdep.cpu.extfamily: 0
machdep.cpu.stepping: 1
python code:
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('/tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat','sdram')"
***
INFO:gdh.unpacker:Finished unpacking. Processed 9306 events.
1 loop, best of 5: 3.9 sec per loop
Jupiter notebook:
CPU times: user 3.66 s, sys: 122 ms, total: 3.78 s
Wall time: 3.84 s
```
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [x] All new and existing tests passed.Louis MoureauxLaurent PetreLouis Moureauxhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/3Feature/tuples as column names2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/tuples as column names## Description
<!--- Describe your changes in detail -->
This MR contains two commits, we have to decide which one we want to follow:
- one with partial implementation of column names tuplization: `('FIELD NAME', slot, link, position)`...## Description
<!--- Describe your changes in detail -->
This MR contains two commits, we have to decide which one we want to follow:
- one with partial implementation of column names tuplization: `('FIELD NAME', slot, link, position)`
- second with extension of the tuple including block id and sub block: `('FIELD NAME', 'BLOCK ID', 'SUB BLOCK', slot, link, position)`
Since the performance drop is not significant (and anyways it is an improvement over the f-strings), I'd go for the second option as this would greatly simplify the analysis. The end goal for analysis would be to convert the `pandas.DataFrame` returned by unpacker to `pandas.MultiIndex` and then it can easily be sliced or cross-sectioned to select required subset of data (e.g. select all VFATs, or all VFATs from AMC N GEB M etc.)
Note: the documentation will require an update. I do not include it here for two reasons:
- First we need to decide on the tuple complexity
- There will be separate MR on the documentation build
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #11
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested by unpacking a test dataset. Performance result on MBP 13" Fall 2014:
```shell
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('./doc/notebooks/test_data/test_sdram.dat','sdram')"
1 loop, best of 5: 3.39 sec per loop
```
For comparison, previous result was 3.9s
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Pieter EveraertsLouis MoureauxLaurent PetrePieter Everaertshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/4Feature/documentation build2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/documentation build## Description
<!--- Describe your changes in detail -->
Implements the documentation build system based on `sphinx` with read-the-docs theme.
Changelog:
- Provide `sphinx` configuration with required extensions
- Include markdown CONT...## Description
<!--- Describe your changes in detail -->
Implements the documentation build system based on `sphinx` with read-the-docs theme.
Changelog:
- Provide `sphinx` configuration with required extensions
- Include markdown CONTRIBUTING and DEVELOPERS files
- Note: toc tree and some links are messed up. Fix TBD
- problem: it is not possible to have working representation both for GitLab and for
`recommonmark` due to anchoring and referencing differencies. Currently keep `GitLab` anchoring.
- Include Jupyter notebook as user story
- Provide API doc and cleanup comments in the code
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #10
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
The documentation has been built in local virtual environment.
Uploaded for testing and looking around [here](https://mdalchen.web.cern.ch/gem-online-analysis/index.html)
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Pieter EveraertsLouis MoureauxLaurent PetrePieter Everaertshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/5Refactoring2022-06-15T14:10:34+02:00Mykhailo DalchenkoRefactoring## Description
<!--- Describe your changes in detail -->
After setting up basic CI with linting, need in refactoring became evident.
The code is brought in accordance with PEP8 standard and few improvements were also made,
notably lazy ...## Description
<!--- Describe your changes in detail -->
After setting up basic CI with linting, need in refactoring became evident.
The code is brought in accordance with PEP8 standard and few improvements were also made,
notably lazy logging is now used, so the logger should not bring too much overhead
(this should be visible with test bages). The CI file was pre-pushed to the develop in order to have convenient badges on the gitlab page. Sorry :blush:
If there're any comments to CI file, the can be addressed in the current MR,
however I'd consider it to be a pretty good basic template. As soon as eos website for service account will be created, the docs deployment job can be used.
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #12
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested both locally and with the CI. The pylint score changed from 4.06 to 9.82
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Louis MoureauxLaurent PetreCamilla GalloniLouis Moureauxhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/6Draft: Analysis suite framework2021-11-01T10:48:14+01:00Pranjal SharmaDraft: Analysis suite framework## Description
This PR if for the implementation of the Analysis Suite which framework provides a robust way to handle results of GEM hardware calibration procedures and data. It provides a way to:
- analyze all kind of calibration runs...## Description
This PR if for the implementation of the Analysis Suite which framework provides a robust way to handle results of GEM hardware calibration procedures and data. It provides a way to:
- analyze all kind of calibration runs taken;
- combine different analyses together if needed;
- derive the best settings to improve the detector efficiency and performance;
- allow to evaluate the detector health and performances, at any given point in time or as it evolved over time, through the usage and - - display of the data, i.e. results of the scans and comparisons.
More details can be found here: https://codimd.web.cern.ch/lDdIGeVZR_C28XUNE4h90g?view
<!--- Describe your changes in detail -->
What's New:
- Added a data Importer which takes raw data and stores it after some preliminary processing
- Data Importer currently supports jpg, txt, csv and json file formats
- data_importer.py : the master data importer file which leverages specialized importers (json_loader, text_loader etc) for reading files and processing. Gradually, the specialized loaders can be improved upon to add more and more functionality
- tasks/ : contains all law tasks
- law.cfg : configuration file for law
- config.yaml : configuration file for the analysis_suite
- analysis_catalog/ : will house python functions for specialized physics-based analysis of the data (histograms)
- data/ : stores the pre-processed files generated from the data_importer
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Pranjal SharmaPranjal Sharmahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/7S-bit rate analysis2022-02-22T16:02:46+01:00Anagha AravindS-bit rate analysis## Description
<!-- Describe your changes in detail -->
Analysis of S-bit rate vs THR_ARM_DAC scan.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, pl...## Description
<!-- Describe your changes in detail -->
Analysis of S-bit rate vs THR_ARM_DAC scan.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Anagha AravindAnagha Aravindhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/8Draft: Latency scan analysis2021-08-24T15:49:44+02:00Itana BubanjaDraft: Latency scan analysisItana BubanjaItana Bubanjahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/9DAC scan analysis2021-08-24T15:45:26+02:00Laurent PetreDAC scan analysis## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Anagha AravindAnagha Aravindhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/10Convert the repository to poetry2021-08-18T14:32:42+02:00Laurent PetreConvert the repository to poetry## Description
<!-- Describe your changes in detail -->
This MR converts the repository to `poetry` instead of `tito` for easier tracking of the dependencies and easier safe usage. The CI is also fixed and adapted. As an optimization, a...## Description
<!-- Describe your changes in detail -->
This MR converts the repository to `poetry` instead of `tito` for easier tracking of the dependencies and easier safe usage. The CI is also fixed and adapted. As an optimization, a custom CI Docker image is produced.
RPM creation will be added in a subsequent MR.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The repository can be used locally following the instruction in the developers' guide.
The CI is also fully working:
* A custom CI Docker image is produced on demand: https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/container_registry/10566
* The documentation is published (manual job for `master`): https://cmsgemos.web.cern.ch/analysis/doc/
* Python packages will be released in the GitLab PyPi repository (manual job for `master`): will appear in https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/packages
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/11Feature/analysis suite rebase2021-08-26T15:38:00+02:00Pranjal SharmaFeature/analysis suite rebase## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/12Draft: Feature/analysis suite new2021-08-25T18:29:32+02:00Pranjal SharmaDraft: Feature/analysis suite new## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/13Feature/analysis suite new2021-08-25T20:19:06+02:00Pranjal SharmaFeature/analysis suite new## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/14Feature/analsysis suite for rebase2021-08-26T15:40:05+02:00Pranjal SharmaFeature/analsysis suite for rebase## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/15Analysis suite template2021-08-26T18:07:51+02:00Laurent PetreAnalysis suite template## Description
<!-- Describe your changes in detail -->
This MR provides a template for the analysis suite under construction. The Python `gemanalysis` package has been created and is installed with its expected dependencies. The docume...## Description
<!-- Describe your changes in detail -->
This MR provides a template for the analysis suite under construction. The Python `gemanalysis` package has been created and is installed with its expected dependencies. The documentation also mentions PostgreSQL that is installed on the official GEM development machines.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Can be installed with the updated instructions.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/16Fix typo in DEVELOPERS.md instructions2021-08-27T18:51:25+02:00Camilla GalloniFix typo in DEVELOPERS.md instructions## Description
<!-- Describe your changes in detail -->
Merge request to fix a typo in the instructions.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change...## Description
<!-- Describe your changes in detail -->
Merge request to fix a typo in the instructions.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/17Create the script producing GBT configuration2021-09-13T17:11:23+02:00Laurent PetreCreate the script producing GBT configuration## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds a new script producing GBT configuration files to be uploaded to the back-end board based on the output of the GBT phase scan routine.
Depends on h...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds a new script producing GBT configuration files to be uploaded to the back-end board based on the output of the GBT phase scan routine.
Depends on https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/191.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
With `fed1478-slot02.cfg` as the output of the GBT phase scan routine, `gemos create-config gbt fed1478-slot02.cfg test-output` creates the expected files in the expected location:
```
test-output/
└── fed1478-slot2
├── config-oh0-gbt.cfg
├── config-oh1-gbt.cfg
└── config-oh2-gbt.cfg
```
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/18Create script to analyze DAC scans2022-02-22T16:02:46+01:00Antonello PellecchiaCreate script to analyze DAC scans## Description
This MR adds an option to the `gemos` command that takes as an input the result of a DAC scan and creates an output file with the optimal DAC values to be set for each VFAT. Example of usage:
``` sh
poetry run gemos anal...## Description
This MR adds an option to the `gemos` command that takes as an input the result of a DAC scan and creates an output file with the optimal DAC values to be set for each VFAT. Example of usage:
``` sh
poetry run gemos analysis dac ~/daq/examples/gemdata/dacScan/fed1478-slot07.txt ~/daq/examples/results/dacScan/fed1478-slot07.cfg
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on the result of DAC scan performed on a GE2/1 chamber.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/19S-bit rate scan analysis2022-02-22T16:05:13+01:00Antonello PellecchiaS-bit rate scan analysis## Description
Add a command to `gemos` to analyze the results of the sbit rate scans. The command takes as input the files containing the result of the threshold scan (can be for different VFATs and different iterations) and the rate to...## Description
Add a command to `gemos` to analyze the results of the sbit rate scans. The command takes as input the files containing the result of the threshold scan (can be for different VFATs and different iterations) and the rate to be accepted to determine the VFAT threshold. The output file contains the thresholds in DAC units for each VFAT.
Example of usage:
```
poetry run gemos analyze sbit -r 100 /gemdata/sBitRateScan/fed1478-slot07.txt /gemdata/results/sBitRateScan/fed1478-slot07.cfg
```
Example of sbit scan plot:
![fed1478-slot7-oh1](/uploads/e2f84079948e55640a64c20e32234506/fed1478-slot7-oh1.png)
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
Tested on the results of a threshold scan taken on a GE2/1 module.
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/20Retrieve VFAT calibration parameters from database2021-11-05T15:17:07+01:00Antonello PellecchiaRetrieve VFAT calibration parameters from database## Description
This MR adds to `gemos` a command that dumps to a CSV-like file the VFAT calibration parameters obtained from a query to the database.
Example of usage:
```
poetry run gemos poetry run gemos query vfat-calibration vfat-c...## Description
This MR adds to `gemos` a command that dumps to a CSV-like file the VFAT calibration parameters obtained from a query to the database.
Example of usage:
```
poetry run gemos poetry run gemos query vfat-calibration vfat-calibrations.dat
```
The file `vfat-calibrations.dat` is sanitize and contains the following information:
```
chip-id;vref-adc;iref;adc0-m;adc0-b;cal-dac-m;cal-dac-b
1230;3;29;1.9089;-324.593;-0.272049;67.5893
1231;3;39;1.89825;-318.341;-0.229325;56.4414
1232;3;29;1.98346;-325.632;-0.245428;61.0245
1233;3;33;1.87333;-307.779;-0.229754;56.3122
1234;3;35;1.90041;-317.408;-0.245102;60.4882
1235;3;27;1.94132;-319.831;-0.23473;57.6574
1236;3;33;1.9382;-320.212;-0.23664;58.5432
1237;3;35;1.96087;-326.629;-0.251636;62.2413
1238;3;31;1.92872;-329.099;-0.257939;63.8939
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #34.
## How Has This Been Tested?
Tested on some VFATs found lying around in the lab :)
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petre