cmsgemos-analysis merge requestshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests2022-10-27T16:56:30+02:00https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/38Add a feature to produce OptoHybrid configurations with S-bit delays2022-10-27T16:56:30+02:00Laurent PetreAdd a feature to produce OptoHybrid configurations with S-bit delays## Description
<!-- Describe your changes in detail -->
_Based on top of !37._
All is in the title... The routine ingest file structured like:
```
fed;slot;oh;vfat;group;delay
1467;1;0;0;0;2
1467;1;0;0;1;2
1467;1;0;0;2;2
1467;1;0;0;3;2...## Description
<!-- Describe your changes in detail -->
_Based on top of !37._
All is in the title... The routine ingest file structured like:
```
fed;slot;oh;vfat;group;delay
1467;1;0;0;0;2
1467;1;0;0;1;2
1467;1;0;0;2;2
1467;1;0;0;3;2
1467;1;0;0;4;2
1467;1;0;0;5;2
1467;1;0;0;6;2
[...]
```
where `delay` is the expected effective delay in BX for the group. And produces OptoHybrid configuration files such as:
```
TRIG.SBIT_BX_DELAY.VFAT0_GROUP0 1
TRIG.SBIT_BX_DELAY.VFAT0_GROUP1 1
TRIG.SBIT_BX_DELAY.VFAT0_GROUP2 1
TRIG.SBIT_BX_DELAY.VFAT0_GROUP3 1
TRIG.SBIT_BX_DELAY.VFAT0_GROUP4 1
TRIG.SBIT_BX_DELAY.VFAT0_GROUP5 1
TRIG.SBIT_BX_DELAY.VFAT0_GROUP6 1
[...]
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP0 1
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP1 1
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP2 1
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP3 1
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP4 1
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP5 1
TRIG.SBIT_BX_DELAY_EN.VFAT0_GROUP6 1
[...]
```
with the usual `fed1467-slot1/config-oh0.cfg` naming convention.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Got an example CSV-file with the desired effective delays. The per-OH output files match the expectations.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/44Add InfluxDB client dependency2023-02-08T12:26:26+01:00Laurent PetreAdd InfluxDB client dependency## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the official Python InfluxDB client as a dependency of this project. While there is no script currently using the library, it should help with the l...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the official Python InfluxDB client as a dependency of this project. While there is no script currently using the library, it should help with the library deployment in the CMS cluster and the development of custom user scripts.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Library installed; CI passes.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/27Add latency to create VFAT configuration files2022-03-09T14:52:56+01:00Camilla GalloniAdd latency to create VFAT configuration files## Description
<!-- Describe your changes in detail -->
Title says it all.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an iss...## Description
<!-- Describe your changes in detail -->
Title says it all.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Will be tested at p5.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/45Add PyYaml library dependency2023-02-27T18:27:43+01:00Laurent PetreAdd PyYaml library dependency## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the `PyYaml` library as a dependency of this project. While there is no script currently using the library, it should help with the deployment in th...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the `PyYaml` library as a dependency of this project. While there is no script currently using the library, it should help with the deployment in the CMS cluster and the development of custom user scripts. In particular, any script parsing the layout tree. It is expected for a layout tree parser to be included within this repository soon.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Missed in !44.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Library installed; CI passes.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/33Add sanity check on VFAT config DB dump2022-04-07T09:34:11+02:00Daniel EstradaAdd sanity check on VFAT config DB dump## Description
The `dump_database` routine in `vfat_calibratios.py` produces from DB a `DataFrame` with a duplicate VFAT, this creates warnings downstream as in the case of issue #43.
## Related Issue
Closes #41
<!-- This project only ...## Description
The `dump_database` routine in `vfat_calibratios.py` produces from DB a `DataFrame` with a duplicate VFAT, this creates warnings downstream as in the case of issue #43.
## Related Issue
Closes #41
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues according to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
I checked if the output file contained any duplicate line with
```
uniq -c output_file -d
```
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
I included a line of code to drop duplicates chips before writing the VFAT calibration parameters file.
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/21Add script to create VFAT configuration file from scan results2021-11-05T19:48:10+01:00Antonello PellecchiaAdd script to create VFAT configuration file from scan results## Description
Add an option to `gemos` to create VFAT configuration files. The command takes as input a folder with a well-known file structure* and produces as output a folder with the same file structure and updated parameters.
In t...## Description
Add an option to `gemos` to create VFAT configuration files. The command takes as input a folder with a well-known file structure* and produces as output a folder with the same file structure and updated parameters.
In the absence of an input file, a default configuration file can be used. This is typically used to bootstrap a new configuration based on the setup description.
The following parameters are currently supported:
* The VFAT calibration parameters retrieved from the database
* The results of the DAC scans
* The thresholds based on the results of the S-bit rate scans analysis routine
Usage example:
``` sh
gemos create-config vfat --default default.cfg -c calibration/fed0001-slot01.cfg -d dacScan/dac_Scans.cfg -t sBitRateScan/sbits_1.cfg -t sBitRateScan/sbits_2.cfg results/vfatConfig/
```
* The files will be organized in subfolders named `fed<xxxx>-slot<y>`.
## Related Issue
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #35.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested with results of scans performed on the VFATs available in the lab on a GE2/1 setup as well as the GE1/1 integration setup chambers.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/48Add VFAT occupancy analysis2023-12-11T14:55:19+01:00Camilla GalloniAdd VFAT occupancy analysis## Description
<!-- Describe your changes in detail -->
This MR aims at defining an analysis to spot the noisy channels by looking at the VFAT occupancy in a run taken with the local DAQ and already properly unpacked.
At the moment th...## Description
<!-- Describe your changes in detail -->
This MR aims at defining an analysis to spot the noisy channels by looking at the VFAT occupancy in a run taken with the local DAQ and already properly unpacked.
At the moment the definition of noisy channels is based on quartiles, and iterations are done until the number of outliers is 0 (by default).
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues according to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on latency run of 20220924.
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/51Allow the create VFAT configurations with channel masking2023-05-26T12:00:42+02:00Laurent PetreAllow the create VFAT configurations with channel masking## Description
<!-- Describe your changes in detail -->
As the title says, this MR allows the creation VFAT configurations with channel masking. More specifically, the S-curve analysis results (noisy and emtpy/dead channels) can be fed ...## Description
<!-- Describe your changes in detail -->
As the title says, this MR allows the creation VFAT configurations with channel masking. More specifically, the S-curve analysis results (noisy and emtpy/dead channels) can be fed to the `gemos create-config vfat` command.
Under the hood, the VFAT configuration parameters can now include masked registers. This is done in order to avoid multiplying the number of VFAT registers to set.
Additionally, a new option is introduced to bump the thresholds by a fixed amount of DAC units compared to the thresholds input file.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Fixes #50
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Can locally produce correct configurations based on the output of the S-curve analysis as well as all the previous scans.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/15Analysis suite template2021-08-26T18:07:51+02:00Laurent PetreAnalysis suite template## Description
<!-- Describe your changes in detail -->
This MR provides a template for the analysis suite under construction. The Python `gemanalysis` package has been created and is installed with its expected dependencies. The docume...## Description
<!-- Describe your changes in detail -->
This MR provides a template for the analysis suite under construction. The Python `gemanalysis` package has been created and is installed with its expected dependencies. The documentation also mentions PostgreSQL that is installed on the official GEM development machines.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Can be installed with the updated instructions.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/55Bring AlmaLinux 9 compatibility2023-11-21T19:11:25+01:00Laurent PetreBring AlmaLinux 9 compatibility## Description
Similarly to https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/315, this MR brings compatibility with the AlmaLinux 9 distribution and, hopefully, any EL9-compatible one (they are **not** tested). Most...## Description
Similarly to https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/315, this MR brings compatibility with the AlmaLinux 9 distribution and, hopefully, any EL9-compatible one (they are **not** tested). Most of the commits should speak for themselves. No behavioral changes were implemented.
The most interesting part of this MR relates to the complete overhaul of the publication jobs, which now use the GitLab (CI) environments feature to keep track of the deployments and to present a review/preview website with the updated content. Additional updates and changes regarding the website structure could be added in the future, following the feedback on the current proposal.
## Related Issue
None...
## How Has This Been Tested?
The packages can be built, the tests pass, and the CI passes and deploys the build artifacts to the desired location.
The RPM package also has been deployed on the Alma 9 machines at P5. Initial testing of both basic and more advanced features was successful; deeper tests are pending. It should however not prevent any decision on this MR since CC7 support should be unaffected.
## Types of changes
* [x] Bug fix (non-breaking change which fixes an issue)
* [x] New feature (non-breaking change which adds functionality)
* [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
* [x] My code follows the code style of this project.
* [x] My change requires a change to the documentation.
* [x] I have updated the documentation accordingly.
* [x] I have read the **CONTRIBUTING** document.
* [ ] I have added tests to cover my changes.
* [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/42Cleanup of the legacy tools2022-12-08T17:18:53+01:00Laurent PetreCleanup of the legacy tools## Description
<!-- Describe your changes in detail -->
This MR cleans up a few legacy tools, procedures, and dependencies. This repository was containing a significant amount of unused/dead code and dependencies, impacting new developm...## Description
<!-- Describe your changes in detail -->
This MR cleans up a few legacy tools, procedures, and dependencies. This repository was containing a significant amount of unused/dead code and dependencies, impacting new developments and the size of the deployed RPM.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Trivial changes, only dead code touched.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/10Convert the repository to poetry2021-08-18T14:32:42+02:00Laurent PetreConvert the repository to poetry## Description
<!-- Describe your changes in detail -->
This MR converts the repository to `poetry` instead of `tito` for easier tracking of the dependencies and easier safe usage. The CI is also fixed and adapted. As an optimization, a...## Description
<!-- Describe your changes in detail -->
This MR converts the repository to `poetry` instead of `tito` for easier tracking of the dependencies and easier safe usage. The CI is also fixed and adapted. As an optimization, a custom CI Docker image is produced.
RPM creation will be added in a subsequent MR.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The repository can be used locally following the instruction in the developers' guide.
The CI is also fully working:
* A custom CI Docker image is produced on demand: https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/container_registry/10566
* The documentation is published (manual job for `master`): https://cmsgemos.web.cern.ch/analysis/doc/
* Python packages will be released in the GitLab PyPi repository (manual job for `master`): will appear in https://gitlab.cern.ch/cmsgemonline/gem-online-analysis/-/packages
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/18Create script to analyze DAC scans2022-02-22T16:02:46+01:00Antonello PellecchiaCreate script to analyze DAC scans## Description
This MR adds an option to the `gemos` command that takes as an input the result of a DAC scan and creates an output file with the optimal DAC values to be set for each VFAT. Example of usage:
``` sh
poetry run gemos anal...## Description
This MR adds an option to the `gemos` command that takes as an input the result of a DAC scan and creates an output file with the optimal DAC values to be set for each VFAT. Example of usage:
``` sh
poetry run gemos analysis dac ~/daq/examples/gemdata/dacScan/fed1478-slot07.txt ~/daq/examples/results/dacScan/fed1478-slot07.cfg
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on the result of DAC scan performed on a GE2/1 chamber.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreAntonello PellecchiaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/17Create the script producing GBT configuration2021-09-13T17:11:23+02:00Laurent PetreCreate the script producing GBT configuration## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds a new script producing GBT configuration files to be uploaded to the back-end board based on the output of the GBT phase scan routine.
Depends on h...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds a new script producing GBT configuration files to be uploaded to the back-end board based on the output of the GBT phase scan routine.
Depends on https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/191.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
With `fed1478-slot02.cfg` as the output of the GBT phase scan routine, `gemos create-config gbt fed1478-slot02.cfg test-output` creates the expected files in the expected location:
```
test-output/
└── fed1478-slot2
├── config-oh0-gbt.cfg
├── config-oh1-gbt.cfg
└── config-oh2-gbt.cfg
```
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/9DAC scan analysis2021-08-24T15:45:26+02:00Laurent PetreDAC scan analysis## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If...## Description
<!-- Describe your changes in detail -->
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Anagha AravindAnagha Aravindhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/4Feature/documentation build2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/documentation build## Description
<!--- Describe your changes in detail -->
Implements the documentation build system based on `sphinx` with read-the-docs theme.
Changelog:
- Provide `sphinx` configuration with required extensions
- Include markdown CONT...## Description
<!--- Describe your changes in detail -->
Implements the documentation build system based on `sphinx` with read-the-docs theme.
Changelog:
- Provide `sphinx` configuration with required extensions
- Include markdown CONTRIBUTING and DEVELOPERS files
- Note: toc tree and some links are messed up. Fix TBD
- problem: it is not possible to have working representation both for GitLab and for
`recommonmark` due to anchoring and referencing differencies. Currently keep `GitLab` anchoring.
- Include Jupyter notebook as user story
- Provide API doc and cleanup comments in the code
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #10
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
The documentation has been built in local virtual environment.
Uploaded for testing and looking around [here](https://mdalchen.web.cern.ch/gem-online-analysis/index.html)
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Pieter EveraertsLouis MoureauxLaurent PetrePieter Everaertshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/1Feature/initial repository setup2022-06-15T14:10:30+02:00Mykhailo DalchenkoFeature/initial repository setupCloses #1
Details are provided in the underlying issue and in commit messagesCloses #1
Details are provided in the underlying issue and in commit messagesLaurent PetreCamilla GalloniLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/3Feature/tuples as column names2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/tuples as column names## Description
<!--- Describe your changes in detail -->
This MR contains two commits, we have to decide which one we want to follow:
- one with partial implementation of column names tuplization: `('FIELD NAME', slot, link, position)`...## Description
<!--- Describe your changes in detail -->
This MR contains two commits, we have to decide which one we want to follow:
- one with partial implementation of column names tuplization: `('FIELD NAME', slot, link, position)`
- second with extension of the tuple including block id and sub block: `('FIELD NAME', 'BLOCK ID', 'SUB BLOCK', slot, link, position)`
Since the performance drop is not significant (and anyways it is an improvement over the f-strings), I'd go for the second option as this would greatly simplify the analysis. The end goal for analysis would be to convert the `pandas.DataFrame` returned by unpacker to `pandas.MultiIndex` and then it can easily be sliced or cross-sectioned to select required subset of data (e.g. select all VFATs, or all VFATs from AMC N GEB M etc.)
Note: the documentation will require an update. I do not include it here for two reasons:
- First we need to decide on the tuple complexity
- There will be separate MR on the documentation build
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #11
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested by unpacking a test dataset. Performance result on MBP 13" Fall 2014:
```shell
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('./doc/notebooks/test_data/test_sdram.dat','sdram')"
1 loop, best of 5: 3.39 sec per loop
```
For comparison, previous result was 3.9s
<!--- ## Screenshots (if appropriate): -->
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Pieter EveraertsLouis MoureauxLaurent PetrePieter Everaertshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/2Feature/unpacker skeleton2022-06-15T14:10:33+02:00Mykhailo DalchenkoFeature/unpacker skeleton## Description
Detailed description with some implementation details has been presented at the [GEM DAQ SW meeting](https://indico.cern.ch/event/918874/contributions/3861739/attachments/2038025/3412756/test_unpacker.pdf)
The general stru...## Description
Detailed description with some implementation details has been presented at the [GEM DAQ SW meeting](https://indico.cern.ch/event/918874/contributions/3861739/attachments/2038025/3412756/test_unpacker.pdf)
The general structure remains the same, however important improvements were made:
- fixed bug with arbitrary hardware block ordering
- pandas.DataFrame conversion is not made once for all events instead of event-by-event (@lmoureau)
- `bitstring` package was replaced by `cbitstruct` (@lpetre)
- input data is read as `numpy.ndarray` (`fromfile` method)
- formatting strings (dictionaries) are now compiled
## Related Issue
<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- If addressing multiple issues, comment on their relation if needed -->
<!--- Please link issues accordgin to the automation rules: -->
<!--- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Closes #3, partially addresses #4 and #2
## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->
Tested by running on verified test data set, consisting of 9306 events
### Test on `gem904daq01`
```shell
CPU info:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 58
Model name: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
C++ code:
time ./gemtreewriter/bin/linux/x86_64_centos7/unpacker /tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat sdram
[GEMUnpacker]: ---> Main()
real 0m0.704s
user 0m0.556s
sys 0m0.151s
python code:
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('/tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat','sdram')"
***
INFO:gdh.unpacker:Finished unpacking. Processed 9306 events.
1 loops, best of 3: 3.36 sec per loop
Jupiter notebook:
CPU times: user 3.15 s, sys: 78.1 ms, total: 3.23 s
Wall time: 3.22 s
```
### Test on MBP 13" Fall 2014
```
CPU info:
machdep.cpu.max_basic: 13
machdep.cpu.max_ext: 2147483656
machdep.cpu.vendor: GenuineIntel
machdep.cpu.brand_string: Intel(R) Core(TM) i5-4308U CPU @ 2.80GHz
machdep.cpu.family: 6
machdep.cpu.model: 69
machdep.cpu.extmodel: 4
machdep.cpu.extfamily: 0
machdep.cpu.stepping: 1
python code:
python -m timeit -n 1 "from gdh import unpacker; unpacker.run('/tmp/run000434_Cosmic_CERNQC7_2020-03-02_chunk_0.dat','sdram')"
***
INFO:gdh.unpacker:Finished unpacking. Processed 9306 events.
1 loop, best of 5: 3.9 sec per loop
Jupiter notebook:
CPU times: user 3.66 s, sys: 122 ms, total: 3.78 s
Wall time: 3.84 s
```
## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [x] All new and existing tests passed.Louis MoureauxLaurent PetreLouis Moureauxhttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/39Fix crash while generating front-end configuation files2022-10-31T18:26:25+01:00Laurent PetreFix crash while generating front-end configuation files## Description
<!-- Describe your changes in detail -->
The title says it all... Fixes a bug introduced in 3a45cb6...
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new fea...## Description
<!-- Describe your changes in detail -->
The title says it all... Fixes a bug introduced in 3a45cb6...
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
None, discovered within hours when deployed to production.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The fix routines can generate seamlessly correct front-end configuration files in the CMS production system.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petre