cmsgemos-analysis merge requestshttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests2024-03-22T14:58:22+01:00https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/41Implement latency scan analysis2024-03-22T14:58:22+01:00Camilla GalloniImplement latency scan analysis## Description
<!-- Describe your changes in detail -->
This MR aims at introducing the analysis of the latency scan taken with the local DAQ.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-...## Description
<!-- Describe your changes in detail -->
This MR aims at introducing the analysis of the latency scan taken with the local DAQ.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Fixes #24.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on `/afs/cern.ch/work/c/cgalloni/public/4Laurent/latency_150b_2022092402-be-fe-00-00-75-bc-raw.root`. See comments for more additional details.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/57Option for including ZCC threshold files in the VFAT configuration2024-01-17T13:54:26+01:00Antonello PellecchiaOption for including ZCC threshold files in the VFAT configuration## Description
This MR adds a command line option named `-z` or `--zcc-threshold` to the command `gemos create-config vfat` to set the `CFG_THR_ZCC_DAC` register from a ZCC s-bit rate scan result.
## Related Issue
<!-- This project only...## Description
This MR adds a command line option named `-z` or `--zcc-threshold` to the command `gemos create-config vfat` to set the `CFG_THR_ZCC_DAC` register from a ZCC s-bit rate scan result.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
Tested on the ME0 stack.
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/48Add VFAT occupancy analysis2023-12-11T14:55:19+01:00Camilla GalloniAdd VFAT occupancy analysis## Description
<!-- Describe your changes in detail -->
This MR aims at defining an analysis to spot the noisy channels by looking at the VFAT occupancy in a run taken with the local DAQ and already properly unpacked.
At the moment th...## Description
<!-- Describe your changes in detail -->
This MR aims at defining an analysis to spot the noisy channels by looking at the VFAT occupancy in a run taken with the local DAQ and already properly unpacked.
At the moment the definition of noisy channels is based on quartiles, and iterations are done until the number of outliers is 0 (by default).
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues according to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Tested on latency run of 20220924.
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/55Bring AlmaLinux 9 compatibility2023-11-21T19:11:25+01:00Laurent PetreBring AlmaLinux 9 compatibility## Description
Similarly to https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/315, this MR brings compatibility with the AlmaLinux 9 distribution and, hopefully, any EL9-compatible one (they are **not** tested). Most...## Description
Similarly to https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/315, this MR brings compatibility with the AlmaLinux 9 distribution and, hopefully, any EL9-compatible one (they are **not** tested). Most of the commits should speak for themselves. No behavioral changes were implemented.
The most interesting part of this MR relates to the complete overhaul of the publication jobs, which now use the GitLab (CI) environments feature to keep track of the deployments and to present a review/preview website with the updated content. Additional updates and changes regarding the website structure could be added in the future, following the feedback on the current proposal.
## Related Issue
None...
## How Has This Been Tested?
The packages can be built, the tests pass, and the CI passes and deploys the build artifacts to the desired location.
The RPM package also has been deployed on the Alma 9 machines at P5. Initial testing of both basic and more advanced features was successful; deeper tests are pending. It should however not prevent any decision on this MR since CC7 support should be unaffected.
## Types of changes
* [x] Bug fix (non-breaking change which fixes an issue)
* [x] New feature (non-breaking change which adds functionality)
* [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
* [x] My code follows the code style of this project.
* [x] My change requires a change to the documentation.
* [x] I have updated the documentation accordingly.
* [x] I have read the **CONTRIBUTING** document.
* [ ] I have added tests to cover my changes.
* [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/56Update the VFAT calibration query to latest DB version2023-11-07T15:00:33+01:00Laurent PetreUpdate the VFAT calibration query to latest DB version## Description
<!-- Describe your changes in detail -->
The title says it all... From the commit message:
```
A significant amount of work has been invested into cleaning up the VFAT
calibration DB/table (the work itself was originally ...## Description
<!-- Describe your changes in detail -->
The title says it all... From the commit message:
```
A significant amount of work has been invested into cleaning up the VFAT
calibration DB/table (the work itself was originally started due to a
bugged schema/view).
This commit updates (i.e. simplifies) the DB query function,
particularly the filtering stage. This is thanks to (1) the use of a
unified schema instead of two revisions and (2) the removal/masking of
the legacy cruft.
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
None. :confused:
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
The VFAT calibration data can successfully be retrieved from the CMS VFAT DB.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/54Improve S-curve channels distribution plots2023-06-30T22:13:08+02:00Laurent PetreImprove S-curve channels distribution plots## Description
<!-- Describe your changes in detail -->
The title says it all...
From the commit message:
```
Replace STD band with IQR band in S-curve channels distribution plots
As per request of the operation team in order to bett...## Description
<!-- Describe your changes in detail -->
The title says it all...
From the commit message:
```
Replace STD band with IQR band in S-curve channels distribution plots
As per request of the operation team in order to better visualize and
get a better intuition of the noisy channels.
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
A complete set of S-curves from GE1/1 at P5 was successfully analyzed. Output plots match the expectations.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Camilla GalloniDaniel EstradaCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/24Use GBT and elink numbers in GBT configuration2023-06-23T17:57:42+02:00Antonello PellecchiaUse GBT and elink numbers in GBT configuration## Description
Created to match with [cmsgemos MR 241](https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/241). The new output configuration file uses the format `PHASE_GBT<n>_ELINK<m> <phase>` for each GBT and each el...## Description
Created to match with [cmsgemos MR 241](https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos/-/merge_requests/241). The new output configuration file uses the format `PHASE_GBT<n>_ELINK<m> <phase>` for each GBT and each elink.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
Tested in GE2/1 integration stand at B904.
Tested again after newest changes, still working.
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Antonello PellecchiaAntonello Pellecchiahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/22file restructuring done and poetry integration for the revamped analysis suite2023-06-23T17:57:42+02:00Pranjal Sharmafile restructuring done and poetry integration for the revamped analysis suite## Description
<!-- Describe your changes in detail -->
This MR implements the Analysis Suite framework which provides a robust way to handle results of GEM hardware calibration procedures and data. It provides a command-line interface ...## Description
<!-- Describe your changes in detail -->
This MR implements the Analysis Suite framework which provides a robust way to handle results of GEM hardware calibration procedures and data. It provides a command-line interface to:
- Import data into the analysis suite
- Keep track of imported data, the run number, and file versions (revision) for each run
- retrieve the files imported for given file name, revision, run number
More details can be found here: https://codimd.web.cern.ch/lDdIGeVZR_C28XUNE4h90g?view
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/53Misc S-curves analysis enhancements2023-06-23T17:15:30+02:00Camilla GalloniMisc S-curves analysis enhancements## Description
<!-- Describe your changes in detail -->
This MR mainly aims at fixing consistency problems between the labeled noisy channels and those seen in the boxplot graphs.
Additionally, multiple small improvements are included:...## Description
<!-- Describe your changes in detail -->
This MR mainly aims at fixing consistency problems between the labeled noisy channels and those seen in the boxplot graphs.
Additionally, multiple small improvements are included:
* Record S-curve fit warnings in the output `TTree`
* Copy the complete ROOT input file in the output file when the `--full-output` mode is selected
* Allow choosing the IQR multiplicative factor used to identify noisy channels
* Allow selecting whether or not to mask noisy and empty VFAT channels during the configuration generation
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Ensure that no more discrepancy is observed on P5 data which were previously problematic.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreDaniel EstradaLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/52Update the cluster mask algorithm for correctness2023-06-08T23:03:55+02:00Laurent PetreUpdate the cluster mask algorithm for correctness## Description
<!-- Describe your changes in detail -->
From the commit message:
```
After much investigation and a thorough look at real data from the GE1/1
detector installed at P5, it was noticed that the cluster mask scan
analysis a...## Description
<!-- Describe your changes in detail -->
From the commit message:
```
After much investigation and a thorough look at real data from the GE1/1
detector installed at P5, it was noticed that the cluster mask scan
analysis algorithm was flawed and non-functional.
Finding the best delay mask based on statistical properties is between
hard and unrealistic. Either the quantities are not available or the
final criteria are so strong that almost all delays end up masked.
Instead, this commit implements a more realistic approach based on an
arbitrary cut. That is the amount of high-mutiplicity events accepted.
The current default value corresponds to a maximum of 5%.
Additionally, the plotting routine and output are enhanced in order to
improve the visualization, in particular, of the delays that will end up
being masked online.
```
The algorithm itself is described in detail in the source code.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
None. :confused:
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
A reasonable list of delays to be masked is produced as well as the associated plots. Ran on data from local run `2.8349.05103L`.
## Screenshots
Typical example:
![fed1467-slot1-oh0](/uploads/1e312e09efb62136a3942279c0550b4f/fed1467-slot1-oh0.png)
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/51Allow the create VFAT configurations with channel masking2023-05-26T12:00:42+02:00Laurent PetreAllow the create VFAT configurations with channel masking## Description
<!-- Describe your changes in detail -->
As the title says, this MR allows the creation VFAT configurations with channel masking. More specifically, the S-curve analysis results (noisy and emtpy/dead channels) can be fed ...## Description
<!-- Describe your changes in detail -->
As the title says, this MR allows the creation VFAT configurations with channel masking. More specifically, the S-curve analysis results (noisy and emtpy/dead channels) can be fed to the `gemos create-config vfat` command.
Under the hood, the VFAT configuration parameters can now include masked registers. This is done in order to avoid multiplying the number of VFAT registers to set.
Additionally, a new option is introduced to bump the thresholds by a fixed amount of DAC units compared to the thresholds input file.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Fixes #50
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Can locally produce correct configurations based on the output of the S-curve analysis as well as all the previous scans.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/50Identify noisy channels based on the S-curves results2023-05-23T18:06:54+02:00Daniel EstradaIdentify noisy channels based on the S-curves results## Description
An extra feature is added to enhance the S-curves analysis (worked on !47) to detect noisy channels and flag them in its output tree. The identification has been done through the outliers returned by the boxplots by VFAT....## Description
An extra feature is added to enhance the S-curves analysis (worked on !47) to detect noisy channels and flag them in its output tree. The identification has been done through the outliers returned by the boxplots by VFAT.
## Related Issue
Fixes #52.
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Daniel EstradaDaniel Estradahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/47Implement S-curves analysis2023-05-22T15:19:15+02:00Laurent PetreImplement S-curves analysis## Description
The S-curve analysis is implemented. It takes as input:
* An arbitrary number of ROOT files containing S-curve histograms.
* An optional VFAT calibration constants file (only required if "fC" units are desired).
* An opt...## Description
The S-curve analysis is implemented. It takes as input:
* An arbitrary number of ROOT files containing S-curve histograms.
* An optional VFAT calibration constants file (only required if "fC" units are desired).
* An optional VFAT mapping file (only required if "fC" units are desired).
It produces as outputs:
* A ROOT file containing a `TTree` with the results of the S-curve analysis containing the following branches:
- Usual branches of channel identification ("fed", "slot", ..., "channel").
- A boolean flag indicating whether the S-curve fit was successful (based on the `ierr` curve_fit indicator or any RuntimeError that could be thrown).
- Computed threshold values.
- Computed noise/ENC values.
- Another boolean flag indicating whether a channel has no data (it is empty).
- (Optional) A `TDirectory` per VFAT with the 2D histograms `chanHitvsCalDAC2D`.
* A directory containing the following plots:
- A summary figure per OH, with 2D histograms for each VFAT.
- A figure per OH containing box plots with the thresholds/activation values, and the box plot with the noise/ENC.
- (Optional) A complete overview of data distribution per VFAT. These plots show the data for each channel and contrast with histogram distributions.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Fixes https://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/issues/15
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
```bash
usage: gemos analyze scurves [-h] [-p {essentials,full,no-plot}] [-f] [-u {DAC,fC}] [-m MAPPING_FILE] [-c CALIBRATION_FILE]
inputfiles [inputfiles ...] outputdir
positional arguments:
inputfiles Files containing the S-curves histograms ROOT files
outputdir Output directory to save the results of the analysis
optional arguments:
-h, --help show this help message and exit
-p {essentials,full,no-plot}, --plotting {essentials,full,no-plot}
Specify in case summary and/or boxplot scurve plots are desired
-f, --full-output Specify when an output file that includes the input histograms be desired
-u {DAC,fC}, --units {DAC,fC}
Specify in case fC or DAC units are desired
-m MAPPING_FILE, --mapping MAPPING_FILE
File containing the VFAT chip ID to logical address mapping (see cmsgemos gem-dump-vfat-mapping)
-c CALIBRATION_FILE, --calibration CALIBRATION_FILE
File containing the VFAT calibration parameters (see gemos query vfat-calibration)
```
<!-- ## Screenshots (if appropriate): -->
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Daniel EstradaDaniel Estradahttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/49Update Poetry to version 1.4.22023-04-19T15:38:22+02:00Laurent PetreUpdate Poetry to version 1.4.2## Description
<!-- Describe your changes in detail -->
As the title says, this MR updates Poetry to version 1.4.2.
It revealed necessary due to a breaking change in the PyPI API that was leaving the poetry.lock metadata.files table em...## Description
<!-- Describe your changes in detail -->
As the title says, this MR updates Poetry to version 1.4.2.
It revealed necessary due to a breaking change in the PyPI API that was leaving the poetry.lock metadata.files table empty. Instead of staying on an old and unpinned version, this commit updates Poetry to the latest stable version and pins it.
Note that it introduces a new poetry.lock version (2.0) that requires a modern Poetry version. Update your installations (and clean your caches)!
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
* [x] The development workflow behaves as expected
* [x] The CI passes
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [x] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/37Make a routine to analyze the cluster mask rate scan2023-04-06T15:38:40+02:00Camilla GalloniMake a routine to analyze the cluster mask rate scan## Description
<!-- Describe your changes in detail -->
MR is prepared and left in draft just in case the format of the masking is decided soon in the firmware.
At the moment the masking is implemented as a window of a certain width, st...## Description
<!-- Describe your changes in detail -->
MR is prepared and left in draft just in case the format of the masking is decided soon in the firmware.
At the moment the masking is implemented as a window of a certain width, starting at some delay (BX units).
It seems convenient to implement a mask based on individual delays, since in most of the onto hybrids the scan results favor delays that are not adjacent.
Output files for both cases are produced, with plots (optional).
At the moment the pool implementation for making plots in parallel is not working.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Output of the analysis of a scan taken in p5 is : `/afs/cern.ch/work/c/cgalloni/public/4Laurent/cluster-mask-scan-out-width1-2sigma-2022082604`
<!-- ## Screenshots (if appropriate): -->
![fed1467-slot1-oh2](/uploads/529f2ba1801aca0a373a1a5d13144936/fed1467-slot1-oh2.png)
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.
- [ ] All new and existing tests passed.Camilla GalloniCamilla Gallonihttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/46Replace cx_Oracle with python-oracledb2023-03-07T18:20:19+01:00Laurent PetreReplace cx_Oracle with python-oracledb## Description
<!-- Describe your changes in detail -->
The title says it all: this MR replaces the `cx_Oracle` dependency with `python-oracledb`.
See the commit message and issue #51 for more details.
```
In addition to being the new...## Description
<!-- Describe your changes in detail -->
The title says it all: this MR replaces the `cx_Oracle` dependency with `python-oracledb`.
See the commit message and issue #51 for more details.
```
In addition to being the new and updated version of the cx_Oracle
library, python-oracledb has the benefit to provide the so-called "Thin"
mode. Enabled by default, it avoids any dependency on the Oracle Client
libraries in most of the (basic) cases.
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Fixes #51
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Code runs seamlessly without requiring the Oracle Client libraries. The VFAT calibration constants output file is identical before and after this MR.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/45Add PyYaml library dependency2023-02-27T18:27:43+01:00Laurent PetreAdd PyYaml library dependency## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the `PyYaml` library as a dependency of this project. While there is no script currently using the library, it should help with the deployment in th...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the `PyYaml` library as a dependency of this project. While there is no script currently using the library, it should help with the deployment in the CMS cluster and the development of custom user scripts. In particular, any script parsing the layout tree. It is expected for a layout tree parser to be included within this repository soon.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
Missed in !44.
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Library installed; CI passes.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/44Add InfluxDB client dependency2023-02-08T12:26:26+01:00Laurent PetreAdd InfluxDB client dependency## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the official Python InfluxDB client as a dependency of this project. While there is no script currently using the library, it should help with the l...## Description
<!-- Describe your changes in detail -->
As the title says, this MR adds the official Python InfluxDB client as a dependency of this project. While there is no script currently using the library, it should help with the library deployment in the CMS cluster and the development of custom user scripts.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Library installed; CI passes.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/43Update the VFAT DB query to support the Phase-II schema2023-01-30T12:51:35+01:00Laurent PetreUpdate the VFAT DB query to support the Phase-II schema## Description
<!-- Describe your changes in detail -->
The title says it all. More details can be found in the commit message:
```
In order to include the new Phase-II VFAT3 plugin cards in the CMS DB a
new schema has been implemented...## Description
<!-- Describe your changes in detail -->
The title says it all. More details can be found in the commit message:
```
In order to include the new Phase-II VFAT3 plugin cards in the CMS DB a
new schema has been implemented in the CMS_GEM_MUON_VIEW.GEM_VFAT3_PROD_SUMMARY_V_RH
view. While it has the advantage of not modifying existing GE1/1 VFAT3
hybrids data, this change creates additional complications in an
already messy database view.
This commit overhauls the VFAT DB query and sanitation function in order
to deal with the added complexity in the application logic code rather
than in the Oracle SQL query.
The GE1/1 entries are identical before and after this commit, modulo the
additional columns. The Phase-II entries match the expectations, based
on raw QC results files.
```
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Data can be fetched from the online DB and contains the correct information.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petrehttps://gitlab.cern.ch/cmsgemonline/gem-daq/cmsgemos-analysis/-/merge_requests/42Cleanup of the legacy tools2022-12-08T17:18:53+01:00Laurent PetreCleanup of the legacy tools## Description
<!-- Describe your changes in detail -->
This MR cleans up a few legacy tools, procedures, and dependencies. This repository was containing a significant amount of unused/dead code and dependencies, impacting new developm...## Description
<!-- Describe your changes in detail -->
This MR cleans up a few legacy tools, procedures, and dependencies. This repository was containing a significant amount of unused/dead code and dependencies, impacting new developments and the size of the deployed RPM.
## Related Issue
<!-- This project only accepts pull requests related to open issues -->
<!-- If suggesting a new feature or change, please discuss it in an issue first -->
<!-- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!-- If addressing multiple issues, comment on their relation if needed -->
<!-- Please link issues accordgin to the automation rules: -->
<!-- https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#closing-issues-automatically -->
## How Has This Been Tested?
<!-- Please describe in detail how you tested your changes. -->
<!-- Include details of your testing environment, and the tests you ran to -->
<!-- see how your change affects other areas of the code, etc. -->
Trivial changes, only dead code touched.
## Types of changes
<!-- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
## Checklist:
<!-- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!-- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] My code follows the code style of this project.
- [x] My change requires a change to the documentation.
- [x] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.Laurent PetreLaurent Petre