atlas issueshttps://gitlab.cern.ch/groups/atlas/-/issues2022-08-11T12:42:57+02:00https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/10Add ROOTSYS env var to release setup script2022-08-11T12:42:57+02:00Will ButtingerAdd ROOTSYS env var to release setup scriptWe've had a request for the ROOTSYS environment variable to point at the release location when the release is setup. I think this seems like a reasonable request. @zewolffs can you perhaps ask Attila how to add such an environment variab...We've had a request for the ROOTSYS environment variable to point at the release location when the release is setup. I think this seems like a reasonable request. @zewolffs can you perhaps ask Attila how to add such an environment variable to the auto-generated env_setup_install.sh script? I could see a way to hack it into the setup.sh.in in the atlasexternals module but I suspect that isn't necessarily the right place for it to go and instead it should become part of the generated env_setup_install.sh of the release.
Thanks!Zef WolffsZef Wolffshttps://gitlab.cern.ch/atlas/StatAnalysis/-/issues/12xRooFit (and RooFit?) tests break with const. terms optimization2022-07-22T16:07:36+02:00Zef WolffsxRooFit (and RooFit?) tests break with const. terms optimizationZef WolffsZef Wolffshttps://gitlab.cern.ch/atlas/StatAnalysis/-/issues/15Make StatAnalysis work on SWAN2022-08-11T13:31:09+02:00Will ButtingerMake StatAnalysis work on SWANIt would be nice to have a recipe that allows us to create SWAN notebooks with StatAnalysis. I will investigate ...It would be nice to have a recipe that allows us to create SWAN notebooks with StatAnalysis. I will investigate ...Will ButtingerWill Buttingerhttps://gitlab.cern.ch/atlas/StatAnalysis/-/issues/16StatAnalysis as a potential replacement for hcomb-docker2023-07-12T15:26:55+02:00Alexander HeldStatAnalysis as a potential replacement for hcomb-dockerWe have been using [hcomb-docker](https://gitlab.cern.ch/atlas_higgs_combination/software/hcomb-docker) for building images for Higgs Combination purposes. I think it would be great to be able to adopt StatAnalysis for this. From a preli...We have been using [hcomb-docker](https://gitlab.cern.ch/atlas_higgs_combination/software/hcomb-docker) for building images for Higgs Combination purposes. I think it would be great to be able to adopt StatAnalysis for this. From a preliminary look, I see that there is some software that we have in hcomb-docker which is not currently included in StatAnalysis:
- https://gitlab.cern.ch/cburgard/CMSFitExtensions
- https://gitlab.cern.ch/cburgard/RooFitUtils
- https://gitlab.cern.ch/atlas-physics/stat/tools/StatisticsTools
Another thing that has been important in the past is the ability to build custom ROOT versions from branches we control (e.g. to cherry-pick fixes / improvements). We have been using this fork https://gitlab.cern.ch/atlas_higgs_combination/software/hcombroot for this purpose, there is also https://github.com/atlascollaboration/root available. Looking at the setup in this repository, we could probably override
https://gitlab.cern.ch/atlas/StatAnalysis/-/blob/d690f4ee44429c5ffcf12cfb2512582850916d04/Externals/ROOT/CMakeLists.txt#L111
```cmake
set( ATLAS_ROOT_SOURCE "GIT_REPOSITORY;https://github.com/root-project/root.git"
CACHE STRING "Source for the ROOT build" )
```
if needed for that purpose. Do you expect any issues with a workflow like that if we would need a custom tag of StatAnalysis built from a fork?
Another thing that might be useful to compare are details regarding the ROOT compilation in comparison to https://gitlab.cern.ch/atlas-amglab/atlstats (cc @feickert), some things had been requested there over time and perhaps they would be similarly useful to StatAnalysis.
The image size might also be a point of interest: StatAnalysis is slightly over 5 GB, the corresponding hcomb-docker images are a bit over 2 GB (largely due to the very nicely built atlstats images, which are very compact). A multi-stage build might help reduce the size for StatAnalysis as well.
We should also add the StatAnalysis images to https://gitlab.cern.ch/unpacked/sync to make them available on CVMFS, which is crucial for using them at scale (see [slides](https://indico.cern.ch/event/1038133/contributions/4361896/attachments/2243958/3805205/20210512_HComb_docker_CVMFS.pdf)).
_edit:_ Users could just run `asetup` on the grid instead presumably, so the addition of the images to CVMFS is not as important as it was for hcomb-docker, but nevertheless I think it makes sense to be added.https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/19Enable CVMFS deployment via Gitlab rather than Jenkins2022-08-16T17:00:53+02:00Zef WolffsEnable CVMFS deployment via Gitlab rather than JenkinsCurrently it is a bit of work to deploy a full release to CVMFS since you would have to start and configure the job on jenkins. Ideally we would be able to do this through gitlab with a single push of a button.
My idea is to make a manu...Currently it is a bit of work to deploy a full release to CVMFS since you would have to start and configure the job on jenkins. Ideally we would be able to do this through gitlab with a single push of a button.
My idea is to make a manually triggered "schedule" for the Gitlab CI that sends an API request to jenkins to start the CVMFS deployment job with the right configuration parameters etc... . A schedule is easily started by going to CI/CD->Schedules. The main issue with this is that we do not want to save the jenkins user credentials in plaintext in the configuration. We can use Gitlab vault for this, the approach to which is described here: https://docs.gitlab.com/ee/ci/secrets/#configure-your-vault-server.
I will first see if such a vault already exists for atlas and in particular the Jenkins credentials, and speak to Alex Undrus about whether this is safe.Zef WolffsZef Wolffshttps://gitlab.cern.ch/atlas/StatAnalysis/-/issues/25Add Robert Les/StatComparisons tests to StatAnalysis CI/CD2022-10-10T14:47:15+02:00Zef WolffsAdd Robert Les/StatComparisons tests to StatAnalysis CI/CDAdding @rles work on statistics tests to StatAnalysis, either we:
- Copy the relevant code from the StatComparisons repo into StatAnalysis
- Keep the StatComparisons repo as is, and activate the CI there somehow from the CI here.
The l...Adding @rles work on statistics tests to StatAnalysis, either we:
- Copy the relevant code from the StatComparisons repo into StatAnalysis
- Keep the StatComparisons repo as is, and activate the CI there somehow from the CI here.
The latter might not be easy to do with Gitlab, we should look into that first. If it is not easily doable we go to the first option.https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/26Compile StatAnalysis on M1 Mac2022-10-27T10:40:43+02:00Zef WolffsCompile StatAnalysis on M1 MacZef WolffsZef Wolffshttps://gitlab.cern.ch/atlas/StatAnalysis/-/issues/27Potentially Improve docker and CI structure to avoid yum installations every ...2023-02-07T11:00:58+01:00Will ButtingerPotentially Improve docker and CI structure to avoid yum installations every timeI have been wondering if its possible to create a base image for our project with the yum installations all done already? This image could be the basis of the docker images, i.e. we could skip the `RUN sudo yum...` in the DockerFile. Add...I have been wondering if its possible to create a base image for our project with the yum installations all done already? This image could be the basis of the docker images, i.e. we could skip the `RUN sudo yum...` in the DockerFile. Additionally our CI could be faster as they could use the base image rather than the yum installs in the build and test steps.
In summary can we have instead of `atlas/centos7-atlasos:latest-gcc11` base image have a base image consisting of that image but with the yum installation on top?
@zewolffs do you think it could be something you can figure out how to technically achieve?https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/30rootbrowse command not working2023-01-19T18:51:52+01:00Will Buttingerrootbrowse command not workingI'd never heard of the rootbrowse command until today but a student tried to use it in StatAnalysis and it didnt work. the issue is the rootbrowse command (which is actually a python script) has at the top of it:
```
#!/usr/bin/env /bui...I'd never heard of the rootbrowse command until today but a student tried to use it in StatAnalysis and it didnt work. the issue is the rootbrowse command (which is actually a python script) has at the top of it:
```
#!/usr/bin/env /builds/atlas/StatAnalysis/ci_build/x86_64-centos7-gcc11-opt/bin/python3
```
This is from: /cvmfs/atlas.cern.ch/repo/sw/software/0.1/StatAnalysis/0.1.2/InstallArea/x86_64-centos7-gcc11-opt/bin/rootbrowse
So it looks like we need to have something in the installation step override this python path. Should look at if and how this is done for the Athena builds?https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/34Fix tests in master2023-05-30T18:14:46+02:00Will ButtingerFix tests in masterWe currently disable package builds by setting the relevant cmake variable to empty string .... we need the test project to look at these variables to decide which tests to do ....
I had a go at fixing this quickly but will need further...We currently disable package builds by setting the relevant cmake variable to empty string .... we need the test project to look at these variables to decide which tests to do ....
I had a go at fixing this quickly but will need further work ... for now will permit tests to fail in masterZef WolffsZef Wolffshttps://gitlab.cern.ch/atlas/StatAnalysis/-/issues/40Syncing images to CVMFS2023-09-18T17:20:18+02:00Alexander HeldSyncing images to CVMFSI noticed that the images are not synced to CVMFS yet via https://gitlab.cern.ch/unpacked/sync. Is there any reason to not enable this? This is perhaps less crucial as `asetup` is available instead but I see no harm in adding them in add...I noticed that the images are not synced to CVMFS yet via https://gitlab.cern.ch/unpacked/sync. Is there any reason to not enable this? This is perhaps less crucial as `asetup` is available instead but I see no harm in adding them in addition.https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/41lzma broken in Python installation2023-09-26T17:00:21+02:00Alexander Heldlzma broken in Python installationAs flagged by @jojamies:
```bash
~ ❯ docker run -it --rm gitlab-registry.cern.ch/atlas/statanalysis:main
[...]
[bash][atlas]:workdir > python
impPython 3.10.6 (main, Sep 13 2023, 19:27:47) [GCC 11.2.0] on linux
Type "help", "copyright", ...As flagged by @jojamies:
```bash
~ ❯ docker run -it --rm gitlab-registry.cern.ch/atlas/statanalysis:main
[...]
[bash][atlas]:workdir > python
impPython 3.10.6 (main, Sep 13 2023, 19:27:47) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import lzma
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/StatAnalysis/999.999.999/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.10/lzma.py", line 27, in <module>
from _lzma import *
ModuleNotFoundError: No module named '_lzma'
```
This affects all versions I am aware of. A possible fix might be installing `xz-devel` into the base image at https://gitlab.cern.ch/atlas/StatAnalysis/-/blob/0ea3a138066954ecabc3c353245d7ba99bf3bcc2/docker/baseImage/Dockerfile#L9.https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/42CI bug for image destination2024-01-03T16:02:14+01:00Alexander HeldCI bug for image destinationThere is a missing `--destination` here in one CI workflow:
https://gitlab.cern.ch/atlas/StatAnalysis/-/blob/0ea3a138066954ecabc3c353245d7ba99bf3bcc2/.gitlab-ci.yml#L92 (compared to https://gitlab.cern.ch/atlas/StatAnalysis/-/blob/0ea3a...There is a missing `--destination` here in one CI workflow:
https://gitlab.cern.ch/atlas/StatAnalysis/-/blob/0ea3a138066954ecabc3c353245d7ba99bf3bcc2/.gitlab-ci.yml#L92 (compared to https://gitlab.cern.ch/atlas/StatAnalysis/-/blob/0ea3a138066954ecabc3c353245d7ba99bf3bcc2/.gitlab-ci.yml#L144, where it is correct).
!213 includes that fix to get the CI test working again. This affects `main` and `0.2` (possibly others).https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/44Update Python version to 3.112023-12-08T18:19:52+01:00Matthew FeickertUpdate Python version to 3.11For long term usefulness it is generally advisable to be pushing CPython to the latest releases that supports the required wheels for tools like `scipy` and `numpy`. At the moment the CPython version used is Python 3.10, though Python 3....For long term usefulness it is generally advisable to be pushing CPython to the latest releases that supports the required wheels for tools like `scipy` and `numpy`. At the moment the CPython version used is Python 3.10, though Python 3.11 has been out for over a year now (and the Scientific Python ecosystem now is building wheels against CPython release candidates and so that's never a problem now).
```console
$ eol python
┌───────┬────────────┬─────────┬────────────────┬────────────┬────────────┐
│ cycle │ release │ latest │ latest release │ support │ eol │
├───────┼────────────┼─────────┼────────────────┼────────────┼────────────┤
│ 3.12 │ 2023-10-02 │ 3.12.1 │ 2023-12-07 │ 2025-04-02 │ 2028-10-02 │
│ 3.11 │ 2022-10-24 │ 3.11.7 │ 2023-12-04 │ 2024-04-01 │ 2027-10-24 │
│ 3.10 │ 2021-10-04 │ 3.10.13 │ 2023-08-24 │ 2023-04-05 │ 2026-10-04 │
│ 3.9 │ 2020-10-05 │ 3.9.18 │ 2023-08-24 │ 2022-05-17 │ 2025-10-05 │
│ 3.8 │ 2019-10-14 │ 3.8.18 │ 2023-08-24 │ 2021-05-03 │ 2024-10-14 │
│ 3.7 │ 2018-06-26 │ 3.7.17 │ 2023-06-05 │ 2020-06-27 │ 2023-06-27 │
│ 3.6 │ 2016-12-22 │ 3.6.15 │ 2021-09-03 │ 2018-12-24 │ 2021-12-23 │
│ 3.5 │ 2015-09-12 │ 3.5.10 │ 2020-09-05 │ False │ 2020-09-13 │
│ 3.4 │ 2014-03-15 │ 3.4.10 │ 2019-03-18 │ False │ 2019-03-18 │
│ 3.3 │ 2012-09-29 │ 3.3.7 │ 2017-09-19 │ False │ 2017-09-29 │
│ 2.7 │ 2010-07-03 │ 2.7.18 │ 2020-04-19 │ False │ 2020-01-01 │
│ 2.6 │ 2008-10-01 │ 2.6.9 │ 2013-10-29 │ False │ 2013-10-29 │
└───────┴────────────┴─────────┴────────────────┴────────────┴────────────┘
```
Are there any of the C++ ecosystem tools that interface with Python that **can't** support Python 3.11 as of now? If not, I would strongly recommend updating.https://gitlab.cern.ch/atlas/StatAnalysis/-/issues/47Incremental build no longer working due to failed upload of cache2024-02-21T13:14:00+01:00Will ButtingerIncremental build no longer working due to failed upload of cacheIt looks as if gitlab admins have imposed a new upload limit on the build artifact, meaning we can't upload the cache.zip any more (see e.g. failed job: https://gitlab.cern.ch/atlas/StatAnalysis/-/jobs/36331308 ... even on max compressi...It looks as if gitlab admins have imposed a new upload limit on the build artifact, meaning we can't upload the cache.zip any more (see e.g. failed job: https://gitlab.cern.ch/atlas/StatAnalysis/-/jobs/36331308 ... even on max compression the cache.zip is 1.8GB which apparently is too big now??).
For now we have no choice but to disable the saving of the cache to the artifact :-(