Skip to content
Snippets Groups Projects

feat: Update to Release 24 (!6)

Merged feat: Update to Release 24 (!6)
feat/switch-to-release-24 into master
Merged Matthew Feickert requested to merge feat/switch-to-release-24 into master 
* Update Release 22 files to use Release 24 as Release 22 won't be used for analysis.
* Update to using gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.2 for testing
  in CI.
* Use cvmfs-venv to create a Python3 virtual environment with use control given updated setuptools
  releases that highlight problems with PYTHONPATH pollution between Release 22 and Release 24 that
  don't allow for `--system-site-packages` to be sufficient alone.
   - c.f. https://github.com/matthewfeickert/cvmfs-venv
   - c.f. https://its.cern.ch/jira/browse/ATLINFR-4874
   - c.f. https://gitlab.cern.ch/recast-atlas/examples/analysis-base-container-python-venv/-/merge_requests/6
* Set Shell during Docker image build to be Bash for cvmfs-venv support.
Edited by Matthew Feickert

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Matthew Feickert assigned to @feickert

    assigned to @feickert

  • Matthew Feickert added 1 commit

    added 1 commit

    Compare with previous version

  • Matthew Feickert added 1 commit

    added 1 commit

    • 506d724e - Update to v0.0.4 of cvmfs-venv

    Compare with previous version

  • Matthew Feickert added 2 commits

    added 2 commits

    Compare with previous version

  • Matthew Feickert changed title from feat: Update to Release 24 to feat: Update to Release 24 (!6)

    changed title from feat: Update to Release 24 to feat: Update to Release 24 (!6)

  • Matthew Feickert changed the description

    changed the description

  • Matthew Feickert
    Matthew Feickert @feickert
    Author Owner

    So there was definitely some change between Release 22 and Release 24 given:

    Release 22

    Able to updated packages that exist in the image already if --system-site-packages is used.

    $ docker run --rm -ti gitlab-registry.cern.ch/atlas/athena/analysisbase:22.2.113        
             _ _____ _      _   ___ 
            /_\_   _| |    /_\ / __|
           / _ \| | | |__ / _ \\__ \
          /_/ \_\_| |____/_/ \_\___/

This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:

          source /release_setup.sh

[bash][atlas]:workdir > . /release_setup.sh 
Configured GCC from: /opt/lcg/gcc/11.2.0-8a51a/x86_64-centos7/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/22.2.113/InstallArea/x86_64-centos7-gcc11-opt
[bash][atlas AnalysisBase-22.2.113]:workdir > python -m venv --system-site-packages venv && . venv/bin/activate
(venv) [bash][atlas AnalysisBase-22.2.113]:workdir > python -m pip show pip
Name: pip
Version: 22.0.4
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg
Requires: 
Required-by: 
(venv) [bash][atlas AnalysisBase-22.2.113]:workdir > python -m pip install --upgrade pip                 
Requirement already satisfied: pip in /usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg (22.0.4)
Collecting pip
  Downloading pip-23.0.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 10.7 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Not uninstalling pip at /usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg, outside environment /workdir/venv
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-23.0.1
(venv) [bash][atlas AnalysisBase-22.2.113]:workdir > python -m pip show pip
Name: pip
Version: 23.0.1
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /workdir/venv/lib/python3.9/site-packages
Requires: 
Required-by: 
(venv) [bash][atlas AnalysisBase-22.2.113]:workdir > echo $PATH
/workdir/venv/bin:/usr/AnalysisBase/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/share:/usr/AnalysisBase/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/bin:/usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/bin:/usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/share:/opt/lcg/gcc/11.2.0-8a51a/x86_64-centos7/bin:/opt/lcg/binutils/2.37-355ed/x86_64-centos7/bin:/opt/cmake/3.24.3/Linux-x86_64/bin:/usr/sue/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
(venv) [bash][atlas AnalysisBase-22.2.113]:workdir > echo $PYTHONPATH
/usr/AnalysisBase/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/python:/usr/AnalysisBase/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib:/usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib:/usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/python:/usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
(venv) [bash][atlas AnalysisBase-22.2.113]:workdir >

    Release 24

    Stuck with the AnalaysisBase version

    > docker run --rm -ti gitlab-registry.cern.ch/atlas/athena/analysisbase:24.2.2  
             _ _____ _      _   ___ 
            /_\_   _| |    /_\ / __|
           / _ \| | | |__ / _ \\__ \
          /_/ \_\_| |____/_/ \_\___/

This is a self-contained ATLAS AnalysisBase image.
To set up the analysis release of the image, please
execute:

          source /release_setup.sh

[bash][atlas]:workdir > . /release_setup.sh 
Configured GCC from: /opt/lcg/gcc/11.2.0-8a51a/x86_64-centos7/bin/gcc
Configured AnalysisBase from: /usr/AnalysisBase/24.2.2/InstallArea/x86_64-centos7-gcc11-opt
[bash][atlas AnalysisBase-24.2.2]:workdir > python -m venv --system-site-packages venv && . venv/bin/activate
(venv) [bash][atlas AnalysisBase-24.2.2]:workdir > python -m pip show pip
Name: pip
Version: 22.0.4
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
Requires: 
Required-by: 
(venv) [bash][atlas AnalysisBase-24.2.2]:workdir > python -m pip install --upgrade pip
Requirement already satisfied: pip in /usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages (22.0.4)
Collecting pip
  Downloading pip-23.0.1-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 10.3 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Not uninstalling pip at /usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages, outside environment /workdir/venv
    Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-23.0.1
(venv) [bash][atlas AnalysisBase-24.2.2]:workdir > python -m pip show pip  # pip is showing the AnalysisBase version and not the one that was installed in the virtual environment's site-packages/
Name: pip
Version: 22.0.4
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
Requires: 
Required-by: 
(venv) [bash][atlas AnalysisBase-24.2.2]:workdir > echo $PATH
/workdir/venv/bin:/usr/AnalysisBase/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/share:/usr/AnalysisBase/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/bin:/usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/bin:/usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/share:/opt/lcg/gcc/11.2.0-8a51a/x86_64-centos7/bin:/opt/lcg/binutils/2.37-355ed/x86_64-centos7/bin:/opt/cmake/3.24.3/Linux-x86_64/bin:/usr/sue/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
(venv) [bash][atlas AnalysisBase-24.2.2]:workdir > echo $PYTHONPATH
/usr/AnalysisBase/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/python:/usr/AnalysisBase/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib:/usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib:/usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/python:/usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
(venv) [bash][atlas AnalysisBase-24.2.2]:workdir >
  • Matthew Feickert
    Matthew Feickert @feickert
    Author Owner

    This is confusing given that the /release_setup.sh is the same

    #
# Environment configuration file for the ATLAS user. Can be sourced to set up
# the analysis release installed in the image.
#

# Execute all of the environment setup scripts of the image/OS.
for script in /etc/profile.d/*.sh; do
   source ${script}
done

# Set up a nice prompt:
export PS1='\[\033[01;35m\][bash]\[\033[01;31m\][\u AnalysisBase-$AnalysisBase_VERSION]\[\033[01;34m\]:\W >\[\033[00m\] ';

# Set up the compiler:
source /opt/lcg/binutils/*/x86_64-*/setup.sh
source /opt/lcg/gcc/*/x86_64-*/setup.sh
echo "Configured GCC from: ${CC}"

# Configure Calibration via HTTP
export PATHRESOLVER_ALLOWHTTPDOWNLOAD=1

# Set up the analysis release:
source /usr/AnalysisBase/*/InstallArea/*/setup.sh
echo "Configured AnalysisBase from: $AnalysisBase_DIR"
  • Matthew Feickert
    Matthew Feickert @feickert
    Author Owner

    Ah it is because Release 22 was still using the deprecated easy_install to install packages while Release 24 is still getting eggs by doesn't have easy-install.pth, because as I pointed out in https://its.cern.ch/jira/browse/ATLINFR-4874?focusedCommentId=4777356&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-4777356,

    easy_install is very deprecated and the top-level module and console script were removed in setuptools v52.0.0 meaning that people should very much treat it is being gone. So the updated setuptools is highlighting the problems with packages in read only system site-packages/ that are put on PYTHONPATH.

    Release 22

    [bash][atlas AnalysisBase-22.2.113]:workdir > python -m pip show pip
Name: pip
Version: 22.0.4
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages/pip-22.0.4-py3.9.egg
Requires: 
Required-by: 
[bash][atlas AnalysisBase-22.2.113]:workdir > ls -lhtra /usr/AnalysisBaseExternals/22.2.113/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
total 4.3M
-rw-r--r--  1 root root  119 Mar 16 02:25 README.txt
-rw-r--r--  1 root root 2.3K Mar 16 02:25 site.py
-rw-r--r--  1 root root   30 Mar 16 02:25 setuptools.pth
-rw-r--r--  1 root root 1.1M Mar 16 02:25 setuptools-44.1.0-py3.9.egg
-rwxr-xr-x  1 root root 3.2M Mar 16 02:30 lhapdf.cpython-39-x86_64-linux-gnu.so
-rw-r--r--  1 root root  260 Mar 16 02:30 LHAPDF-6.5.1-py3.9.egg-info
-rw-r--r--  1 root root  251 Mar 16 02:30 xrootd-5.4.3-py3.9.egg-info
-rw-r--r--  1 root root  156 Mar 16 02:33 easy-install.pth
drwxr-xr-x  4 root root 4.0K Mar 16 16:17 numpy-1.22.3-py3.9-linux-x86_64.egg
drwxr-xr-x  2 root root 4.0K Mar 16 16:17 __pycache__
drwxr-xr-x  4 root root 4.0K Mar 16 16:17 XRootD
drwxr-xr-x  6 root root 4.0K Mar 16 16:17 Cython-0.29.28-py3.9-linux-x86_64.egg
drwxr-xr-x  4 root root 4.0K Mar 16 16:17 pip-22.0.4-py3.9.egg
drwxr-xr-x  4 root root 4.0K Mar 16 16:17 wheel-0.33.4-py3.9.egg
drwxr-xr-x  3 root root 4.0K Mar 16 16:17 pyxrootd
drwxr-xr-x  9 root root 4.0K Mar 16 16:17 .
drwxr-xr-x 36 root root  12K Mar 16 16:17 ..
[bash][atlas AnalysisBase-22.2.113]:~ > python -m pip list | grep 'pip\|setuptools\|wheel'
pip        22.0.4
setuptools 44.1.0
wheel      0.33.4
[bash][atlas AnalysisBase-22.2.113]:workdir >

    Release 24

    [bash][atlas AnalysisBase-24.2.2]:workdir > python -m pip show pip
Name: pip
Version: 22.0.4
Summary: The PyPA recommended tool for installing Python packages.
Home-page: https://pip.pypa.io/
Author: The pip developers
Author-email: distutils-sig@python.org
License: MIT
Location: /usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
Requires: 
Required-by: 
[bash][atlas AnalysisBase-24.2.2]:workdir > ls -lhtra /usr/AnalysisBaseExternals/24.2.2/InstallArea/x86_64-centos7-gcc11-opt/lib/python3.9/site-packages
total 3.3M
-rw-r--r--  1 root root  119 Apr  6 02:25 README.txt
-rw-r--r--  1 root root  152 Apr  6 02:25 distutils-precedence.pth
-rw-r--r--  1 root root  520 Apr  6 02:25 cython.py
-rw-r--r--  1 root root  251 Apr  6 02:29 xrootd-5.4.3-py3.9.egg-info
-rwxr-xr-x  1 root root 3.2M Apr  6 02:32 lhapdf.cpython-39-x86_64-linux-gnu.so
-rw-r--r--  1 root root  260 Apr  6 02:32 LHAPDF-6.5.1-py3.9.egg-info
drwxr-xr-x  2 root root 4.0K Apr  6 14:02 Cython-0.29.28-py3.9.egg-info
drwxr-xr-x  2 root root 4.0K Apr  6 14:02 numpy-1.22.3-py3.9.egg-info
drwxr-xr-x  3 root root 4.0K Apr  6 14:02 _distutils_hack
drwxr-xr-x  2 root root 4.0K Apr  6 14:02 __pycache__
drwxr-xr-x  4 root root 4.0K Apr  6 14:02 XRootD
drwxr-xr-x 13 root root 4.0K Apr  6 14:02 Cython
drwxr-xr-x  2 root root 4.0K Apr  6 14:02 pip-22.0.4-py3.9.egg-info
drwxr-xr-x 19 root root 4.0K Apr  6 14:02 numpy
drwxr-xr-x  2 root root 4.0K Apr  6 14:02 wheel-0.37.1-py3.9.egg-info
drwxr-xr-x  5 root root 4.0K Apr  6 14:02 wheel
drwxr-xr-x  2 root root 4.0K Apr  6 14:02 setuptools-57.1.0-py3.9.egg-info
drwxr-xr-x  7 root root 4.0K Apr  6 14:02 setuptools
drwxr-xr-x  3 root root 4.0K Apr  6 14:02 pyxrootd
drwxr-xr-x  3 root root 4.0K Apr  6 14:02 pyximport
drwxr-xr-x  6 root root 4.0K Apr  6 14:02 pkg_resources
drwxr-xr-x  5 root root 4.0K Apr  6 14:02 pip
drwxr-xr-x 18 root root 4.0K Apr  6 14:02 .
drwxr-xr-x 36 root root  12K Apr  6 14:02 ..
[bash][atlas AnalysisBase-24.2.2]:~ > python -m pip list | grep 'pip\|setuptools\|wheel'
pip        22.0.4
setuptools 57.1.0
wheel      0.37.1
[bash][atlas AnalysisBase-24.2.2]:workdir >

    This is relevant for the discussion in https://its.cern.ch/jira/browse/ATLASG-2477 as well.

    Edited by Matthew Feickert
  • Matthew Feickert changed the description

    changed the description

  • Matthew Feickert merged

    merged

  • Matthew Feickert mentioned in commit 8b22e320

    mentioned in commit 8b22e320

  • Matthew Feickert mentioned in merge request gstark/pycolumnarprototype!12 (merged)

    mentioned in merge request gstark/pycolumnarprototype!12 (merged)

  • Matthew Feickert mentioned in commit gstark/pycolumnarprototype@77519de5

    mentioned in commit gstark/pycolumnarprototype@77519de5

  • Matthew Feickert mentioned in commit gstark/pycolumnarprototype@bb408eba

    mentioned in commit gstark/pycolumnarprototype@bb408eba

Please register or sign in to reply