PyModules: update flake8, remove storm.py and add option for PyPI

Several changes to PyModules:

• Upgrade to flake8 3.9.2, which is required for proper Python 3.8 support. Also remove the now unnecessary importlib_metadata and zipp dependencies that were required for older Python versions.
• Remove storm.py 6.1.0 installation as LCG_100 contains the same version now. cc @ponyisi
• Introduce ATLAS_PYMODULES_ALLOW_PYPI_DOWNLOADS option to explicitly allow module installation from PyPI (disabled by default). This can be used temporarily for testing builds on platforms (e.g. aarch64) where the LCG release does not contain all packages.

New externals have been uploaded to EOS (ATLINFR-4272).

When we push this into athena, we should run this as full-build to make sure we are re-checking all python modules.

AE Build SUCCESS
Build logfiles are available at Jenkins [AE-MERGE-REQUEST-CC7 #538]

I was looking at the issue for a while before understanding it finally...

The issue was that I quickly looked at the CI build, and saw that it was downloading flake8 not from EOS, but from an external location.

...
Looking in indexes: https://pypi.org/simple, https://cern.ch/atlas-software-dist-eos/externals/PyModules/repo
Collecting flake8==3.9.2 (from -r /build2/atlasexternals/master/AthenaExternals/External/PyModules/CMakeFiles/requirements.txt (line 10))
Consolidate compiler generated dependencies of target testInstanceRestore
[ 90%] Built target testInstanceRestore
Consolidate compiler generated dependencies of target testDistCopy
[ 90%] Built target testDistCopy
Consolidate compiler generated dependencies of target testBug58950
[ 90%] Built target testBug58950
Consolidate compiler generated dependencies of target testSaveEngineStatus
[ 90%] Built target testSaveEngineStatus
Consolidate compiler generated dependencies of target testRanecuSequence
[ 90%] Built target testRanecuSequence
  Downloading https://files.pythonhosted.org/packages/fc/80/35a0716e5d5101e643404dabd20f07f5528a21f3ef4032d31a49c913237b/flake8-3.9.2-py2.py3-none-any.whl (73kB)
...

It was only after a while that I realised that there was a warning in the CMake configuration.

...
08:50:48 -- Configuring the build of package: PyModules
08:50:48 -- Found Python: /cvmfs/sft.cern.ch/lcg/releases/LCG_97apython3_ATLAS_1/Python/3.7.6/x86_64-centos7-gcc8-opt/bin/python3.7 (found suitable version "3.7.6", minimum required is "3.7") found components: Interpreter 
08:50:48 -- Found pip: /cvmfs/sft.cern.ch/lcg/releases/LCG_97apython3_ATLAS_1/pip/19.1.1/x86_64-centos7-gcc8-opt/bin  
08:50:48 -- Found pyflakes: /cvmfs/sft.cern.ch/lcg/releases/LCG_97apython3_ATLAS_1/pyflakes/2.1.1/x86_64-centos7-gcc8-opt/bin  
08:50:48 -- Found pycodestyle: /cvmfs/sft.cern.ch/lcg/releases/LCG_97apython3_ATLAS_1/pycodestyle/2.5.0/x86_64-centos7-gcc8-opt/bin  
08:50:48 -- Found mccabe: /cvmfs/sft.cern.ch/lcg/releases/LCG_97apython3_ATLAS_1/mccabe/0.6.1/x86_64-centos7-gcc8-opt/lib/python3.7/site-packages  
08:50:48 CMake Warning at /var/lib/jenkins/workspace/AE-MERGE-REQUEST-CC7/External/PyModules/CMakeLists.txt:53 (message):
08:50:48   Not all required Python packages were found.  They will be downloaded in a
08:50:48   non-reproducible manner!
08:50:48 
...

This is coming from:

https://gitlab.cern.ch/atlas/atlasexternals/-/blob/master/External/PyModules/CMakeLists.txt#L48-58

This is something that I've put in place for our aarch64 builds as a temporary measure. Until all Python modules that we need, would get installed into the ARM LCG releases. (Or until we added all the needed packages to EOS ourselves...) As you can see, you forgot to remove the requirements on importlib_metadata and zipp from there. Leading to the build allowing downloads from anywhere.

At the same time, this revealed yet another issue... The CI of this repository still uses LCG_97apython3_ATLAS_1, since that's the version that we still have in (for instance) https://gitlab.cern.ch/atlas/atlasexternals/-/blob/master/Projects/AthenaExternals/CMakeLists.txt#L30-35.

Since we've been using Python 3.8 in all of our (master based) releases since a while, I don't think there's much value in keeping up support for older LCG releases like this. So I would propose to update the default LCG version for all projects to something newer. Then we could merge in a fixed version of this MR on top of that.

added 1 commit

• 4e4c7a45 - PyModules: update flake8, remove storm.py and add option for PyPI

Compare with previous version

changed title from PyModules: update to flake8 3.9.2 to PyModules: update flake8, remove storm.py and add option for PyPI

changed the description

AE Build SUCCESS
Build logfiles are available at Jenkins [AE-MERGE-REQUEST-CC7 #542]

resolved all threads

mentioned in commit 822576c5

merged