WIP: LCG Hashed RPM Usage, master branch (2020.03.23.)

Attila Krasznahorkayrequested to merge
akraszna/atlasexternals:LCGHashedRPMs-master-20200323 into master

This is first stab at using the "hashed RPMs" provided by the SFT group.

There are 2 main parts to it:

  • The lcg_need_rpm(...) function was updated to set up a dependency on the "hashed RPM" instead of the "link RPM";
    • For this to work, the hash ID of the packages had to be tweaked a bit;
  • A post-installation script had to be introduced for setting up the symbolic links after the "hashed RPMs" are installed.

It's the second part that is quite tricky. I took @rhauser's script, and tried to tweak it according to my own taste. 😛 A couple of points about this:

  • For now I didn't actually look at LCG_97rc4, as the installed files for that on http://cern.ch/lcgpackages is a bit of a mess. Instead I've set up the script (rpmPostInstall.sh.in) to work with existing releases. In practice I was experimenting with LCG_96b.
  • The script itself is quite self-reliant at this point, apart from not knowing automatically where to look for the LCG installation. 😦 I was hoping that I could rely on $RPM_INSTALL_PREFIX for this, but of course the installation prefix can be (and usually is) different for the LCG and ATLAS packages. So instead we would have to run (a)yum with the LCG_INSTALL_PREFIX environment variable set to have this script behave correctly.
  • I didn't bother trying to set up a post-uninstallation script. Unfortunately because of how this script works, uninstallation is just not a well-defined operation for these symbolic links. (When uninstalling a given ATLAS RPM, it's nearly impossible to tell which symbolic links may need to be deleted...)

To give you a bit of a taste, the code produces this type of output when installing an RPM into a Docker image:

Step 9/11 : RUN yum install -y RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt
 ---> Running in 1e3fdb07698b
Loaded plugins: changelog, fastestmirror, kernel-module, ovl, protectbase,
              : tsflags, versionlock
Loading mirror speeds from cached hostfile
252 packages excluded due to repository protections
Resolving Dependencies
--> Running transaction check
---> Package RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt.noarch 0:1.0.0-1 will be installed
--> Processing Dependency: mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt for package: RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt-1.0.0-1.noarch
--> Processing Dependency: Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt for package: RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt-1.0.0-1.noarch
--> Running transaction check
---> Package Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96 will be installed
--> Processing Dependency: Python-e553a_2.7.16_x86_64_centos7_gcc8_opt for package: Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt-1.0.0-96.noarch
---> Package mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96 will be installed
--> Processing Dependency: heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt for package: mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt-1.0.0-96.noarch
--> Running transaction check
---> Package Python-e553a_2.7.16_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96 will be installed
--> Processing Dependency: sqlite-472f2_3280000_x86_64_centos7_gcc8_opt for package: Python-e553a_2.7.16_x86_64_centos7_gcc8_opt-1.0.0-96.noarch
---> Package heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-95 will be installed
--> Processing Dependency: gcc_8.2.0_x86_64_centos7 for package: heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt-1.0.0-95.noarch
--> Running transaction check
---> Package gcc_8.2.0_x86_64_centos7.noarch 0:3.0.0-1 will be installed
---> Package sqlite-472f2_3280000_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96 will be installed
--> Finished Dependency Resolution
Beginning Kernel Module Plugin
Finished Kernel Module Plugin

Dependencies Resolved

================================================================================
 Package                                      Arch   Version  Repository   Size
================================================================================
Installing:
 RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt noarch 1.0.0-1  local       150 k
Installing for dependencies:
 Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt   noarch 1.0.0-96 lcg-testing  17 M
 Python-e553a_2.7.16_x86_64_centos7_gcc8_opt  noarch 1.0.0-96 lcg-testing  20 M
 gcc_8.2.0_x86_64_centos7                     noarch 3.0.0-1  lcg_contrib 121 M
 heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt noarch 1.0.0-95 lcg          20 k
 mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt  noarch 1.0.0-96 lcg-testing  28 k
 sqlite-472f2_3280000_x86_64_centos7_gcc8_opt noarch 1.0.0-96 lcg-testing 7.1 M

Transaction Summary
================================================================================
Install  1 Package (+6 Dependent packages)

Total download size: 165 M
Installed size: 813 M
Downloading packages:
--------------------------------------------------------------------------------
Total                                               18 MB/s | 165 MB  00:09     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : gcc_8.2.0_x86_64_centos7-3.0.0-1.noarch                      1/7 
Replacing /afs/cern.ch/cms/CAF/CMSCOMM/COMM_ECAL/dkonst/GCC/lcgcmake-install -> /opt/lcg ...
  Installing : sqlite-472f2_3280000_x86_64_centos7_gcc8_opt-1.0.0-96.noar   2/7 
Checking for post install in /opt/lcg/sqlite/3280000-472f2/x86_64-centos7-gcc8-opt
Invoking post install /opt/lcg/sqlite/3280000-472f2/x86_64-centos7-gcc8-opt/.post-install.sh
Replacing /build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/install -> /opt/lcg [or ] ...
  Installing : Python-e553a_2.7.16_x86_64_centos7_gcc8_opt-1.0.0-96.noarc   3/7 
Checking for post install in /opt/lcg/Python/2.7.16-e553a/x86_64-centos7-gcc8-opt
Invoking post install /opt/lcg/Python/2.7.16-e553a/x86_64-centos7-gcc8-opt/.post-install.sh
Replacing /build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/install -> /opt/lcg [or ] ...
  Installing : Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt-1.0.0-96.noarch   4/7 
Checking for post install in /opt/lcg/Boost/1.70.0-eebf1/x86_64-centos7-gcc8-opt
Invoking post install /opt/lcg/Boost/1.70.0-eebf1/x86_64-centos7-gcc8-opt/.post-install.sh
Replacing /build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/install -> /opt/lcg [or ] ...
  Installing : heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt-1.0.0-95.noar   5/7 
Checking for post install in /opt/lcg/MCGenerators/heputils/1.3.2-8ac2c/x86_64-centos7-gcc8-opt
Invoking post install /opt/lcg/MCGenerators/heputils/1.3.2-8ac2c/x86_64-centos7-gcc8-opt/.post-install.sh
Replacing /build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/install -> /opt/lcg ...
  Installing : mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt-1.0.0-96.noarc   6/7 
Checking for post install in /opt/lcg/MCGenerators/mcutils/1.3.3-f0b2f/x86_64-centos7-gcc8-opt
Invoking post install /opt/lcg/MCGenerators/mcutils/1.3.3-f0b2f/x86_64-centos7-gcc8-opt/.post-install.sh
Replacing /build/jenkins/workspace/lcg_release_tar/BUILDTYPE/Release/COMPILER/gcc8binutils/LABEL/centos7/install -> /opt/lcg [or ] ...
  Installing : RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt-1.0.0-1.noarc   7/7 
Setting up links under "/opt/lcg" using "http://cern.ch/lcgpackages/tarFiles/releases/LCG_96b_x86_64-centos7-gcc8-opt.txt"
Creating link: /opt/lcg/LCG_96b/sqlite/3280000/x86_64-centos7-gcc8-opt -> ../../../sqlite/3280000-472f2/x86_64-centos7-gcc8-opt
Creating link: /opt/lcg/LCG_96b/MCGenerators/mcutils/1.3.3/x86_64-centos7-gcc8-opt -> ../../../../MCGenerators/mcutils/1.3.3-f0b2f/x86_64-centos7-gcc8-opt
Creating link: /opt/lcg/LCG_96b/MCGenerators/heputils/1.3.2/x86_64-centos7-gcc8-opt -> ../../../../MCGenerators/heputils/1.3.2-8ac2c/x86_64-centos7-gcc8-opt
Creating link: /opt/lcg/LCG_96b/Boost/1.70.0/x86_64-centos7-gcc8-opt -> ../../../Boost/1.70.0-eebf1/x86_64-centos7-gcc8-opt
Creating link: /opt/lcg/LCG_96b/Python/2.7.16/x86_64-centos7-gcc8-opt -> ../../../Python/2.7.16-e553a/x86_64-centos7-gcc8-opt
  Verifying  : gcc_8.2.0_x86_64_centos7-3.0.0-1.noarch                      1/7 
  Verifying  : Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt-1.0.0-96.noarch   2/7 
  Verifying  : sqlite-472f2_3280000_x86_64_centos7_gcc8_opt-1.0.0-96.noar   3/7 
  Verifying  : heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt-1.0.0-95.noar   4/7 
  Verifying  : mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt-1.0.0-96.noarc   5/7 
  Verifying  : Python-e553a_2.7.16_x86_64_centos7_gcc8_opt-1.0.0-96.noarc   6/7 
  Verifying  : RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt-1.0.0-1.noarc   7/7 

Installed:
  RPMTestProject_1.0.0_x86_64-centos7-gcc8-opt.noarch 0:1.0.0-1                 

Dependency Installed:
  Boost-eebf1_1.70.0_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96                  
  Python-e553a_2.7.16_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96                 
  gcc_8.2.0_x86_64_centos7.noarch 0:3.0.0-1                                     
  heputils-8ac2c_1.3.2_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-95                
  mcutils-f0b2f_1.3.3_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96                 
  sqlite-472f2_3280000_x86_64_centos7_gcc8_opt.noarch 0:1.0.0-96                

Complete!

And then I see the following in the image:

[bash][atlas]:workdir > ls -l /opt/lcg/
total 28
drwxr-xr-x 3 root root 4096 Mar 23 18:32 Boost
drwxr-xr-x 6 root root 4096 Mar 23 18:32 LCG_96b
drwxr-xr-x 4 root root 4096 Mar 23 18:32 MCGenerators
drwxr-xr-x 3 root root 4096 Mar 23 18:32 Python
drwxr-xr-x 4 root root 4096 Dec 13 14:20 binutils
drwxr-xr-x 1 root root 4096 Mar 23 18:32 gcc
drwxr-xr-x 3 root root 4096 Mar 23 18:32 sqlite
[bash][atlas]:workdir > ls -l /opt/lcg/LCG_96b/
total 24
drwxr-xr-x 3 root root 4096 Mar 23 18:32 Boost
-rw-r--r-- 1 root root  301 Mar 23 18:32 LCG_externals_x86_64-centos7-gcc8-opt.txt
-rw-r--r-- 1 root root  245 Mar 23 18:32 LCG_generators_x86_64-centos7-gcc8-opt.txt
drwxr-xr-x 4 root root 4096 Mar 23 18:32 MCGenerators
drwxr-xr-x 3 root root 4096 Mar 23 18:32 Python
drwxr-xr-x 3 root root 4096 Mar 23 18:32 sqlite
[bash][atlas]:workdir > ls -l /opt/lcg/LCG_96b/Boost/1.70.0/
total 0
lrwxrwxrwx 1 root root 51 Mar 23 18:32 x86_64-centos7-gcc8-opt -> ../../../Boost/1.70.0-eebf1/x86_64-centos7-gcc8-opt
[bash][atlas]:workdir > more /opt/lcg/LCG_96b/LCG_externals_x86_64-centos7-gcc8-opt.txt
PLATFORM: x86_64-centos7-gcc8-opt
VERSION: 96b
COMPILER: GNU 8.3.0
sqlite; 472f2; 3280000; ./sqlite/3280000/x86_64-centos7-gcc8-opt; 
Boost; eebf1; 1.70.0; ./Boost/1.70.0/x86_64-centos7-gcc8-opt; Python-2.7.16-e553a
Python; e553a; 2.7.16; ./Python/2.7.16/x86_64-centos7-gcc8-opt; sqlite-3280000-472f2
[bash][atlas]:workdir > more /opt/lcg/LCG_96b/LCG_generators_x86_64-centos7-gcc8-opt.txt 
PLATFORM: x86_64-centos7-gcc8-opt
VERSION: 96b
COMPILER: GNU 8.3.0
mcutils; f0b2f; 1.3.3; ./MCGenerators/mcutils/1.3.3/x86_64-centos7-gcc8-opt; heputils-1.3.2-8ac2c
heputils; 8ac2c; 1.3.2; ./MCGenerators/heputils/1.3.2/x86_64-centos7-gcc8-opt; 
[bash][atlas]:workdir >

I.e. It "sort of" works. 😉

As long as we synchronise how TDAQ sets up its post-install script with what we do, this could work...

Also pinging @akazarov, @fwinkl, @wlampl and @emoyse. For this to work, we'll need to make sure that text files like this will continue to be published by SFT on their "repository webpage".

Edited by Frank Winklmeier

Merge request reports