getPackagesDirs behaviour when requested several versions of the same package

We found that getPackagesDirs returns the latest version of a package, but fails if does not find a previous version in some situations.

This is an issue for us, when retrieving the needed packages for generating a subset of /cvmfs with subcvmfs-builder-pipeline) (as required for HPCs without outbound connectivity) we copy only that reported version, but when running it fails because does not find previous versions.

As an example, we are relying on getPackagesDirs to obtain the list of packages required for Gauss:

When running on lxplux:

>>> data_dirs = getPackagesDirs("Gauss","v56r9","x86_64_v2-centos7-gcc11-opt")
>>> data_dirs
{'FieldMap': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/FieldMap/v8r1', 'ParamFiles': '/cvmfs/lhcb.cern.ch/lib/lhcb/PARAM/ParamFiles/v8r59', 'PRConfig': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/PRConfig/v1r85', 'RawEventFormat': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/RawEventFormat/v1r10p1', 'TCK/HltTCK': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/TCK/HltTCK/v3r19p29', 'Geant4Files': '/cvmfs/lhcb.cern.ch/lib/lhcb/PARAM/Geant4Files/v106r1', 'AppConfig': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/AppConfig/v3r449', 'BcVegPyData': '/cvmfs/lhcb.cern.ch/lib/lhcb/PARAM/BcVegPyData/v4r1', 'LamarrData': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/LamarrData/v3r0', 'Det/GDMLData': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/Det/GDMLData/v1r1', 'GenXiccData': '/cvmfs/lhcb.cern.ch/lib/lhcb/PARAM/GenXiccData/v3r3', 'Gen/PGunsData': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/Gen/PGunsData/v2r5', 'Gen/MadgraphData': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/Gen/MadgraphData/v20903r7p7', 'Gen/PowhegboxData': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/Gen/PowhegboxData/v3744r7p0', 'Gen/DecFiles': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/Gen/DecFiles/v32r34', 'LHAPDFSets': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/LHAPDFSets/v62r3p1', 'MIBData': '/cvmfs/lhcb.cern.ch/lib/lhcb/PARAM/MIBData/v3r2p1', 'Vis/XmlVis': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/Vis/XmlVis/v2r0', 'FastCaloSimData': '/cvmfs/lhcb.cern.ch/lib/lhcb/DBASE/FastCaloSimData/v1r1'}

In this case it is reported the FieldMap/v8r1, but when testing and running in a container with only that version, it fails (cannot find data package FieldMap v5r*):

>>> getPackagesDirs("Gauss","v56r9","x86_64_v2-centos7-gcc11-opt")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/cvmfs/lhcb.cern.ch/lhcbdirac/versions/v12.0.0a20-1745900024/Linux-x86_64/lib/python3.11/site-packages/LbEnv/ProjectEnv/lookup.py", line 611, in getPackagesDirs
    out.update((p, findDataPackage(p, v)) for p, v in packages)
  File "/cvmfs/lhcb.cern.ch/lhcbdirac/versions/v12.0.0a20-1745900024/Linux-x86_64/lib/python3.11/site-packages/LbEnv/ProjectEnv/lookup.py", line 611, in <genexpr>
    out.update((p, findDataPackage(p, v)) for p, v in packages)
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/lhcb.cern.ch/lhcbdirac/versions/v12.0.0a20-1745900024/Linux-x86_64/lib/python3.11/site-packages/LbEnv/ProjectEnv/lookup.py", line 497, in findDataPackage
    raise MissingDataPackageError(name, version, path)
LbEnv.ProjectEnv.lookup.MissingDataPackageError: cannot find data package FieldMap v5r* in /cvmfs/lhcb.cern.ch/lib/lhcb

The issue might come because the manifest includes several versions:

$ grep FieldMap /cvmfs/lhcb.cern.ch/lib/lhcb/GAUSS/GAUSS_v56r9/InstallArea/x86_64_v2-centos7-gcc11-opt/manifest.xml
    <package name="FieldMap" version="*" />
    <package name="FieldMap" version="5.0" />

lb-prod-run as getPackagesDirs fails in the process if does not find a previous version, although only FieldMap/v8r1 was reported by getPackagesDirs

(base) Apptainer> prmon --interval 60 --filename prmon_00285766_00050616_1.txt --json-summary prmon_00285766_00050616_1.json -- lb-run --siteroot=/cvmfs/lhcb.cern.ch/lib/ -c x86_64_v2-centos7-gcc11-opt --use=AppConfig.v3r449 --use=Gen/DecFiles.v32r34 --use=ProdConf Gauss/v56r9 gaudirun.py -T '$APPCONFIGOPTS/Gauss/Beam6500GeV-md100-2017-nu1.6.py' '$APPCONFIGOPTS/Gauss/EnableSpillover-25ns.py' '$APPCONFIGOPTS/Gauss/DataType-2017.py' '$APPCONFIGOPTS/Gauss/RICHRandomHits.py' '$DECFILESROOT/options/11102202.py' '$APPCONFIGOPTS/Gauss/SplitSim-GammaConversionFilter.py' '$APPCONFIGOPTS/Gauss/GammaConversionFilter_ExtendZ.py' '$LBPYTHIA8ROOT/options/Pythia8.py' '$APPCONFIGOPTS/Gauss/G4PL_FTFP_BERT_EmOpt2.py' '$APPCONFIGOPTS/Persistency/BasketSize-10.py' '' prodConf_00285766_00050616_1.py
WARNING:lb-run:Decided best container to use is None
WARNING:lb-run:trying old SetupProject (cannot find data package FieldMap v5r* in /cvmfs/lhcb.cern.ch/lib/lhcb:/cvmfs/lhcb.cern.ch/lib/lcg/releases:/cvmfs/lhcb.cern.ch/lib/lcg/app/releases:/cvmfs/lhcb.cern.ch/lib/lcg/external:/cvmfs/lhcb.cern.ch/lib/contrib:/cvmfs/lhcb.cern.ch/lib/cmake:/cvmfs/lhcb.cern.ch/lib/lhcb)
ALWAYS:SetupProject:Configuring Gauss v56r9 from /cvmfs/lhcb.cern.ch/lib/lhcb/GAUSS/GAUSS_v56r9
ERROR:SetupProject:#CMT---> Warning: package AppConfig v3r449  not found (requested by cmt_standalone)
#CMT---> Warning: package DecFiles v32r34 Gen not found (requested by cmt_standalone)
#CMT---> Warning: package ProdConf v*  not found (requested by cmt_standalone)
#CMT---> Warning: package GaussSys *  not found (requested by cmt_standalone)

Please let us know if we are missing something, thank you for your help.