Skip to content

fix: Add /copy site-packages to PYTHONPATH to fix XRootD's fall back to easy install (!58)

Matthew Feickert requested to merge fix/add-pythonpath-for-modern-setuptools into master

Remove restrictions on setuptools version added in MR !55 (merged). XRootD will attempt to install the Python bindings using a fall back to the deprecated easy install, and with setuptools v60.0.0+ (which removes more support of the deprecated distuils) attempting to install XRootD in a location (like /copy/local)that would place the Python bindings not on PYTHONPATH fails with something like

/usr/local/venv/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/local/venv/lib/python3.9/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/local/venv/bin/python -E -c pass
TEST FAILED: //copy/local/lib/python3.9/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    //copy/local/lib/python3.9/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.

A safe work around for this is to simply add the install location's site-packages to PYTHONPATH, which will then allow for the fall back to easy install to generate a easy-install.pth under the install location's site-packages that can be found and used (and copied later).

/usr/local/venv/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/local/venv/lib/python3.9/site-packages/setuptools/command/easy_install.py:156: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
creating xrootd.egg-info
writing manifest file 'xrootd.egg-info/SOURCES.txt'
writing manifest file 'xrootd.egg-info/SOURCES.txt'
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/pyxrootd
copying /code/xrootd/bindings/python/src/__init__.py -> build/lib.linux-x86_64-3.9/pyxrootd
creating build/lib.linux-x86_64-3.9/XRootD
copying /code/xrootd/bindings/python/libs/__init__.py -> build/lib.linux-x86_64-3.9/XRootD
creating build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/_version.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/flags.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/file.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/__init__.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/glob_funcs.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/url.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/filesystem.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/utils.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/env.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/responses.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/copyprocess.py -> build/lib.linux-x86_64-3.9/XRootD/client
copying /code/xrootd/bindings/python/libs/client/finalize.py -> build/lib.linux-x86_64-3.9/XRootD/client
running build_ext
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/code
creating build/temp.linux-x86_64-3.9/code/xrootd
creating build/temp.linux-x86_64-3.9/code/xrootd/bindings
creating build/temp.linux-x86_64-3.9/code/xrootd/bindings/python
creating build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/PyXRootDCopyProcess.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDCopyProcess.o -g
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/PyXRootDCopyProgressHandler.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDCopyProgressHandler.o -g
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/PyXRootDFile.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDFile.o -g
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/PyXRootDFileSystem.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDFileSystem.o -g
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/PyXRootDModule.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDModule.o -g
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/PyXRootDURL.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDURL.o -g
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/code/xrootd/src -I/code/build/src -I/usr/local/venv/include -I/usr/local/include/python3.9 -c /code/xrootd/bindings/python/src/Utils.cc -o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/Utils.o -g
g++ -pthread -shared -Wl,--strip-all build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDCopyProcess.o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDCopyProgressHandler.o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDFile.o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDFileSystem.o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDModule.o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/PyXRootDURL.o build/temp.linux-x86_64-3.9/code/xrootd/bindings/python/src/Utils.o -L/code/build/src -L/code/build/src/XrdCl -L/usr/local/lib -lXrdCl -lXrdUtils -ldl -o build/lib.linux-x86_64-3.9/pyxrootd/client.cpython-39-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/XRootD
creating build/bdist.linux-x86_64/egg/XRootD/client
creating build/bdist.linux-x86_64/egg/pyxrootd
byte-compiling build/bdist.linux-x86_64/egg/XRootD/__init__.py to __init__.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/_version.py to _version.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/flags.py to flags.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/file.py to file.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/__init__.py to __init__.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/glob_funcs.py to glob_funcs.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/url.py to url.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/filesystem.py to filesystem.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/utils.py to utils.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/env.py to env.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/responses.py to responses.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/copyprocess.py to copyprocess.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/XRootD/client/finalize.py to finalize.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyxrootd/__init__.py to __init__.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/pyxrootd/client.py to client.cpython-39.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying xrootd.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xrootd.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xrootd.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying xrootd.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
[91mzip_safe flag not set; analyzing archive contents...
[0m[91mpyxrootd.__pycache__.client.cpython-39: module references __file__
[0mcreating dist
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
creating /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg
Extracting xrootd-5.4.0-py3.9-linux-x86_64.egg to /copy/local/lib/python3.9/site-packages
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/__init__.py to __init__.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/__init__.py to __init__.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/_version.py to _version.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/copyprocess.py to copyprocess.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/env.py to env.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/file.py to file.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/filesystem.py to filesystem.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/finalize.py to finalize.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/flags.py to flags.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/glob_funcs.py to glob_funcs.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/responses.py to responses.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/url.py to url.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/XRootD/client/utils.py to utils.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/pyxrootd/__init__.py to __init__.cpython-39.pyc
byte-compiling /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg/pyxrootd/client.py to client.cpython-39.pyc
Adding xrootd 5.4.0 to easy-install.pth file

Installed /copy/local/lib/python3.9/site-packages/xrootd-5.4.0-py3.9-linux-x86_64.egg

This MR implements this by adding /copy/local/lib/python3.8/site-packages to PYTHONPATH to allow for XRootD to correctly install the Python bindings regardless of installation path target and setuptools version.

For more details c.f. https://github.com/scikit-hep/pyhf/issues/1277.

* Remove restrictions on setuptools versions.
   - Reverts most of MR !55
* Add `/copy/local/lib/python3.8/site-packages` to `PYTHONPATH`
to allow for XRootD to correctly install the Python bindings regardless
of installation path target and setuptools version.
   - With setuptools v60.0.0+, distuils and EasyInstall are deprecated, and
     attempting to install the Python bindings by falling back to using
     EasyInstall .pth files only works if those files exist in a site-packages
     directory under PYTHONPATH. Without adding the install path target to PYTHONPATH
     you'll get an error like:

TEST FAILED: //copy/local/lib/python3.9/site-packages/ does NOT support .pth files
bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    //copy/local/lib/python3.9/site-packages/

and your PYTHONPATH environment variable currently contains:

    ''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
  on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
  variable.  (It must then also be on PYTHONPATH whenever you run
  Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
  using one of the approaches described here:

  https://setuptools.pypa.io/en/latest/deprecated/easy_install.html#custom-installation-locations


Please make the appropriate changes for your system and try again.
Edited by Matthew Feickert

Merge request reports