Skip to content

PyModules: Use pip for package build/install

Frank Winklmeier requested to merge fwinkl/atlasexternals:pymodules_pip into master

Migrated to pip for external python package build and installation. The source of the packages is still

http://cern.ch/atlas-software-dist-eos/externals/PyModules

but this area is now organized as a proper PyPA repository following https://packaging.python.org/guides/hosting-your-own-index.

The advantages over the use of setuptools are:

  • Rely on pip to resolve the dependencies. We just need to make sure that the required dependencies are either available via LCG or manually added to the above repository.
  • There is no global easy-install.pth file to be maintained and we can install python modules via multiple packages (e.g. PyAnalysis, PyModules) into the same installation area without the need of a post-build merge.
  • The executable scripts created by setuptools are not optimal in our environment (see https://gitlab.com/pycqa/flake8/issues/537).

The pip install command is run with several additional options:

  • --user: This is the main "trick" to get pip install everything into our temporary installation directory (set by $PYTHONUSERBASE). The advantage over using --prefix is that pip will never try to "uninstall" outdated dependencies coming e.g. from the read-only LCG area. Instead it will simply install the newer version.
  • --index-url: Use our own repository as the one and only repository to look for packages and dependencies.
  • --disable-pip-version-check: Disables version check of the pip executable itself
  • --no-warn-script-location: Disable check that installed executables are indeed on $PATH. Since we install into a temporary directory first, this warning is always triggered.
  • --no-warn-conflicts: Without this pip would warn about any existing conflicts (e.g. within the packages from LCG).
  • --no-cache-dir: By default pip caches downloaded files and build products in ~/.cache/pip. This speeds up subsequent builds but since we have cmake to take care of that it is probably better to disable this to avoid any chance of cross-talk.

And this is how the build looks like: make.log

Merge request reports