Require Python 2.7 until Python 3 migration is complete

On a machine where both Python 2.7 and Python 3.6 are installed, the Gaudi build scripts can fail because Python 3.6 gets selected by default, and we are not yet compatible with that. This MR addresses that in the short term by explicitly asking for Python 2.7.

In the long term, the best option is to make the build scripts compatible with Python 3, which should happen via !545 (closed) . Once we get there, the changes of this MR can be reverted.

• [2018-02-14 13:53] Validation started with lhcb-gaudi-merge#414
• [2018-02-14 18:37] Validation started with lhcb-gaudi-merge#418
• [2018-02-15 00:05] Validation started with lhcb-gaudi-head-py3#65
• [2018-02-15 00:05] Validation started with lhcb-tdr-test#60
• [2018-02-15 00:07] Validation started with lhcb-gaudi-head#1769
• [2018-02-16 00:04] Validation started with lhcb-gaudi-head-py3#66
• [2018-02-16 00:04] Validation started with lhcb-tdr-test#61
• [2018-02-16 00:06] Validation started with lhcb-gaudi-head#1770
• [2018-02-17 00:03] Validation started with lhcb-gaudi-head-py3#67
• [2018-02-17 00:05] Validation started with lhcb-tdr-test#62
• [2018-02-17 00:13] Validation started with lhcb-gaudi-head#1771
• [2018-02-18 00:03] Validation started with lhcb-gaudi-head-py3#68
• [2018-02-18 00:06] Validation started with lhcb-tdr-test#63
• [2018-02-18 00:08] Validation started with lhcb-gaudi-head#1772
• [2018-02-18 00:47] Validation started with lhcb-tdr-test#63
• [2018-02-18 00:49] Validation started with lhcb-gaudi-head-py3#68
• [2018-02-18 01:03] Validation started with lhcb-gaudi-head#1772
• [2018-02-18 01:26] Validation started with lhcb-tdr-test#63
• [2018-02-18 01:45] Validation started with lhcb-gaudi-head-py3#68
• [2018-02-18 01:47] Validation started with lhcb-gaudi-head#1772
• [2018-02-19 00:03] Validation started with lhcb-gaudi-head-py3#69
• [2018-02-19 00:05] Validation started with lhcb-tdr-test#64
• [2018-02-19 00:13] Validation started with lhcb-gaudi-head#1773
• [2018-02-19 08:14] Validation started with lhcb-gaudi-head#1774
• [2018-02-20 00:04] Validation started with lhcb-gaudi-head-py3#70
• [2018-02-20 00:07] Validation started with lhcb-tdr-test#65
• [2018-02-20 00:10] Validation started with lhcb-gaudi-head#1775
• [2018-02-21 00:03] Validation started with lhcb-gaudi-head-py3#71
• [2018-02-21 00:07] Validation started with lhcb-tdr-test#66
• [2018-02-21 00:08] Validation started with lhcb-gaudi-head#1776
• [2018-02-22 01:03] Validation started with lhcb-gaudi-head-py3#72
• [2018-02-22 01:07] Validation started with lhcb-tdr-test#67
• [2018-02-22 01:08] Validation started with lhcb-gaudi-head#1777
• [2018-02-23 01:03] Validation started with lhcb-gaudi-head-py3#73
• [2018-02-23 01:06] Validation started with lhcb-tdr-test#68
• [2018-02-23 01:09] Validation started with lhcb-gaudi-head#1778
• [2018-02-24 01:03] Validation started with lhcb-gaudi-head-py3#74
• [2018-02-24 01:10] Validation started with lhcb-gaudi-head#1779
• [2018-02-24 01:15] Validation started with lhcb-tdr-test#69
• [2018-02-25 01:04] Validation started with lhcb-gaudi-head-py3#75
• [2018-02-25 01:06] Validation started with lhcb-tdr-test#70
• [2018-02-25 01:08] Validation started with lhcb-gaudi-head#1780
• [2018-02-26 01:03] Validation started with lhcb-gaudi-head-py3#76
• [2018-02-26 01:06] Validation started with lhcb-tdr-test#71
• [2018-02-26 01:09] Validation started with lhcb-gaudi-head#1781
• [2018-02-27 01:03] Validation started with lhcb-gaudi-head-py3#77
• [2018-02-27 01:07] Validation started with lhcb-tdr-test#72
• [2018-02-27 01:11] Validation started with lhcb-gaudi-head#1782
• [2018-02-28 01:03] Validation started with lhcb-gaudi-head-py3#78
• [2018-02-28 01:06] Validation started with lhcb-tdr-test#73
• [2018-02-28 01:12] Validation started with lhcb-gaudi-head#1783
• [2018-03-01 00:03] Validation started with lhcb-gaudi-head-py3#79
• [2018-03-01 00:05] Validation started with lhcb-tdr-test#74
• [2018-03-01 00:12] Validation started with lhcb-gaudi-head#1784
• [2018-03-01 13:05] Validation started with lhcb-upgrade-hackathon#40
• [2018-03-02 00:05] Validation started with lhcb-gaudi-head-py3#80
• [2018-03-02 00:05] Validation started with lhcb-upgrade-hackathon#41
• [2018-03-02 00:06] Validation started with lhcb-tdr-test#75
• [2018-03-02 00:09] Validation started with lhcb-gaudi-head#1785
• [2018-03-02 09:45] Validation started with lhcb-upgrade-hackathon#42
• [2018-03-03 00:03] Validation started with lhcb-gaudi-head-py3#81
• [2018-03-03 00:04] Validation started with lhcb-upgrade-hackathon#43
• [2018-03-03 00:05] Validation started with lhcb-tdr-test#76
• [2018-03-03 00:10] Validation started with lhcb-gaudi-head#1786
• [2018-03-03 09:56] Validation started with lhcb-gaudi-head#1787
• [2018-03-04 00:04] Validation started with lhcb-upgrade-hackathon#44
• [2018-03-04 00:05] Validation started with lhcb-gaudi-head-py3#82
• [2018-03-04 00:05] Validation started with lhcb-tdr-test#77
• [2018-03-04 00:06] Validation started with lhcb-gaudi-head#1788
• [2018-03-05 00:04] Validation started with lhcb-gaudi-head-py3#83
• [2018-03-05 00:08] Validation started with lhcb-upgrade-hackathon#45
• [2018-03-05 00:10] Validation started with lhcb-gaudi-head#1789
• [2018-03-05 00:10] Validation started with lhcb-tdr-test#78
• [2018-03-06 00:03] Validation started with lhcb-gaudi-head-py3#84
• [2018-03-06 00:03] Validation started with lhcb-upgrade-hackathon#46
• [2018-03-06 00:06] Validation started with lhcb-tdr-test#79
• [2018-03-06 00:08] Validation started with lhcb-gaudi-head#1790
• [2018-03-07 00:04] Validation started with lhcb-gaudi-head-py3#85
• [2018-03-07 00:05] Validation started with lhcb-upgrade-hackathon#47
• [2018-03-07 00:06] Validation started with lhcb-tdr-test#80
• [2018-03-07 00:12] Validation started with lhcb-gaudi-head#1791
• [2018-03-07 14:24] Validation started with lhcb-tdr-test#81
• [2018-03-08 00:04] Validation started with lhcb-upgrade-hackathon#48
• [2018-03-08 00:04] Validation started with lhcb-gaudi-head-py3#86
• [2018-03-08 00:05] Validation started with lhcb-tdr-test#82
• [2018-03-08 00:10] Validation started with lhcb-gaudi-head#1792
• [2018-03-09 00:05] Validation started with lhcb-gaudi-head-py3#87
• [2018-03-09 00:07] Validation started with lhcb-upgrade-hackathon#49
• [2018-03-09 00:08] Validation started with lhcb-tdr-test#83
• [2018-03-09 00:10] Validation started with lhcb-gaudi-head#1793
• [2018-03-10 00:04] Validation started with lhcb-upgrade-hackathon#50
• [2018-03-10 00:06] Validation started with lhcb-gaudi-head-py3#88
• [2018-03-10 00:08] Validation started with lhcb-tdr-test#84
• [2018-03-10 00:11] Validation started with lhcb-gaudi-head#1794
• [2018-03-10 00:46] Validation started with lhcb-tdr-test#84
• [2018-03-11 00:04] Validation started with lhcb-upgrade-hackathon#51
• [2018-03-11 00:06] Validation started with lhcb-gaudi-head-py3#89
• [2018-03-11 00:07] Validation started with lhcb-tdr-test#85
• [2018-03-11 00:11] Validation started with lhcb-gaudi-head#1795
• [2018-03-12 00:05] Validation started with lhcb-gaudi-head-py3#90
• [2018-03-12 00:06] Validation started with lhcb-upgrade-hackathon#52
• [2018-03-12 00:07] Validation started with lhcb-tdr-test#86
• [2018-03-12 00:15] Validation started with lhcb-gaudi-head#1796
• [2018-03-13 00:05] Validation started with lhcb-gaudi-head-py3#91
• [2018-03-13 00:06] Validation started with lhcb-upgrade-hackathon#53
• [2018-03-13 00:08] Validation started with lhcb-tdr-test#87
• [2018-03-13 00:10] Validation started with lhcb-gaudi-head#1797
• [2018-03-14 00:04] Validation started with lhcb-gaudi-head-py3#92
• [2018-03-14 00:04] Validation started with lhcb-upgrade-hackathon#54
• [2018-03-14 00:09] Validation started with lhcb-tdr-test#88
• [2018-03-14 00:10] Validation started with lhcb-gaudi-head#1798
• [2018-03-15 00:04] Validation started with lhcb-gaudi-head-py3#93
• [2018-03-15 00:07] Validation started with lhcb-tdr-test#89
• [2018-03-15 00:19] Validation started with lhcb-gaudi-head#1799

added build system label

changed the description

Wouldn't it make more sense to pick out the relevant commits? We don't need the entirety of !545 (closed) to fix this, no?

cc @apearce

I am not sure if I fully understand. Do you suggest migrating just the Python files that intervene in the build system to Python 3?

In this case, my understanding is that the CMake configuration and the build would succeed, but the resulting Gaudi build would be unusable (e.g. unit tests would fail due to Python 3 compatibility errors, as CTest uses the Python interpreter that was selected by CMake).

I didn't realize this. I thought at run time python comes from LCG. What is an example for a unit test that'll fail?

One problem I've come across is that the LCG Python 3 builds don't supply a binary called python. Then when there's a hashbang like #!/usr/bin/env python, the system Python is picked up which is Python 2. This can cause strange behavior because there are Python 2/3 specific bindings in the C++.

I guess the problem this MR addresses is when there is a python binary, but it's Python 3.

@apearce We are probably not discussing the same issue. On this system, there is a /usr/bin/python symlink, which points to /usr/bin/python2.7, but it is not what CMake will pick if you run find_package(PythonInterp) without specifying a version. CMake will actually do the "right" thing and scan all the available Python versions to pick the most recent one that fits the provided version requirement.

@rmatev I would expect any test which runs a Python3-incompatible file (like, say, gaudirun.py) to fail. But this is an expectation, not a reality-checked fact, so let me try things out a bit and see what happens...

marked as a Work In Progress

The reality-checked version, as it turns out, is worse than I thought.

Unless told to do otherwise, CMake's FindPythonInterp module picks the most recent version of Python available, in my case Python 3.6. The corresponding Python binary path (/usr/bin/python3.6) is then copy-pasted into all files which get automatically generated by CMake, including some of the CTest configuration.

However, as @apearce pointed out, other parts of the Gaudi test suite are hardcoded to invoke python (which on my system is a symlink to /usr/bin/python2.7) instead. Therefore, the run-time test configuration actually ends up unpredictably mixing python 2.7 and python 3.6, which obviously does not end well...

---

If you are curious and want to experiment on your own...

• Here are the minimal backports from !545 (closed) that are needed for the CMake configuration to complete successfully: hgraslan/Gaudi@44bca3f1
• Here are additional backports from !545 (closed) that are needed for a build to finish without errors: hgraslan/Gaudi@e4075164

Once you get there, you will find that, for example, neither the original version of gaudirun.py nor the one from !545 (closed) fully works, because the former uses the legacy print syntax (which is not supported by Python 3) and the later uses the "flush" optional argument to the print function (which is not supported by Python 2.7's backport of the print() function).

Personally, this tends to convince me that the original motivation for this merge request is correct: we should definitely eventually move to Python 3 (by means of !545 (closed)), but it is a large undertaking which is not finished yet, and for the time being it is sufficient to fix CMake's configuration so that when both Python 2 and Python 3 are available on a system, Python 2 is selected by CMake.

unmarked as a Work In Progress

added 7 commits

• f1b2e628...fb3faa4c - 6 commits from branch gaudi:master
• d0488ad9 - Require Python 2.7 until Python 3 migration is complete

Compare with previous version

mentioned in merge request !462 (merged)

mentioned in merge request !612 (merged)

changed milestone to %v30r2

added 8 commits

• d0488ad9...cb38e5ac - 7 commits from branch gaudi:master
• 9342c246 - Require Python 2.7 until Python 3 migration is complete

Compare with previous version

approved this merge request

assigned to @clemenci

merged

mentioned in commit 73cadef6