Add Python 3 to CI

• Add a build and a test job using LCG 96python3
• Build on top of !545 (closed) by @apearce, rebasing and adapting where needed
• All tests passing with both python2 and python3!
• Supersedes !545 (closed) and !864 (closed)

Closes #56 (closed)

changed the description

• [2019-04-05 16:05] Validation started with lhcb-gaudi-merge#713

marked as a Work In Progress

WIP so that it does not accidentally break LHCb's nightlies (it shouldn't)

added 2 commits

• a7085603 - Use the released LCG 95a with python3
• df87b2d4 - Fix LCG version override with HEPTOOLS_VERSION

Compare with previous version

I would add only the CI jobs to start with, and keep the Python 3 compatibility changes in other MRs.

Do you mean just the changes to the gitlab ci yaml? In that case after it is merged, all pipelines will fail or at least be yellow

yes, the alternative is to use a dedicated branch that we will merge into master only once it's green

mentioned in merge request !852 (merged)

added 34 commits

• df87b2d4...e3d93abb - 17 commits from branch gaudi:master
• 2fc5244e - Remove explicit python versions
• 8cd57c69 - Be compatible with the CPython's Python 3 API.
• baaefe0f - Avoid unused variable warnings when compiling for Python 3.
• a54400ff - Use Python 3 compatible execfile.
• 33f923a0 - Only worry about loop variable clean-up in Python 2.
• 54bfb809 - Use Python 3 compatible printing.
• acae7e21 - Fix local import for Python 3
• d63adf03 - Do not use has_key (remved in Python 3).
• b89f5e21 - Make genconfuser.py Python 3 compatible
• e56c2d9f - Fixed formatting
• 541d9e60 - Decode `bytes` objects to UTF-8 `str` objects.
• 858ce4c3 - Add a python3 build stage using lcg nightlies
• 651c6cc7 - Make test result parsing Python 3 compatible
• df8c10e7 - Add CI test stage for python3 build
• fa677e2c - Fixed formatting
• 4b57ce10 - Use the released LCG 95a with python3
• 6546d072 - Fix LCG version override with HEPTOOLS_VERSION

Compare with previous version

This has been stale for a month, do you need any help?

Only 7 months left until Python 2.7 is obsolete (see #56 (closed)).

Also: Python 3 is needed to make Gaudi work in CentOS 8 (see RHEL 8 release notes).

added 241 commits

• 6546d072...72705057 - 194 commits from branch gaudi:master
• 4d011e62 - Remove explicit python versions
• 037255c5 - Be compatible with the CPython's Python 3 API.
• 666d73c7 - Avoid unused variable warnings when compiling for Python 3.
• cc0eb5b3 - Use Python 3 compatible execfile.
• b7cda031 - Only worry about loop variable clean-up in Python 2.
• f01599e4 - Use Python 3 compatible printing.
• 37398218 - Fix local import for Python 3
• 2d7e19e8 - Do not use has_key (remved in Python 3).
• 9392d6b3 - Make genconfuser.py Python 3 compatible
• 97f00a9b - Fixed formatting
• f7cf8039 - Decode `bytes` objects to UTF-8 `str` objects.
• d8b5a908 - Pass UTF-8 encoded bytes to hashlib.sha1.
• 268c5eaf - Add a python3 build stage using lcg nightlies
• 64bc8d09 - Make test result parsing Python 3 compatible
• f2ebbbac - Fix LCG version override with HEPTOOLS_VERSION
• 2febdbde - Add CI test stage for python3 build
• 2191d74d - Fixed formatting
• e677ad1b - Use Python 3 compatible exec.
• b8444006 - Use a more robust dynamic import method.
• 9ac9d251 - Do not create __init__.py files in __pycache__/.
• c0c84638 - Use Python 3 compatible metaclass syntax and function attribute accessors.
• e8523939 - Use Python 3 compatible iteration methods.
• 9b0c0614 - Do not mutate dictionaries whilst iterating over them.
• 3655c49f - Use Python 3 compatible absolute imports.
• d2288d2d - Use Python 3 compatible file reading.
• a195b3f0 - Use Python 3 compatible lists of string and number types.
• dad3920c - Use more idiomatic check for dictionary key membership.
• 5aa08291 - Use Python 3 compatible long type.
• fce9a5ab - Define __hash__ on classes that only define __eq__.
• df505169 - Use Python 3 compatible absolute imports.
• 68b8cd90 - Handle missing atexit._exithandlers.
• 769992d9 - Use fixed floating point precision for printing
• 7f1e6cb3 - Ensure prints are flushed when necessary.
• 5a688ecd - Fix CTestXML2HTML when stdout is empty
• 0f9a6731 - Use sorting method compatible with both iterators and lists.
• 9eb4bceb - Use Python 3 compatible `except` syntax.
• b6d2b543 - Stabilize order of printout in tests
• 75a393bf - Ensure pprint in Python 3 behaves like python 2
• 459ebb97 - Use Python 3 compatible __new__ call.
• 6d04b76a - Use Python 3 compatible boolean cast method.
• cb0f8dd2 - Python 3 compatible call to string strip
• f81ca753 - Don't use xrange.
• 343385a2 - Sort properties passed to the JobOptionsSvc
• 9a3ab722 - Treat files as byte streams.
• ecee9bc1 - Use Python 2.6+ compatible lambda syntax.
• a2339094 - Use Python 3 compatible exception classes.
• 0c254a7d - Use Python 3 compatible writing to binary file

Compare with previous version

changed the description

All tests are passing on both Python 2 and 3 now with rc1 of lcg 96 (final version being released now).

Once v32r1 is out we can look into merging this, which means at least testing that the LHCb and ATLAS stacks are unchanged with python 2.

changed the description

added 56 commits

• 0c254a7d...198b08d1 - 11 commits from branch gaudi:master
• d4cf1f53 - Remove explicit python versions
• 38997483 - Be compatible with the CPython's Python 3 API.
• 08fd4792 - Avoid unused variable warnings when compiling for Python 3.
• e52e82a9 - Use Python 3 compatible execfile.
• d48e0385 - Only worry about loop variable clean-up in Python 2.
• cd36145b - Use Python 3 compatible printing.
• 7bd3f09c - Fix local import for Python 3
• a5717fba - Do not use has_key (remved in Python 3).
• 5313e0c0 - Make genconfuser.py Python 3 compatible
• d586ff69 - Fixed formatting
• f93051d4 - Decode `bytes` objects to UTF-8 `str` objects.
• d51d7af3 - Pass UTF-8 encoded bytes to hashlib.sha1.
• f8916ef4 - Add a python3 build stage using lcg nightlies
• 7df39c8e - Make test result parsing Python 3 compatible
• db0e7504 - Fixed formatting
• 807ecf95 - Use Python 3 compatible exec.
• e8ba8f02 - Use a more robust dynamic import method.
• 398a89ad - Do not create __init__.py files in __pycache__/.
• 5bf58847 - Use Python 3 compatible metaclass syntax and function attribute accessors.
• cde8a1fe - Use Python 3 compatible iteration methods.
• ab601a91 - Do not mutate dictionaries whilst iterating over them.
• 7630eab5 - Use Python 3 compatible absolute imports.
• 7bdc2942 - Use Python 3 compatible file reading.
• 086e9981 - Use Python 3 compatible lists of string and number types.
• 0db4295a - Use more idiomatic check for dictionary key membership.
• 2d17ea38 - Use Python 3 compatible long type.
• 08fdf83e - Define __hash__ on classes that only define __eq__.
• f1ee67c3 - Use Python 3 compatible absolute imports.
• 1f9c1e5d - Handle missing atexit._exithandlers.
• d146767b - Use fixed floating point precision for printing
• 67f2e772 - Ensure prints are flushed when necessary.
• c32ef220 - Fix CTestXML2HTML when stdout is empty
• 32cdd133 - Use sorting method compatible with both iterators and lists.
• fd01d87c - Use Python 3 compatible `except` syntax.
• b150e0f8 - Stabilize order of printout in tests
• 30bd793d - Ensure pprint in Python 3 behaves like python 2
• af25d89e - Use Python 3 compatible __new__ call.
• de71978d - Use Python 3 compatible boolean cast method.
• 01865aa9 - Python 3 compatible call to string strip
• 1807111c - Don't use xrange.
• fe53ebb3 - Sort properties passed to the JobOptionsSvc
• 19416e28 - Treat files as byte streams.
• 86bbd84c - Use Python 2.6+ compatible lambda syntax.
• d2cbe998 - Use Python 3 compatible exception classes.
• c0eaaf01 - Use Python 3 compatible writing to binary file

Compare with previous version

added 56 commits

• c0eaaf01...f65d50dc - 11 commits from branch gaudi:master
• 5e11ea6e - Remove explicit python versions
• f80b6c77 - Be compatible with the CPython's Python 3 API.
• 3d104ded - Avoid unused variable warnings when compiling for Python 3.
• feee4706 - Use Python 3 compatible execfile.
• 484237f7 - Only worry about loop variable clean-up in Python 2.
• 49be50e6 - Use Python 3 compatible printing.
• a3921b1c - Fix local import for Python 3
• 259ab8ba - Do not use has_key (remved in Python 3).
• d439de28 - Make genconfuser.py Python 3 compatible
• 53351ede - Fixed formatting
• 4c198181 - Decode `bytes` objects to UTF-8 `str` objects.
• 49635395 - Pass UTF-8 encoded bytes to hashlib.sha1.
• 912993fd - Add a python3 build stage using lcg nightlies
• d5808e03 - Make test result parsing Python 3 compatible
• 6df40507 - Fixed formatting
• 0acc9a1b - Use Python 3 compatible exec.
• 26559b83 - Use a more robust dynamic import method.
• a3fb206a - Do not create __init__.py files in __pycache__/.
• 1fb6e738 - Use Python 3 compatible metaclass syntax and function attribute accessors.
• 4caa7f5a - Use Python 3 compatible iteration methods.
• 45a6f540 - Do not mutate dictionaries whilst iterating over them.
• 13595ae1 - Use Python 3 compatible absolute imports.
• 53de9857 - Use Python 3 compatible file reading.
• 5a044fac - Use Python 3 compatible lists of string and number types.
• 50be2632 - Use more idiomatic check for dictionary key membership.
• c7a1f8fa - Use Python 3 compatible long type.
• 5a19c87e - Define __hash__ on classes that only define __eq__.
• bcebbba2 - Use Python 3 compatible absolute imports.
• ea5965a9 - Handle missing atexit._exithandlers.
• 50bf9fee - Use fixed floating point precision for printing
• c45ea7a3 - Ensure prints are flushed when necessary.
• 2a57e5ed - Fix CTestXML2HTML when stdout is empty
• 8f136463 - Use sorting method compatible with both iterators and lists.
• bc9779f1 - Use Python 3 compatible `except` syntax.
• 401f77fe - Stabilize order of printout in tests
• 22d19c10 - Ensure pprint in Python 3 behaves like python 2
• 1068953a - Use Python 3 compatible __new__ call.
• d817dab5 - Use Python 3 compatible boolean cast method.
• d5a94513 - Python 3 compatible call to string strip
• a2908143 - Don't use xrange.
• ab52e17c - Sort properties passed to the JobOptionsSvc
• ed3f65ad - Treat files as byte streams.
• f92069dd - Use Python 2.6+ compatible lambda syntax.
• 05b8eba3 - Use Python 3 compatible exception classes.
• a5239d06 - Use Python 3 compatible writing to binary file

Compare with previous version

It would be great if we can move forward with this MR so that we can begin work in earnest on Python 3 compatibility in LHCb applications.

assigned to @clemenci

changed milestone to %v32r2

I want to second Alex. We are running out of time here. Python 2 is not going to be supported for much longer and Gaudi is the only software we use that does not support Python 3.

added lhcb-gaudi-head label

I'll test it in LHCb nightlies tonight, if everything goes fine we can merge.

added lhcb-gaudi-head-py3 label

• [2019-09-18 15:30] Validation started with lhcb-gaudi-head-py3#177
• [2019-09-19 00:07] Validation started with lhcb-gaudi-head-py3#178
• [2019-09-19 00:09] Validation started with lhcb-gaudi-head#2390
• [2019-09-19 09:05] Validation started with lhcb-gaudi-merge#819
• [2019-09-19 10:38] Validation started with lhcb-gaudi-head-py3#179
• [2019-09-19 13:18] Validation started with lhcb-gaudi-head-py3#180
• [2019-09-19 13:26] Validation started with lhcb-gaudi-head#2391
• [2019-09-20 00:03] Validation started with lhcb-gaudi-head-py3#181
• [2019-09-20 00:09] Validation started with lhcb-gaudi-head#2392
• [2019-09-20 09:33] Validation started with lhcb-gaudi-merge#824

changed the description

mentioned in merge request !545 (closed)

mentioned in merge request !864 (closed)

unmarked as a Work In Progress

changed the description

mentioned in merge request lhcb/LHCb!2129 (merged)

WIP-ing because I've seen a small impact on LHCb tests and I'd like to check again now that !950 (closed) is off.

marked as a Work In Progress

mentioned in merge request lhcb/LHCb!2130 (merged)

approved this merge request

unmarked as a Work In Progress

merged

mentioned in commit be580110

@clemenci I'm not sure it's due to this, but please look at the two new failing tests in Online, see https://lhcb-nightlies.cern.ch/nightly/compare/lhcb-gaudi-head/2392/lhcb-head/2372/

Indeed it's related. I'll fix it ASAP.

mentioned in merge request !975 (merged)

mentioned in issue #88 (closed)

mentioned in merge request !1102 (merged)