Scott Snyder authored 4 years ago and Adam Edward Barton committed 4 years ago

With python 3.8, the hacks used in pythonize_read_root_file lead to heap
corruption.  Rework to avoid this.

We're rather constrained since this has to work with both py2 and 3,
so the way it's done here is a bit roundabout and ineffecient.
Once we drop py2, this can be done much better by simply
having read_root_file return a PyBytes object

Frank Winklmeier authored 4 years ago and Walter Lampl committed 4 years ago

Delete unused fileutils, path and merge_join modules. Also delete pep8
and avn alias. The pep8 script is superseded by flake8. And the avn
alias was pointing to a non-existent script.

SetSize is being renamed to reshape.
Avoids a root deprecation warning.

When we run aclic, root produces a DSO and tries to link it.
As past of this, ROOT tries to find all the libraries that the new
DSO depends on by listing all its symbols and then walking through
all libraries on LD_LIBRARY_PATH to find a definition.
Due to a bug, cling will crash if it encounters a separate-debug
library in doing so.  Pretty much every library contains
the gprof stub __gmon_start__ as a hidden weak symbol.
Cling ends up searching for this symbol, and since it is hidden,
it will end up trying all libraries.  In Athena builds, it eventually
hits a debug library and dies.

Work around by injecting a dummy defintion of __gmon_start__.

Fixes unit test failure in master.

With some python3 versions, importing past emits deprecation warnings.

 - For python 3, one needs to use PyInstanceMethod_New to create an instance
   method, not PyMethod_New as in python 2.
 - basestring
 - map is now a generator
 - commands -> subprocess
 - it.next() -> next(it)

Updates to work with python3.

Not yet intended to be complete --- only what's needed by basic Athena
and configuration jobs.

Attila Krasznahorkay authored 6 years ago and Atlas Nightlybuild committed 6 years ago

Fallback for Char_t leaf reading in PyUtils/RootUtils.py

See merge request atlas/athena!15151

(cherry picked from commit d29db3ce [formerly 52aa1089e71ddd8b621c78a9d37763b2ef02625c])

83b7ca12 Fall back for Char_t leaf reading

Former-commit-id: 39a6653a

The pythonization of TFile::root was calling the SetSize method
on a PyROOT buffer object.  The argument that SetSize takes is the
number of elements in the buffer, where the size of each element
depends on how the buffer was created.  Here, however, we were
passing to SetSize a number of bytes.

The buffer elements we were seeing were void*, which have size 8.
So the buffer size was getting misset by a factor of 8.
This was leading to a crash in the input file peeker.

This crash was however only observed in the muRunData test of TrigUpgradeTest,
and only when it was run with --stdcmalloc and MALLOC_CHECK_=3.
It's a bit a mystery why this apparently long-standing bug hasn't 
caused problems before.



Former-commit-id: b3df5784

Removing references to PyCintex, Reflex and ROOT6Setup from PyUtils
as final cleanup after the ROOT6 transition
ROOT6Setup remains in Helpers as it is still in use by other packages
Note: Dso.py was earlier disabled and this commit does not fix it

Former-commit-id: 5e9f6d2e

2016-04-27 Walter Lampl <walter.lampl@ cern.ch>
	* rewrite of get-tag-diff.py to be cmt and cmake compatible
	* tag PyUtils-00-14-75

2016-04-12  scott snyder  <snyder@bnl.gov>

	* Tagging PyUtils-00-14-74.
	* python/RootUtils.py: Remove diagnostics.  Keep temp files open
	through the end of program execution to avoid cling caching
	issues.

2016-04-11  scott snyder  <snyder@bnl.gov>

	* Tagging PyUtils-00-14-73.
	* Still working on ATN test failure.  Seems to be related to
	ROOT-7059.

2016-04-11 Will Buttinger <will@cern.ch>
	* bin/getAMIDatasetParameters.py: improved default printout, preserve order of specified datasets, and preserve comments/blanklines from file in the inDsTxt
	* Tagging PyUtils-00-14-72
...
(Long ChangeLog diff - truncated)

	** Merge of PyUtils-00-14-45 changes
	* cmt/requirements
	* bin/icython.py
	- Remove old icython.py script
	* test/PyUtils.xml
	- Change ATN contact info to atlas-srl-Core-Tools@cern.ch
	* ChangeLog
	- Fixup for missing logs for PyUtils-00-14-34-05
	* Tagging PyUtils-00-14-34-06

2015-06-30 Alexandre Vaniachine <vaniachine@anl.gov>
	* python/AthFile/impl.py
	- backporting PyUtils-00-14-42 for ATEAM-186
	* Tagging PyUtils-00-14-34-05

2015-06-26 Alexandre Vaniachine <vaniachine@anl.gov>
	* python/RootUtils.py
	- backporting PyUtils-00-14-40 for ATEAM-182
	* Tagging PyUtils-00-14-34-03

...
(Long ChangeLog diff - truncated)

	* merged diff between PyUtils-00-14-37 PyUtils-00-14-38:
	* python\PoolFile.py (extract_streams_from_tag): workaround for
	PyROOT getattr(tree, branchname) bug in ROOT 5.34.25 for reading
	Tokens from TAG files (https://its.cern.ch/jira/browse/ATLASRECTS-1489)
	* Tagging PyUtils-00-14-34-01

2015-06-04  Marcin Nowak  <Marcin.Nowak@cern.ch>
	* bin\getTagDiff.py: improved handling of LCG and MIG tags
	* Tagging PyUtils-00-14-34

2015-06-01  Marcin Nowak  <Marcin.Nowak@cern.ch>
	* python\scripts\diff_root_files.py: fix --ignore-leaves which
	broke in the last tag
	* Tagging PyUtils-00-14-32
	* make --ignore-leaves able to ignore subbranches
	* Tagging PyUtils-00-14-33

2015-05-28  Marcin Nowak  <Marcin.Nowak@cern.ch>
	* python\scripts\diff_root_files.py: rewrote the code to
	handle differencies better
...
(Long ChangeLog diff - truncated)

	* Tagging PyUtils-00-14-27.
	* python/RootUtils.py: Don't dump transient members.

2015-04-23  Alexandre Vaniachine <vaniachine@anl.gov>

	* Tagging PyUtils-00-14-25
	* python/AthFile/impl.py: Fix for ATEAM-122: IS_DATA for BS files from SFO

2015-04-22  scott snyder  <snyder@bnl.gov>

	* Tagging PyUtils-00-14-24.
	* python/RootUtils.py (RootFileDumper): Fix for non-class
	branches.

2015-04-16 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* bin/diffTAGTree.py
	- Continue to check events that do match between TAG files
	  even when TAG files have different content or have
	  duplicates
...
(Long ChangeLog diff - truncated)

	- Fix pkgco.py for checkout of Gaudi packages
	* python/RootUtils.py
	- Enable cppyy in import_root helper
	* Tagging as PyUtils-00-14-11 (PyUtils-00-14-11)

	* bin/pkgco.py
	- Fix pkgco.py for checkout of Gaudi packages
	* python/RootUtils.py
	- Enable cppyy in import_root helper
	* Tagging as PyUtils-00-14-11

2015-01-16  Marcin Nowak  <Marcin.Nowak@cern.ch>
	* python\Dso.py: Add ROOT6 setup
	* Tagging as PyUtils-00-14-10

2014-12-22 Peter van Gemmeren  <gemmeren@anl.gov>
	* python/scripts/diff_root_files.py: Add ROOT6 setup
	* Tagging as PyUtils-00-14-09

2014-12-18 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/AmiLib.py
	- Remove duplicate results from get_pkg_info
	* python/scripts/tc_submit_tag.py
	- Fix readline history in query_project

reporting, throw proper exceptions.
Prevent AthFileLits from running AthenaMP mode (PyUtils-00-14-07)

	* python/AmiLib.py
	- Remove commented out code
	* Tagging as PyUtils-00-14-07

2014-12-18 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/AmiLib.py
	- Another big refactoring:
	- badresult improved and now returns a boolean and an error message
	- Rename dry_run to dryrun for consistency
	- Introduce PyUtilsAMIException exception, which now gets thrown
	  consistently if there are problems
	- Consolidate the functions of get_project_of_pkg, get_version_of_pkg
	  and tc_submit_tag._get_projects into single method get_pkg_info
	- Rewrite get_version_of_pkg_with_deps to use module utilities
	- Simplify find_pkg to remove callback function
	* python/scripts/tc_find_pkg.py
	* python/scripts/tc_find_tag.py
	- Embed script in try: except PyUtilsAMIException:
	* python/scripts/tc_show_clients.py
	- Update to get_pkg_info (N.B. Script is still non-functional and disabled)
...
(Long ChangeLog diff - truncated)

gen_klass of AlgTool: Remove obsolete usage of StoreGateSvc (done by AthAlgTool base now) (PyUtils-00-13-22)