This branch adds a few python modules and classes to support the
reporting of additional metadata produced in event generation. It
is also needed to fix a bug reported in AGENE-1586


Former-commit-id: a5aed9f264b0efd1c1887253754608b2f0258d7b

Former-commit-id: b3f5a0ec35ca31991113b5a74be1b380698a5bc8

This new class will support substep aware configurations through
string arguments


Former-commit-id: b73da89df2df655516065011188391f2e11f827f

Former-commit-id: cbf41cd4d5e686e1254c566e1f9d2448d4ca0605

This branch provides transform support for WebDAV TURLs. Remote input
files may be specified with davs://


Former-commit-id: 6ff72308ca080fbf0530cf62d9e5db17f35dd443

steering


Former-commit-id: 0731180d

Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346] (PyJobTransforms-03-00-31)

	* python/trfUtils.py
	- Added utility method isInteractiveEnv()
	* python/trfExe.py
	- Refactored checks for interactive environment using new method in trfUtils.
	* python/transform.py
	- Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346]
	* Tagging PyJobTransforms-03-00-31

2016-11-17 Marcelo Vogel <mavogel@cern.ch>
	* scripts/makeTrfJSONSignatures.py
	- Replace script by updated version provided by AMI team [ATLASJT-284]
	* Tagging PyJobTransforms-03-00-30

2016-11-14 Marcelo Vogel <mavogel@cern.ch>
	* python/transform.py
	- Add property trfPredata to be set from env variable at trf init
	* python/trfReports.py
	- Add trfPredata to job report under key 'trfPredata'
	- Report version 2.0.7
	* Tagging PyJobTransforms-03-00-29
...
(Long ChangeLog diff - truncated)


Former-commit-id: 0db01cfba03324f8b6939d2f979bc9b02f861660

Remove = after pre/postInclude in output of GetTfCommand.py [ATLASAMITG-99] (PyJobTransforms-03-00-24)

	* scripts/GetTfCommand.py
	- Fix bug preventing correct output for option --printOnlyCmdLine
	* python/trfAMI.py
	- Remove '=' after pre/postInclude in _argsToString() [ATLASAMITG-99]
	* Tagging PyJobTransforms-03-00-24

2016-09-09 Marcelo Vogel <mavogel@cern.ch>
	* python/transform.py
	- Configure Event Service job via a special argument
	  eventService=True, which expands to other parameters [ATLASJT-320]
	* python/trfMPTools.py
	- Add switch to turn on/off output file checks in athenaMPOutputHandler()
	* python/trfExe.py
	- Turn on/off output file checks in athenaMP if eventService=False/True
	* test/test_trfMPTools.py
	- Update interface of athenaMPOutputHandler() in unittest
	* Tagging PyJobTransforms-03-00-23

2016-08-29 Umesh Worlikar <umesh.worlikar@cern.ch>
        * python/trfArgClasses.py
...
(Long ChangeLog diff - truncated)


Former-commit-id: e4f14ea885a883e8def88d064f1eb8d5e538e37d

Changed athenaMP default strategy to SharedQueue [ATLASJT-341]. ReportVersion 2.0.6 (PyJobTransforms-03-00-22)

	* python/trfExe.py
	- Change athenaMP default strategy to SharedQueue [ATLASJT-341].
	* Tagging PyJobTransforms-03-00-22

2016-08-22 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfExe.py
	- add preExecute and validation start and stop times
	* python/trfReports.py
	- reportVersion 2.0.6 cpu- and wallTimes for preExe, postExe, val and
	total

2016-08-19 Marcelo Vogel <mavogel@cern.ch>
	* python/trfReports.py
	- Include processedEvents in job report if not null
	* Tagging PyJobTransforms-03-00-21

2016-08-19 Marcelo Vogel <mavogel@cern.ch>
	* python/trfArgClasses.py
	- Add new argument class, argEVNTFile
	* scripts/ValidateFiles_tf.py
...
(Long ChangeLog diff - truncated)


Former-commit-id: b40505642679cdb65fce3809b4e01f6f588d0202

Fixed bug that enabled athenaMP when expectedEvents < N cores. [ATLASJT-337] (PyJobTransforms-03-00-20)

	* python/trfExe.py
	- Fix bug that was preventing to turn off athenaMP when
	- expectedEvents < N cores. [ATLASJT-337]
	* Tagging PyJobTransforms-03-00-20

2016-07-27 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* ATLASJT-335 fixed
	* python/trfArgClasses.py
	- Add a new "dumpvalue" property to argSubstepSteering to hold the alias
	  value used to set the steering and dump that if requested
	* python/trfUtils.py
	- In dumpJSON and dumpPickle use the "dumpvalue" method, if the argument
	  supports it
	* Tagging PyJobTransforms-03-00-19

2016-07-25 Marcelo Vogel <mavogel@cern.ch>
	* python/trfArgs.py
	- Fix bug in getExtraDPDList. Executor for NTUP_MUONCALIB was added twice
	* Tagging PyJobTransforms-03-00-18

...
(Long ChangeLog diff - truncated)


Former-commit-id: 7abefd9c494b5338b7b9cb09c273a2a366d60e9c

	* python/trfArgClasses.py
	- Correct argBSFile.selfMerge to return the executor properly
	* Tagging PyJobTransforms-03-00-13

2016-06-21 Umesh Worlikar <umesh.worlikar@cern.ch>
        * python/transform.py
        - transform.execute() now ensures validate() in case of exception raised
          during execute() and postExecute(). This fixes missing 'bad_alloac' error
          in jobReport [ATLASJT-303]


Former-commit-id: 3a6ba277646497e13dd4e2016855b1f6e36ed1d5

Fixed typo in python/trfExe.py (ATLASJT-316) and updated tests that were failing in RTT (PyJobTransforms-03-00-11)

2016-06-10 Marcelo Vogel <mavogel@cern.ch>
	* python/trfExe.py
	- Fix typo 'ahtenaopts' to 'athenaopts'
	* test/test_TAGMerge_tf.py
	- Remove test on metadata (beam_type) which is no longer included in jobReport.json
	* test/test_RAWMerge_tf.py
	- Remove tests on metadata which is no longer included in jobReport.json
	  (conditions_tag, geometry and beam_type)
	* Tagging PyJobTransforms-03-00-11

2016-06-03 Marcelo Vogel <mavogel@cern.ch>
	* python/trfArgs.py
	- Add missing nargs='+' option to --triggerConfig
	* Tagging PyJobTransforms-03-00-10

2016-05-31 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfExe.py
	- Move disableMP argument from transformExecutor class to athenaExecutor class (ATLASJT-318)
	* python/trfArgClasses.py
	- Update selfMerge methods for POOL files to use disableMP=True in executor
...
(Long ChangeLog diff - truncated)


Former-commit-id: 6ecf15c0a774d51cd31ef4760c9f75fc83f7e05b

2016-05-06  scott snyder  <snyder@bnl.gov>
	* Tagging PyJobTransforms-03-00-05.
	* test/test_trfReports.py, test/test_trfArgs.py: Update for last
	changes.

2016-04-29 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfReports.py
	- Remove the ~useless "argValues" section from the job report
	- Remove the ~useless 'integrity' and 'file_type'sections from subfile reports
	- Update report number to 2.0.0

2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfUtils.py
	- Minor tweaks on reported environment variables for CMake builds

2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfUtils.py
	- Update asetupReport to use "lstags" (works for both cmt and cmake)

2016-04-27 Graeme Stewart <graeme.andrew.stewart@cern.ch>
...
(Long ChangeLog diff - truncated)


Former-commit-id: 00312877a3f91ecfa3d4bc19d289cf1416509530

- Specifically to test fork after first event for memory savings
- Also collects many other minor fixes
- EXCLUDES lightweight output file metadata patch (r731518) (PyJobTransforms-02-03-76-01)

wwwpackage PyJobTransforms
2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	** Merge in trunk changes from r740535

2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	** Manually patch in trunk fixes athenaMPMergeTargetSize handling (ATLASJT-298)
	*** WARNING - Do not attempt to merge r731518 onto this branch ***

2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	** Merge in trunk changes, excluding lightweight metadata fixes

2016-04-11 Azzah Alshehri<azzah.aziz.alshehri@cern.ch>
	** Change the self merge method to save the merge information
	   * python/trfExe.py
	   - Change the stanalone variable 'myMerger'inside the method to  a class variable.
	   - Save the 'merge executor instance' you get back in the athenaExecutor _smartMerge method.
	   - Add an accessor to the base executor class, that returns the self._myMerge value.
	   * python/trfArgClasses.py
	   - Re-set 'myMerger' to the real merger executor in the actual merge.
	   - Return myMerger for each of the specific file mergers method.
...
(Long ChangeLog diff - truncated)

	- Add a new option --athenaMPStrategy to set the MP event reading strategy
	* python/trfExe.py
	- Handle athenaMPStrategy setting
	* python/trfJobOptions.py
	- Modify job options to only se athenaMPStrategy if the default is True
	* Tagging PyJobTransforms-02-03-64 (PyJobTransforms-02-03-64)

	* python/trfArgs.py
	- Add a new option --athenaMPStrategy to set the MP event reading strategy
	* python/trfExe.py
	- Handle athenaMPStrategy setting
	* python/trfJobOptions.py
	- Modify job options to only se athenaMPStrategy if the default is True
	* Tagging PyJobTransforms-02-03-64

2015-09-29 Graeme Stewart for Frank Volkmer <volkmer@cern.ch>
	* python/performanceDataUploader.py
	- Remove this - no longer used in Run2
	* python/transform.py
	- Update TRF_UNKOWN (sic) error code to TRF_UNKNOWN
	* python/trfExe.py
	- Set self._athenaMPStrategy to 'FileScheduling' if n_inputs = n_workers
	  (see ATLASRECTS-2452 and ATLASJT-269)
	* python/trfJobOptions.py
	- Convert executor _athenaMPStrategy to MP job option
	* Tagging PyJobTransforms-02-03-63

...
(Long ChangeLog diff - truncated)

	* python/transform.py
	- Switch off text, pilot.pickle and classic file reports
	* python/trfExitCodes.py
	- New TRF_EXTERNAL code
	* python/trfDecorators.py
	- Catch IOError from subprocess in timelimited decorator and handle with TRF_EXTERNAL
	* python/trfExe.py
	- Ensure that DAOD outputs for reduction are type 'AOD' (not 'aod' - was causing a
	  failure in MP derivation jobs)
	* python/trfReports.py
	- Suppress messages when /etc/machinefeatures/hs06 is missing
	* Tagging PyJobTransforms-02-03-62

2015-09-08 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfArgClasses.py
	- Append G4Version to retrieveKeys for HITSFiles
	* python/trfFileUtils.py
	- Get G4Version information from metadata in AthenaLiteFileInfo

2015-08-27 Lisa Errenst <lisa.errenst@cern.ch>
...
(Long ChangeLog diff - truncated)

	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
	* scripts/makeTrfJSONSignatures.py
	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
	  an abort when using __import__
	* test/test_trfArgs.py
	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
	  in a different project, this a problem in ATN during full builds)
	* Tagging PyJobTransforms-02-03-60 (PyJobTransforms-02-03-60)

	* python/trfArgs.py
	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
	* scripts/makeTrfJSONSignatures.py
	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
	  an abort when using __import__
	* test/test_trfArgs.py
	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
	  in a different project, this a problem in ATN during full builds)
	* Tagging PyJobTransforms-02-03-60

2015-07-08 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfReports.py
	- add HEPSPEC to machine report. Report version 1.0.5

2015-07-03 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- Refactor handling of end of transform to always try to kill stray children
	* python/trfArgClasses.py
	- Move _mergeArgs methog from argAthenaFile to argFile so that argNTUP has access
	* python/trfArgs.py
...
(Long ChangeLog diff - truncated)

	- Add new auxiliaryFile property to argFile, signaling that a file should not
	  be validated or appear in the jobReport
	* python/trfExe.py
	- Add memMonitor property to scriptExecutor, that will try to use the external
	  MemoryMonitor binary to memory monitor the script executor's child process
	- MemoryMonitor is launched immediately after the scriptExecutor's child process
	  and the memory summary JSON file parsed afterwards and added to the executor
	  memory report
	- Deprecate the perfMonFile property of athenaExecutor and stop parsing any
	  perfmon outputs
	- For DBRelease files, dereference all soft links, allowing for use with the
	  pre-release DBRelease-pcache-current.tar.gz link
	- Change suffix on multiple-merged outputs after AthenaMP has run to _N (as
	  expected by PanDA)
	* python/trfFileUtils.py
	- Add inpFileInterestingKeys (not yet used)
	* python/trfReports.py
	- Update report version number to 1.0.1
	- Modify top level report 'resource' entry to have entries per executor
	  with MemoryMonitor summary information and a separate entry for the
	  transform and its externals
	- Add machine report information to resource entry, with information
	  also parsed from /proc/cpuinfo
	* python/trfUtils.py
	- Add small debug message when setting up a DBRelease from CVMFS
	* python/trfValidation.py
	- Ensure that auxilliary files are never validated
	* test/test_trfUtilsDBRelease.py
	- Add new test for DBRelease-pcache-current.tar.gz link
	* Tag PyJobTransforms-02-03-48 (PyJobTransforms-02-03-48)

	* python/trfArgClasses.py
	- Add new auxiliaryFile property to argFile, signaling that a file should not
	  be validated or appear in the jobReport
	* python/trfExe.py
	- Add memMonitor property to scriptExecutor, that will try to use the external
	  MemoryMonitor binary to memory monitor the script executor's child process
	- MemoryMonitor is launched immediately after the scriptExecutor's child process
	  and the memory summary JSON file parsed afterwards and added to the executor
	  memory report
	- Deprecate the perfMonFile property of athenaExecutor and stop parsing any
	  perfmon outputs
	- For DBRelease files, dereference all soft links, allowing for use with the
	  pre-release DBRelease-pcache-current.tar.gz link
	- Change suffix on multiple-merged outputs after AthenaMP has run to _N (as
	  expected by PanDA)
	* python/trfFileUtils.py
	- Add inpFileInterestingKeys (not yet used)
	* python/trfReports.py
	- Update report version number to 1.0.1
	- Modify top level report 'resource' entry to have entries per executor
...
(Long ChangeLog diff - truncated)

	- Add an extension to transform._doSteering to allow for 'manual' steering
	* python/trfAMI.py
	- Add proper parsing for 'plain' string and number substep types
	* test/test_trfAMI.py
	- Small fixes to q120 tag
	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
	* Tag PyJobTransforms-02-03-45 (PyJobTransforms-02-03-45)

	* python/transform.py
	- Add an extension to transform._doSteering to allow for 'manual' steering
	* python/trfAMI.py
	- Add proper parsing for 'plain' string and number substep types
	* test/test_trfAMI.py
	- Small fixes to q120 tag
	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
	* Tag PyJobTransforms-02-03-45

2015-03-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfAMI.py, scripts/GetTfCommand.py
	- Rollback last buggy commit r654630->r649652

2015-03-17 Frank Volkmer <volkmer@cern.ch>
	* python/trfAMI.py
	- add isNewAMITag
	- add suppressNonJobOptions handling
	- add own get_ami_tag func
	- use remove_enclosing_quotes to clean up strings
	- adapt getTrfConfigFromAMI to new AMI tags
...
(Long ChangeLog diff - truncated)

	- Set the input filename correctly from the AMI tag, but
	  fallback to trfDefaultFiles if the value is bad
	* python/trfDefaultFiles.py
	- Use more sensible variable names in getInputFileName function
	* Tag PyJobTransforms-02-03-44 (PyJobTransforms-02-03-44)

	* python/trfAMI.py
	- Set the input filename correctly from the AMI tag, but
	  fallback to trfDefaultFiles if the value is bad
	* python/trfDefaultFiles.py
	- Use more sensible variable names in getInputFileName function
	* Tag PyJobTransforms-02-03-44

2015-02-24 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- If CLI file parameters are passed then do not take files from the AMI
	  tag itself (prevents double input file defintions of different types)
	* python/trfGraph.py
	- Print input and output data types at INFO level
	* test/PyJobTransforms_TestConfiguration.xml
	- Add trfSubstepIntegration test to RTT
	* test/test_trfAMI.py
	- Fix q120 test as redefined by PROC
	* test/test_trfSubstepIntegration.py
	- Test that Reco_tf and FullChain_tf can import substeps correctly
	* Tag PyJobTransforms-02-03-43
...
(Long ChangeLog diff - truncated)

New way to elevate std::bad_alloc message to "CATASTROPHE" (level = fatal + 10)
Improved GetTfCommand.py
Wider test coverage
* Tag PyJobTransforms-02-03-37 (PyJobTransforms-02-03-37)

	* Tag PyJobTransforms-02-03-37

2014-12-18  Azzah Alshehri <azzah.aziz.alshehri@cern.ch>
	* python/trfArgClasses.py
	- Convert the member variable self._desc in prodsysDescription method to be a local variable.
	- Pass the argument groups into the json signature so that the AMI can use it to layout arguments sensibly when defining 	tags

2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfArgs.py
	- Add new argument group 'Metadata'
	- Add task, job, attempt arguments in metadata group
	- Make metadata arguments a part of standard transform arguments
	- Fix missing group for athenaMPMergeTargetSize
	* test/test_trfValidation.py
	- Clean up all test logfiles

2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- Small change in startup message
	* python/trfExe.py
...
(Long ChangeLog diff - truncated)

	- Change default option in hybridMerge to be "slow" merging, due
	to persistent issues with fast merged metadata
	* trfReports.py
	- Fix bad reference to _maxMsgLen in python report generator
	* trfValidation.py
	- Add file count validation for DAOD merging transform
	* test_AODMerge_tf.py
	- Account for metadata 'compression' in metadata value test case
	* Tag PyJobTransforms-02-03-29 (PyJobTransforms-02-03-29)

	* trfExe.py
	- Change default option in hybridMerge to be "slow" merging, due
	to persistent issues with fast merged metadata
	* trfReports.py
	- Fix bad reference to _maxMsgLen in python report generator
	* trfValidation.py
	- Add file count validation for DAOD merging transform
	* test_AODMerge_tf.py
	- Account for metadata 'compression' in metadata value test case
	* Tag PyJobTransforms-02-03-29

2014-09-19 Graeme Stewart <graeme.andrew.stewart@cern.ch>
    * trfAMI.py
    - Introduce test for 'new transform' and make sure that command line
    arguments are prefixed by "--"
    * trfReport.py
    - Truncate very long error messages in exitMsg
    * trfArgClasses.py
    - Refactor EOS globbing code to improve robustness and error messages
    * Tag PyJobTransforms-02-03-28
...
(Long ChangeLog diff - truncated)

        - Change format of DAOD output filename to match
          grid expectations
    * Tag PyJobTransforms-02-03-26 (PyJobTransforms-02-03-26)