For jobs running on grid, PyJobTransforms detects the non-interactive
environment and enables text report. The debug message would help
detecting potential issues on grid jobs which otherwise would be
difficult to reproduce in interactive mode.


Former-commit-id: 43b31f69c027fff8120c241c154dc00774ff0bd3

The time reporting info includes cpu and wall times used during
transformSetup, inFileValidation, outFileValidation, and
reportGeneration.
All time values which were integers in earlier versions of jobReport,
are now reported as floating point values rounded to 3 decimal places.
Move utility methods calcCpuTime & calcWallTime from trfExe to trfUtils.


Former-commit-id: 7c036b7f41b370f82b3e322859c6321c6e6ac5df

	* python/transform.py
	- Added properties for cpu and walltime for transformSetup, inFileValidation
	  and outFileValidation.
	* python/trfReports
	- Added time reporting key-value pairs for transformSetup, inFileValidation
	  and outFileValidation.
	* python/trfUtils
	- Added utility methods calcCpuTime and CalcWallTime.
	* Tagging PyJobTransforms-03-00-34  [ATLASJT-291]


Former-commit-id: ecb568dbd078b855d6a005861339a3c089c43cdb

Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346] (PyJobTransforms-03-00-31)

	* python/trfUtils.py
	- Added utility method isInteractiveEnv()
	* python/trfExe.py
	- Refactored checks for interactive environment using new method in trfUtils.
	* python/transform.py
	- Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346]
	* Tagging PyJobTransforms-03-00-31

2016-11-17 Marcelo Vogel <mavogel@cern.ch>
	* scripts/makeTrfJSONSignatures.py
	- Replace script by updated version provided by AMI team [ATLASJT-284]
	* Tagging PyJobTransforms-03-00-30

2016-11-14 Marcelo Vogel <mavogel@cern.ch>
	* python/transform.py
	- Add property trfPredata to be set from env variable at trf init
	* python/trfReports.py
	- Add trfPredata to job report under key 'trfPredata'
	- Report version 2.0.7
	* Tagging PyJobTransforms-03-00-29
...
(Long ChangeLog diff - truncated)


Former-commit-id: 0db01cfba03324f8b6939d2f979bc9b02f861660

Remove = after pre/postInclude in output of GetTfCommand.py [ATLASAMITG-99] (PyJobTransforms-03-00-24)

	* scripts/GetTfCommand.py
	- Fix bug preventing correct output for option --printOnlyCmdLine
	* python/trfAMI.py
	- Remove '=' after pre/postInclude in _argsToString() [ATLASAMITG-99]
	* Tagging PyJobTransforms-03-00-24

2016-09-09 Marcelo Vogel <mavogel@cern.ch>
	* python/transform.py
	- Configure Event Service job via a special argument
	  eventService=True, which expands to other parameters [ATLASJT-320]
	* python/trfMPTools.py
	- Add switch to turn on/off output file checks in athenaMPOutputHandler()
	* python/trfExe.py
	- Turn on/off output file checks in athenaMP if eventService=False/True
	* test/test_trfMPTools.py
	- Update interface of athenaMPOutputHandler() in unittest
	* Tagging PyJobTransforms-03-00-23

2016-08-29 Umesh Worlikar <umesh.worlikar@cern.ch>
        * python/trfArgClasses.py
...
(Long ChangeLog diff - truncated)


Former-commit-id: e4f14ea8

Changed athenaMP default strategy to SharedQueue [ATLASJT-341]. ReportVersion 2.0.6 (PyJobTransforms-03-00-22)

	* python/trfExe.py
	- Change athenaMP default strategy to SharedQueue [ATLASJT-341].
	* Tagging PyJobTransforms-03-00-22

2016-08-22 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfExe.py
	- add preExecute and validation start and stop times
	* python/trfReports.py
	- reportVersion 2.0.6 cpu- and wallTimes for preExe, postExe, val and
	total

2016-08-19 Marcelo Vogel <mavogel@cern.ch>
	* python/trfReports.py
	- Include processedEvents in job report if not null
	* Tagging PyJobTransforms-03-00-21

2016-08-19 Marcelo Vogel <mavogel@cern.ch>
	* python/trfArgClasses.py
	- Add new argument class, argEVNTFile
	* scripts/ValidateFiles_tf.py
...
(Long ChangeLog diff - truncated)


Former-commit-id: b4050564

	* python/trfArgClasses.py
	- Correct argBSFile.selfMerge to return the executor properly
	* Tagging PyJobTransforms-03-00-13

2016-06-21 Umesh Worlikar <umesh.worlikar@cern.ch>
        * python/transform.py
        - transform.execute() now ensures validate() in case of exception raised
          during execute() and postExecute(). This fixes missing 'bad_alloac' error
          in jobReport [ATLASJT-303]


Former-commit-id: 3a6ba277646497e13dd4e2016855b1f6e36ed1d5

2016-05-06  scott snyder  <snyder@bnl.gov>
	* Tagging PyJobTransforms-03-00-05.
	* test/test_trfReports.py, test/test_trfArgs.py: Update for last
	changes.

2016-04-29 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfReports.py
	- Remove the ~useless "argValues" section from the job report
	- Remove the ~useless 'integrity' and 'file_type'sections from subfile reports
	- Update report number to 2.0.0

2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfUtils.py
	- Minor tweaks on reported environment variables for CMake builds

2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfUtils.py
	- Update asetupReport to use "lstags" (works for both cmt and cmake)

2016-04-27 Graeme Stewart <graeme.andrew.stewart@cern.ch>
...
(Long ChangeLog diff - truncated)


Former-commit-id: 00312877

	- Add a new option --athenaMPStrategy to set the MP event reading strategy
	* python/trfExe.py
	- Handle athenaMPStrategy setting
	* python/trfJobOptions.py
	- Modify job options to only se athenaMPStrategy if the default is True
	* Tagging PyJobTransforms-02-03-64 (PyJobTransforms-02-03-64)

	* python/trfArgs.py
	- Add a new option --athenaMPStrategy to set the MP event reading strategy
	* python/trfExe.py
	- Handle athenaMPStrategy setting
	* python/trfJobOptions.py
	- Modify job options to only se athenaMPStrategy if the default is True
	* Tagging PyJobTransforms-02-03-64

2015-09-29 Graeme Stewart for Frank Volkmer <volkmer@cern.ch>
	* python/performanceDataUploader.py
	- Remove this - no longer used in Run2
	* python/transform.py
	- Update TRF_UNKOWN (sic) error code to TRF_UNKNOWN
	* python/trfExe.py
	- Set self._athenaMPStrategy to 'FileScheduling' if n_inputs = n_workers
	  (see ATLASRECTS-2452 and ATLASJT-269)
	* python/trfJobOptions.py
	- Convert executor _athenaMPStrategy to MP job option
	* Tagging PyJobTransforms-02-03-63

...
(Long ChangeLog diff - truncated)

	* python/transform.py
	- Switch off text, pilot.pickle and classic file reports
	* python/trfExitCodes.py
	- New TRF_EXTERNAL code
	* python/trfDecorators.py
	- Catch IOError from subprocess in timelimited decorator and handle with TRF_EXTERNAL
	* python/trfExe.py
	- Ensure that DAOD outputs for reduction are type 'AOD' (not 'aod' - was causing a
	  failure in MP derivation jobs)
	* python/trfReports.py
	- Suppress messages when /etc/machinefeatures/hs06 is missing
	* Tagging PyJobTransforms-02-03-62

2015-09-08 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfArgClasses.py
	- Append G4Version to retrieveKeys for HITSFiles
	* python/trfFileUtils.py
	- Get G4Version information from metadata in AthenaLiteFileInfo

2015-08-27 Lisa Errenst <lisa.errenst@cern.ch>
...
(Long ChangeLog diff - truncated)

	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
	* scripts/makeTrfJSONSignatures.py
	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
	  an abort when using __import__
	* test/test_trfArgs.py
	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
	  in a different project, this a problem in ATN during full builds)
	* Tagging PyJobTransforms-02-03-60 (PyJobTransforms-02-03-60)

	* python/trfArgs.py
	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
	* scripts/makeTrfJSONSignatures.py
	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
	  an abort when using __import__
	* test/test_trfArgs.py
	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
	  in a different project, this a problem in ATN during full builds)
	* Tagging PyJobTransforms-02-03-60

2015-07-08 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfReports.py
	- add HEPSPEC to machine report. Report version 1.0.5

2015-07-03 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- Refactor handling of end of transform to always try to kill stray children
	* python/trfArgClasses.py
	- Move _mergeArgs methog from argAthenaFile to argFile so that argNTUP has access
	* python/trfArgs.py
...
(Long ChangeLog diff - truncated)

	- Add an extension to transform._doSteering to allow for 'manual' steering
	* python/trfAMI.py
	- Add proper parsing for 'plain' string and number substep types
	* test/test_trfAMI.py
	- Small fixes to q120 tag
	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
	* Tag PyJobTransforms-02-03-45 (PyJobTransforms-02-03-45)

	* python/transform.py
	- Add an extension to transform._doSteering to allow for 'manual' steering
	* python/trfAMI.py
	- Add proper parsing for 'plain' string and number substep types
	* test/test_trfAMI.py
	- Small fixes to q120 tag
	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
	* Tag PyJobTransforms-02-03-45

2015-03-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfAMI.py, scripts/GetTfCommand.py
	- Rollback last buggy commit r654630->r649652

2015-03-17 Frank Volkmer <volkmer@cern.ch>
	* python/trfAMI.py
	- add isNewAMITag
	- add suppressNonJobOptions handling
	- add own get_ami_tag func
	- use remove_enclosing_quotes to clean up strings
	- adapt getTrfConfigFromAMI to new AMI tags
...
(Long ChangeLog diff - truncated)

	- Set the input filename correctly from the AMI tag, but
	  fallback to trfDefaultFiles if the value is bad
	* python/trfDefaultFiles.py
	- Use more sensible variable names in getInputFileName function
	* Tag PyJobTransforms-02-03-44 (PyJobTransforms-02-03-44)

	* python/trfAMI.py
	- Set the input filename correctly from the AMI tag, but
	  fallback to trfDefaultFiles if the value is bad
	* python/trfDefaultFiles.py
	- Use more sensible variable names in getInputFileName function
	* Tag PyJobTransforms-02-03-44

2015-02-24 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- If CLI file parameters are passed then do not take files from the AMI
	  tag itself (prevents double input file defintions of different types)
	* python/trfGraph.py
	- Print input and output data types at INFO level
	* test/PyJobTransforms_TestConfiguration.xml
	- Add trfSubstepIntegration test to RTT
	* test/test_trfAMI.py
	- Fix q120 test as redefined by PROC
	* test/test_trfSubstepIntegration.py
	- Test that Reco_tf and FullChain_tf can import substeps correctly
	* Tag PyJobTransforms-02-03-43
...
(Long ChangeLog diff - truncated)

	- Remove never implemented transform.doTrfValidation method
	- Update file validation routines to support new fileValidation style boolean arguments
	* python/trfArgs.py
	- Refactor getting extra DPD list to getExtraDPDList() function
	- Support new NTUPMergerArgs flag to addExtraDPDTypes, adding input and output merge
	  file arguments
	- Add new fileValidation style boolean arguments to addFileValidationArguments
	* Tag PyJobTransforms-02-03-40 (PyJobTransforms-02-03-40)

	* python/transform.py
	- Remove never implemented transform.doTrfValidation method
	- Update file validation routines to support new fileValidation style boolean arguments
	* python/trfArgs.py
	- Refactor getting extra DPD list to getExtraDPDList() function
	- Support new NTUPMergerArgs flag to addExtraDPDTypes, adding input and output merge
	  file arguments
	- Add new fileValidation style boolean arguments to addFileValidationArguments
	* Tag PyJobTransforms-02-03-40

New way to elevate std::bad_alloc message to "CATASTROPHE" (level = fatal + 10)
Improved GetTfCommand.py
Wider test coverage
* Tag PyJobTransforms-02-03-37 (PyJobTransforms-02-03-37)

	* Tag PyJobTransforms-02-03-37

2014-12-18  Azzah Alshehri <azzah.aziz.alshehri@cern.ch>
	* python/trfArgClasses.py
	- Convert the member variable self._desc in prodsysDescription method to be a local variable.
	- Pass the argument groups into the json signature so that the AMI can use it to layout arguments sensibly when defining 	tags

2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfArgs.py
	- Add new argument group 'Metadata'
	- Add task, job, attempt arguments in metadata group
	- Make metadata arguments a part of standard transform arguments
	- Fix missing group for athenaMPMergeTargetSize
	* test/test_trfValidation.py
	- Clean up all test logfiles

2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- Small change in startup message
	* python/trfExe.py
...
(Long ChangeLog diff - truncated)

        - Change format of DAOD output filename to match
          grid expectations
    * Tag PyJobTransforms-02-03-26 (PyJobTransforms-02-03-26)