Former-commit-id: 20cf8aebf12692d816aa03e99c5c32dbc3d087fd

It has been demonstrated that the checkpoint-restart technique is not
bringing us any benefits. On contrary, by keeping dmtcp in atlasexternals
we are causing troubles to building our software on PowerPC and MacOS


Former-commit-id: 9edfd0c8

This branch provides support for activating the Event Service mode
of AthenaMP for general transforms, specifically in reconstruction.
It also updates the support to simulation transforms by refactoring
the existing code.


Former-commit-id: 8695d7f1

This branch provides support for handling the output of AthenaMP in sharedWriter
mode. AthenaMP typically produces an output file per core, which are then merged
after processing. However, in sharedWriter mode, AthenaMP produces a single output
file and no merging is necessary. A new argument flag was added (--sharedWriter True)
to instruct the transforms to properly handle the single output file.


Former-commit-id: c3e85a35c7b09fbc7b864910198d08bf0e6c1988

PyJobTransforms: introduced two new command-line options --checkpoint (bool)
and --restart (string). It is yet to be defined what value we should be passing
to the --restart option

AthenaMP: added code to checkpoint itself just before forking and then to
exit immediately. The code cannot yet be released because it depends on private
installation of DMTCP 2.4.5. Pending the appearance of DMTCP as Athena external.

Former-commit-id: 72d35ff6c66c6671cb88ad32cd771f5f2a380f0d

Fixed bug that enabled athenaMP when expectedEvents < N cores. [ATLASJT-337] (PyJobTransforms-03-00-20)

	* python/trfExe.py
	- Fix bug that was preventing to turn off athenaMP when
	- expectedEvents < N cores. [ATLASJT-337]
	* Tagging PyJobTransforms-03-00-20

2016-07-27 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* ATLASJT-335 fixed
	* python/trfArgClasses.py
	- Add a new "dumpvalue" property to argSubstepSteering to hold the alias
	  value used to set the steering and dump that if requested
	* python/trfUtils.py
	- In dumpJSON and dumpPickle use the "dumpvalue" method, if the argument
	  supports it
	* Tagging PyJobTransforms-03-00-19

2016-07-25 Marcelo Vogel <mavogel@cern.ch>
	* python/trfArgs.py
	- Fix bug in getExtraDPDList. Executor for NTUP_MUONCALIB was added twice
	* Tagging PyJobTransforms-03-00-18

...
(Long ChangeLog diff - truncated)


Former-commit-id: 7abefd9c

	* python/trfArgs.py
	- Add new output type NTUP_MUONCALIB
        * Tagging PyJobTransforms-03-00-12


Former-commit-id: da0cedbfda3f4a1ff3dcec2d9f88564890d53f6f

Fixed typo in python/trfExe.py (ATLASJT-316) and updated tests that were failing in RTT (PyJobTransforms-03-00-11)

2016-06-10 Marcelo Vogel <mavogel@cern.ch>
	* python/trfExe.py
	- Fix typo 'ahtenaopts' to 'athenaopts'
	* test/test_TAGMerge_tf.py
	- Remove test on metadata (beam_type) which is no longer included in jobReport.json
	* test/test_RAWMerge_tf.py
	- Remove tests on metadata which is no longer included in jobReport.json
	  (conditions_tag, geometry and beam_type)
	* Tagging PyJobTransforms-03-00-11

2016-06-03 Marcelo Vogel <mavogel@cern.ch>
	* python/trfArgs.py
	- Add missing nargs='+' option to --triggerConfig
	* Tagging PyJobTransforms-03-00-10

2016-05-31 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfExe.py
	- Move disableMP argument from transformExecutor class to athenaExecutor class (ATLASJT-318)
	* python/trfArgClasses.py
	- Update selfMerge methods for POOL files to use disableMP=True in executor
...
(Long ChangeLog diff - truncated)


Former-commit-id: 6ecf15c0a774d51cd31ef4760c9f75fc83f7e05b

2016-05-06  scott snyder  <snyder@bnl.gov>
	* Tagging PyJobTransforms-03-00-05.
	* test/test_trfReports.py, test/test_trfArgs.py: Update for last
	changes.

2016-04-29 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfReports.py
	- Remove the ~useless "argValues" section from the job report
	- Remove the ~useless 'integrity' and 'file_type'sections from subfile reports
	- Update report number to 2.0.0

2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfUtils.py
	- Minor tweaks on reported environment variables for CMake builds

2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfUtils.py
	- Update asetupReport to use "lstags" (works for both cmt and cmake)

2016-04-27 Graeme Stewart <graeme.andrew.stewart@cern.ch>
...
(Long ChangeLog diff - truncated)


Former-commit-id: 00312877

- Specifically to test fork after first event for memory savings
- Also collects many other minor fixes
- EXCLUDES lightweight output file metadata patch (r731518) (PyJobTransforms-02-03-76-01)

wwwpackage PyJobTransforms
2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	** Merge in trunk changes from r740535

2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	** Manually patch in trunk fixes athenaMPMergeTargetSize handling (ATLASJT-298)
	*** WARNING - Do not attempt to merge r731518 onto this branch ***

2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	** Merge in trunk changes, excluding lightweight metadata fixes

2016-04-11 Azzah Alshehri<azzah.aziz.alshehri@cern.ch>
	** Change the self merge method to save the merge information
	   * python/trfExe.py
	   - Change the stanalone variable 'myMerger'inside the method to  a class variable.
	   - Save the 'merge executor instance' you get back in the athenaExecutor _smartMerge method.
	   - Add an accessor to the base executor class, that returns the self._myMerge value.
	   * python/trfArgClasses.py
	   - Re-set 'myMerger' to the real merger executor in the actual merge.
	   - Return myMerger for each of the specific file mergers method.
...
(Long ChangeLog diff - truncated)

	- Add a new option --athenaMPStrategy to set the MP event reading strategy
	* python/trfExe.py
	- Handle athenaMPStrategy setting
	* python/trfJobOptions.py
	- Modify job options to only se athenaMPStrategy if the default is True
	* Tagging PyJobTransforms-02-03-64 (PyJobTransforms-02-03-64)

	* python/trfArgs.py
	- Add a new option --athenaMPStrategy to set the MP event reading strategy
	* python/trfExe.py
	- Handle athenaMPStrategy setting
	* python/trfJobOptions.py
	- Modify job options to only se athenaMPStrategy if the default is True
	* Tagging PyJobTransforms-02-03-64

2015-09-29 Graeme Stewart for Frank Volkmer <volkmer@cern.ch>
	* python/performanceDataUploader.py
	- Remove this - no longer used in Run2
	* python/transform.py
	- Update TRF_UNKOWN (sic) error code to TRF_UNKNOWN
	* python/trfExe.py
	- Set self._athenaMPStrategy to 'FileScheduling' if n_inputs = n_workers
	  (see ATLASRECTS-2452 and ATLASJT-269)
	* python/trfJobOptions.py
	- Convert executor _athenaMPStrategy to MP job option
	* Tagging PyJobTransforms-02-03-63

...
(Long ChangeLog diff - truncated)

	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
	* scripts/makeTrfJSONSignatures.py
	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
	  an abort when using __import__
	* test/test_trfArgs.py
	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
	  in a different project, this a problem in ATN during full builds)
	* Tagging PyJobTransforms-02-03-60 (PyJobTransforms-02-03-60)

	* python/trfArgs.py
	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
	* scripts/makeTrfJSONSignatures.py
	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
	  an abort when using __import__
	* test/test_trfArgs.py
	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
	  in a different project, this a problem in ATN during full builds)
	* Tagging PyJobTransforms-02-03-60

2015-07-08 Lisa Errenst <lisa.errenst@cern.ch>
	* python/trfReports.py
	- add HEPSPEC to machine report. Report version 1.0.5

2015-07-03 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- Refactor handling of end of transform to always try to kill stray children
	* python/trfArgClasses.py
	- Move _mergeArgs methog from argAthenaFile to argFile so that argNTUP has access
	* python/trfArgs.py
...
(Long ChangeLog diff - truncated)

	- Add an extension to transform._doSteering to allow for 'manual' steering
	* python/trfAMI.py
	- Add proper parsing for 'plain' string and number substep types
	* test/test_trfAMI.py
	- Small fixes to q120 tag
	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
	* Tag PyJobTransforms-02-03-45 (PyJobTransforms-02-03-45)

	* python/transform.py
	- Add an extension to transform._doSteering to allow for 'manual' steering
	* python/trfAMI.py
	- Add proper parsing for 'plain' string and number substep types
	* test/test_trfAMI.py
	- Small fixes to q120 tag
	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
	* Tag PyJobTransforms-02-03-45

2015-03-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfAMI.py, scripts/GetTfCommand.py
	- Rollback last buggy commit r654630->r649652

2015-03-17 Frank Volkmer <volkmer@cern.ch>
	* python/trfAMI.py
	- add isNewAMITag
	- add suppressNonJobOptions handling
	- add own get_ami_tag func
	- use remove_enclosing_quotes to clean up strings
	- adapt getTrfConfigFromAMI to new AMI tags
...
(Long ChangeLog diff - truncated)

	- Set the input filename correctly from the AMI tag, but
	  fallback to trfDefaultFiles if the value is bad
	* python/trfDefaultFiles.py
	- Use more sensible variable names in getInputFileName function
	* Tag PyJobTransforms-02-03-44 (PyJobTransforms-02-03-44)

	* python/trfAMI.py
	- Set the input filename correctly from the AMI tag, but
	  fallback to trfDefaultFiles if the value is bad
	* python/trfDefaultFiles.py
	- Use more sensible variable names in getInputFileName function
	* Tag PyJobTransforms-02-03-44

2015-02-24 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- If CLI file parameters are passed then do not take files from the AMI
	  tag itself (prevents double input file defintions of different types)
	* python/trfGraph.py
	- Print input and output data types at INFO level
	* test/PyJobTransforms_TestConfiguration.xml
	- Add trfSubstepIntegration test to RTT
	* test/test_trfAMI.py
	- Fix q120 test as redefined by PROC
	* test/test_trfSubstepIntegration.py
	- Test that Reco_tf and FullChain_tf can import substeps correctly
	* Tag PyJobTransforms-02-03-43
...
(Long ChangeLog diff - truncated)

	- Remove never implemented transform.doTrfValidation method
	- Update file validation routines to support new fileValidation style boolean arguments
	* python/trfArgs.py
	- Refactor getting extra DPD list to getExtraDPDList() function
	- Support new NTUPMergerArgs flag to addExtraDPDTypes, adding input and output merge
	  file arguments
	- Add new fileValidation style boolean arguments to addFileValidationArguments
	* Tag PyJobTransforms-02-03-40 (PyJobTransforms-02-03-40)

	* python/transform.py
	- Remove never implemented transform.doTrfValidation method
	- Update file validation routines to support new fileValidation style boolean arguments
	* python/trfArgs.py
	- Refactor getting extra DPD list to getExtraDPDList() function
	- Support new NTUPMergerArgs flag to addExtraDPDTypes, adding input and output merge
	  file arguments
	- Add new fileValidation style boolean arguments to addFileValidationArguments
	* Tag PyJobTransforms-02-03-40

New way to elevate std::bad_alloc message to "CATASTROPHE" (level = fatal + 10)
Improved GetTfCommand.py
Wider test coverage
* Tag PyJobTransforms-02-03-37 (PyJobTransforms-02-03-37)

	* Tag PyJobTransforms-02-03-37

2014-12-18  Azzah Alshehri <azzah.aziz.alshehri@cern.ch>
	* python/trfArgClasses.py
	- Convert the member variable self._desc in prodsysDescription method to be a local variable.
	- Pass the argument groups into the json signature so that the AMI can use it to layout arguments sensibly when defining 	tags

2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/trfArgs.py
	- Add new argument group 'Metadata'
	- Add task, job, attempt arguments in metadata group
	- Make metadata arguments a part of standard transform arguments
	- Fix missing group for athenaMPMergeTargetSize
	* test/test_trfValidation.py
	- Clean up all test logfiles

2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
	* python/transform.py
	- Small change in startup message
	* python/trfExe.py
...
(Long ChangeLog diff - truncated)

        - Change format of DAOD output filename to match
          grid expectations
    * Tag PyJobTransforms-02-03-26 (PyJobTransforms-02-03-26)