- Apr 20, 2017
-
-
Marcelo Vogel authored
G4 exceptions can be fatal, therefore, even warnings associated to them are added to the job report. However, there was no limit on the number of messages that could be added. This has led to grid tasks producing job report files with over 8MB. The fixes included in this branch place a limit on the number of messages that can be included in the job report from G4 exceptions.
-
- Apr 10, 2017
-
-
Alexey Boldyrev authored
-
- Dec 16, 2016
-
-
Umesh Dharmaji Worlikar authored
Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346] (PyJobTransforms-03-00-31) * python/trfUtils.py - Added utility method isInteractiveEnv() * python/trfExe.py - Refactored checks for interactive environment using new method in trfUtils. * python/transform.py - Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346] * Tagging PyJobTransforms-03-00-31 2016-11-17 Marcelo Vogel <mavogel@cern.ch> * scripts/makeTrfJSONSignatures.py - Replace script by updated version provided by AMI team [ATLASJT-284] * Tagging PyJobTransforms-03-00-30 2016-11-14 Marcelo Vogel <mavogel@cern.ch> * python/transform.py - Add property trfPredata to be set from env variable at trf init * python/trfReports.py - Add trfPredata to job report under key 'trfPredata' - Report version 2.0.7 * Tagging PyJobTransforms-03-00-29 ... (Long ChangeLog diff - truncated)
-
- Aug 19, 2016
-
-
Marcelo Vogel authored
Fixed bug that enabled athenaMP when expectedEvents < N cores. [ATLASJT-337] (PyJobTransforms-03-00-20) * python/trfExe.py - Fix bug that was preventing to turn off athenaMP when - expectedEvents < N cores. [ATLASJT-337] * Tagging PyJobTransforms-03-00-20 2016-07-27 Graeme Stewart <graeme.andrew.stewart@cern.ch> * ATLASJT-335 fixed * python/trfArgClasses.py - Add a new "dumpvalue" property to argSubstepSteering to hold the alias value used to set the steering and dump that if requested * python/trfUtils.py - In dumpJSON and dumpPickle use the "dumpvalue" method, if the argument supports it * Tagging PyJobTransforms-03-00-19 2016-07-25 Marcelo Vogel <mavogel@cern.ch> * python/trfArgs.py - Fix bug in getExtraDPDList. Executor for NTUP_MUONCALIB was added twice * Tagging PyJobTransforms-03-00-18 ... (Long ChangeLog diff - truncated)
-
- Jun 14, 2016
-
-
Marcelo Vogel authored
Fixed typo in python/trfExe.py (ATLASJT-316) and updated tests that were failing in RTT (PyJobTransforms-03-00-11) 2016-06-10 Marcelo Vogel <mavogel@cern.ch> * python/trfExe.py - Fix typo 'ahtenaopts' to 'athenaopts' * test/test_TAGMerge_tf.py - Remove test on metadata (beam_type) which is no longer included in jobReport.json * test/test_RAWMerge_tf.py - Remove tests on metadata which is no longer included in jobReport.json (conditions_tag, geometry and beam_type) * Tagging PyJobTransforms-03-00-11 2016-06-03 Marcelo Vogel <mavogel@cern.ch> * python/trfArgs.py - Add missing nargs='+' option to --triggerConfig * Tagging PyJobTransforms-03-00-10 2016-05-31 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/trfExe.py - Move disableMP argument from transformExecutor class to athenaExecutor class (ATLASJT-318) * python/trfArgClasses.py - Update selfMerge methods for POOL files to use disableMP=True in executor ... (Long ChangeLog diff - truncated)
-
- May 27, 2016
-
-
Scott Snyder authored
2016-05-06 scott snyder <snyder@bnl.gov> * Tagging PyJobTransforms-03-00-05. * test/test_trfReports.py, test/test_trfArgs.py: Update for last changes. 2016-04-29 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/trfReports.py - Remove the ~useless "argValues" section from the job report - Remove the ~useless 'integrity' and 'file_type'sections from subfile reports - Update report number to 2.0.0 2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/trfUtils.py - Minor tweaks on reported environment variables for CMake builds 2016-04-28 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/trfUtils.py - Update asetupReport to use "lstags" (works for both cmt and cmake) 2016-04-27 Graeme Stewart <graeme.andrew.stewart@cern.ch> ... (Long ChangeLog diff - truncated)
-
- May 09, 2016
-
-
Graeme Stewart authored
- Specifically to test fork after first event for memory savings - Also collects many other minor fixes - EXCLUDES lightweight output file metadata patch (r731518) (PyJobTransforms-02-03-76-01) wwwpackage PyJobTransforms 2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch> ** Merge in trunk changes from r740535 2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch> ** Manually patch in trunk fixes athenaMPMergeTargetSize handling (ATLASJT-298) *** WARNING - Do not attempt to merge r731518 onto this branch *** 2016-04-15 Graeme Stewart <graeme.andrew.stewart@cern.ch> ** Merge in trunk changes, excluding lightweight metadata fixes 2016-04-11 Azzah Alshehri<azzah.aziz.alshehri@cern.ch> ** Change the self merge method to save the merge information * python/trfExe.py - Change the stanalone variable 'myMerger'inside the method to a class variable. - Save the 'merge executor instance' you get back in the athenaExecutor _smartMerge method. - Add an accessor to the base executor class, that returns the self._myMerge value. * python/trfArgClasses.py - Re-set 'myMerger' to the real merger executor in the actual merge. - Return myMerger for each of the specific file mergers method. ... (Long ChangeLog diff - truncated)
-
- Aug 05, 2015
-
-
Graeme Stewart authored
- Allow --skipEvents to be multi-valued (needed by FTK transforms) * scripts/makeTrfJSONSignatures.py - Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes an abort when using __import__ * test/test_trfArgs.py - Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is in a different project, this a problem in ATN during full builds) * Tagging PyJobTransforms-02-03-60 (PyJobTransforms-02-03-60) * python/trfArgs.py - Allow --skipEvents to be multi-valued (needed by FTK transforms) * scripts/makeTrfJSONSignatures.py - Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes an abort when using __import__ * test/test_trfArgs.py - Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is in a different project, this a problem in ATN during full builds) * Tagging PyJobTransforms-02-03-60 2015-07-08 Lisa Errenst <lisa.errenst@cern.ch> * python/trfReports.py - add HEPSPEC to machine report. Report version 1.0.5 2015-07-03 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/transform.py - Refactor handling of end of transform to always try to kill stray children * python/trfArgClasses.py - Move _mergeArgs methog from argAthenaFile to argFile so that argNTUP has access * python/trfArgs.py ... (Long ChangeLog diff - truncated)
-
- May 17, 2015
-
-
Graeme Stewart authored
- Add new auxiliaryFile property to argFile, signaling that a file should not be validated or appear in the jobReport * python/trfExe.py - Add memMonitor property to scriptExecutor, that will try to use the external MemoryMonitor binary to memory monitor the script executor's child process - MemoryMonitor is launched immediately after the scriptExecutor's child process and the memory summary JSON file parsed afterwards and added to the executor memory report - Deprecate the perfMonFile property of athenaExecutor and stop parsing any perfmon outputs - For DBRelease files, dereference all soft links, allowing for use with the pre-release DBRelease-pcache-current.tar.gz link - Change suffix on multiple-merged outputs after AthenaMP has run to _N (as expected by PanDA) * python/trfFileUtils.py - Add inpFileInterestingKeys (not yet used) * python/trfReports.py - Update report version number to 1.0.1 - Modify top level report 'resource' entry to have entries per executor with MemoryMonitor summary information and a separate entry for the transform and its externals - Add machine report information to resource entry, with information also parsed from /proc/cpuinfo * python/trfUtils.py - Add small debug message when setting up a DBRelease from CVMFS * python/trfValidation.py - Ensure that auxilliary files are never validated * test/test_trfUtilsDBRelease.py - Add new test for DBRelease-pcache-current.tar.gz link * Tag PyJobTransforms-02-03-48 (PyJobTransforms-02-03-48) * python/trfArgClasses.py - Add new auxiliaryFile property to argFile, signaling that a file should not be validated or appear in the jobReport * python/trfExe.py - Add memMonitor property to scriptExecutor, that will try to use the external MemoryMonitor binary to memory monitor the script executor's child process - MemoryMonitor is launched immediately after the scriptExecutor's child process and the memory summary JSON file parsed afterwards and added to the executor memory report - Deprecate the perfMonFile property of athenaExecutor and stop parsing any perfmon outputs - For DBRelease files, dereference all soft links, allowing for use with the pre-release DBRelease-pcache-current.tar.gz link - Change suffix on multiple-merged outputs after AthenaMP has run to _N (as expected by PanDA) * python/trfFileUtils.py - Add inpFileInterestingKeys (not yet used) * python/trfReports.py - Update report version number to 1.0.1 - Modify top level report 'resource' entry to have entries per executor ... (Long ChangeLog diff - truncated)
-
- Jan 19, 2015
-
-
Graeme Stewart authored
New way to elevate std::bad_alloc message to "CATASTROPHE" (level = fatal + 10) Improved GetTfCommand.py Wider test coverage * Tag PyJobTransforms-02-03-37 (PyJobTransforms-02-03-37) * Tag PyJobTransforms-02-03-37 2014-12-18 Azzah Alshehri <azzah.aziz.alshehri@cern.ch> * python/trfArgClasses.py - Convert the member variable self._desc in prodsysDescription method to be a local variable. - Pass the argument groups into the json signature so that the AMI can use it to layout arguments sensibly when defining tags 2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/trfArgs.py - Add new argument group 'Metadata' - Add task, job, attempt arguments in metadata group - Make metadata arguments a part of standard transform arguments - Fix missing group for athenaMPMergeTargetSize * test/test_trfValidation.py - Clean up all test logfiles 2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch> * python/transform.py - Small change in startup message * python/trfExe.py ... (Long ChangeLog diff - truncated)
-
- Nov 05, 2014
-
-
Graeme Stewart authored
- Change default option in hybridMerge to be "slow" merging, due to persistent issues with fast merged metadata * trfReports.py - Fix bad reference to _maxMsgLen in python report generator * trfValidation.py - Add file count validation for DAOD merging transform * test_AODMerge_tf.py - Account for metadata 'compression' in metadata value test case * Tag PyJobTransforms-02-03-29 (PyJobTransforms-02-03-29) * trfExe.py - Change default option in hybridMerge to be "slow" merging, due to persistent issues with fast merged metadata * trfReports.py - Fix bad reference to _maxMsgLen in python report generator * trfValidation.py - Add file count validation for DAOD merging transform * test_AODMerge_tf.py - Account for metadata 'compression' in metadata value test case * Tag PyJobTransforms-02-03-29 2014-09-19 Graeme Stewart <graeme.andrew.stewart@cern.ch> * trfAMI.py - Introduce test for 'new transform' and make sure that command line arguments are prefixed by "--" * trfReport.py - Truncate very long error messages in exitMsg * trfArgClasses.py - Refactor EOS globbing code to improve robustness and error messages * Tag PyJobTransforms-02-03-28 ... (Long ChangeLog diff - truncated)
-
- Sep 19, 2014
-
-
Graeme Stewart authored
- Change format of DAOD output filename to match grid expectations * Tag PyJobTransforms-02-03-26 (PyJobTransforms-02-03-26)
-