Skip to content
Snippets Groups Projects
  1. Oct 10, 2015
    • Graeme Stewart's avatar
      * Tagging PyJobTransforms-02-03-62 (PyJobTransforms-02-03-62) · b5c09e2b
      Graeme Stewart authored
      	* python/transform.py
      	- Switch off text, pilot.pickle and classic file reports
      	* python/trfExitCodes.py
      	- New TRF_EXTERNAL code
      	* python/trfDecorators.py
      	- Catch IOError from subprocess in timelimited decorator and handle with TRF_EXTERNAL
      	* python/trfExe.py
      	- Ensure that DAOD outputs for reduction are type 'AOD' (not 'aod' - was causing a
      	  failure in MP derivation jobs)
      	* python/trfReports.py
      	- Suppress messages when /etc/machinefeatures/hs06 is missing
      	* Tagging PyJobTransforms-02-03-62
      
      2015-09-08 Lisa Errenst <lisa.errenst@cern.ch>
      	* python/trfArgClasses.py
      	- Append G4Version to retrieveKeys for HITSFiles
      	* python/trfFileUtils.py
      	- Get G4Version information from metadata in AthenaLiteFileInfo
      
      2015-08-27 Lisa Errenst <lisa.errenst@cern.ch>
      ...
      (Long ChangeLog diff - truncated)
      b5c09e2b
  2. Aug 05, 2015
    • Graeme Stewart's avatar
      * python/trfArgs.py · cc7760b4
      Graeme Stewart authored
      	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
      	* scripts/makeTrfJSONSignatures.py
      	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
      	  an abort when using __import__
      	* test/test_trfArgs.py
      	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
      	  in a different project, this a problem in ATN during full builds)
      	* Tagging PyJobTransforms-02-03-60 (PyJobTransforms-02-03-60)
      
      	* python/trfArgs.py
      	- Allow --skipEvents to be multi-valued (needed by FTK transforms)
      	* scripts/makeTrfJSONSignatures.py
      	- Skip beamSpotT0_Vertex_tf.py, which is a not a PyJobTransforms transform and causes
      	  an abort when using __import__
      	* test/test_trfArgs.py
      	- Do configureFromJSON test with Athena_tf.py instead of Reco_tf.py (which is
      	  in a different project, this a problem in ATN during full builds)
      	* Tagging PyJobTransforms-02-03-60
      
      2015-07-08 Lisa Errenst <lisa.errenst@cern.ch>
      	* python/trfReports.py
      	- add HEPSPEC to machine report. Report version 1.0.5
      
      2015-07-03 Graeme Stewart <graeme.andrew.stewart@cern.ch>
      	* python/transform.py
      	- Refactor handling of end of transform to always try to kill stray children
      	* python/trfArgClasses.py
      	- Move _mergeArgs methog from argAthenaFile to argFile so that argNTUP has access
      	* python/trfArgs.py
      ...
      (Long ChangeLog diff - truncated)
      cc7760b4
  3. May 17, 2015
    • Graeme Stewart's avatar
      * python/trfArgClasses.py · 492dc1e7
      Graeme Stewart authored
      	- Add new auxiliaryFile property to argFile, signaling that a file should not
      	  be validated or appear in the jobReport
      	* python/trfExe.py
      	- Add memMonitor property to scriptExecutor, that will try to use the external
      	  MemoryMonitor binary to memory monitor the script executor's child process
      	- MemoryMonitor is launched immediately after the scriptExecutor's child process
      	  and the memory summary JSON file parsed afterwards and added to the executor
      	  memory report
      	- Deprecate the perfMonFile property of athenaExecutor and stop parsing any
      	  perfmon outputs
      	- For DBRelease files, dereference all soft links, allowing for use with the
      	  pre-release DBRelease-pcache-current.tar.gz link
      	- Change suffix on multiple-merged outputs after AthenaMP has run to _N (as
      	  expected by PanDA)
      	* python/trfFileUtils.py
      	- Add inpFileInterestingKeys (not yet used)
      	* python/trfReports.py
      	- Update report version number to 1.0.1
      	- Modify top level report 'resource' entry to have entries per executor
      	  with MemoryMonitor summary information and a separate entry for the
      	  transform and its externals
      	- Add machine report information to resource entry, with information
      	  also parsed from /proc/cpuinfo
      	* python/trfUtils.py
      	- Add small debug message when setting up a DBRelease from CVMFS
      	* python/trfValidation.py
      	- Ensure that auxilliary files are never validated
      	* test/test_trfUtilsDBRelease.py
      	- Add new test for DBRelease-pcache-current.tar.gz link
      	* Tag PyJobTransforms-02-03-48 (PyJobTransforms-02-03-48)
      
      	* python/trfArgClasses.py
      	- Add new auxiliaryFile property to argFile, signaling that a file should not
      	  be validated or appear in the jobReport
      	* python/trfExe.py
      	- Add memMonitor property to scriptExecutor, that will try to use the external
      	  MemoryMonitor binary to memory monitor the script executor's child process
      	- MemoryMonitor is launched immediately after the scriptExecutor's child process
      	  and the memory summary JSON file parsed afterwards and added to the executor
      	  memory report
      	- Deprecate the perfMonFile property of athenaExecutor and stop parsing any
      	  perfmon outputs
      	- For DBRelease files, dereference all soft links, allowing for use with the
      	  pre-release DBRelease-pcache-current.tar.gz link
      	- Change suffix on multiple-merged outputs after AthenaMP has run to _N (as
      	  expected by PanDA)
      	* python/trfFileUtils.py
      	- Add inpFileInterestingKeys (not yet used)
      	* python/trfReports.py
      	- Update report version number to 1.0.1
      	- Modify top level report 'resource' entry to have entries per executor
      ...
      (Long ChangeLog diff - truncated)
      492dc1e7
  4. Mar 20, 2015
    • Graeme Stewart's avatar
      * python/transform.py · 13e59631
      Graeme Stewart authored
      	- Add an extension to transform._doSteering to allow for 'manual' steering
      	* python/trfAMI.py
      	- Add proper parsing for 'plain' string and number substep types
      	* test/test_trfAMI.py
      	- Small fixes to q120 tag
      	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
      	* Tag PyJobTransforms-02-03-45 (PyJobTransforms-02-03-45)
      
      	* python/transform.py
      	- Add an extension to transform._doSteering to allow for 'manual' steering
      	* python/trfAMI.py
      	- Add proper parsing for 'plain' string and number substep types
      	* test/test_trfAMI.py
      	- Small fixes to q120 tag
      	- Update Tier-0 tag test to x309 instead of q220 (latter is mutable)
      	* Tag PyJobTransforms-02-03-45
      
      2015-03-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
      	* python/trfAMI.py, scripts/GetTfCommand.py
      	- Rollback last buggy commit r654630->r649652
      
      2015-03-17 Frank Volkmer <volkmer@cern.ch>
      	* python/trfAMI.py
      	- add isNewAMITag
      	- add suppressNonJobOptions handling
      	- add own get_ami_tag func
      	- use remove_enclosing_quotes to clean up strings
      	- adapt getTrfConfigFromAMI to new AMI tags
      ...
      (Long ChangeLog diff - truncated)
      13e59631
  5. Mar 03, 2015
    • Graeme Stewart's avatar
      * python/trfAMI.py · 2936e122
      Graeme Stewart authored
      	- Set the input filename correctly from the AMI tag, but
      	  fallback to trfDefaultFiles if the value is bad
      	* python/trfDefaultFiles.py
      	- Use more sensible variable names in getInputFileName function
      	* Tag PyJobTransforms-02-03-44 (PyJobTransforms-02-03-44)
      
      	* python/trfAMI.py
      	- Set the input filename correctly from the AMI tag, but
      	  fallback to trfDefaultFiles if the value is bad
      	* python/trfDefaultFiles.py
      	- Use more sensible variable names in getInputFileName function
      	* Tag PyJobTransforms-02-03-44
      
      2015-02-24 Graeme Stewart <graeme.andrew.stewart@cern.ch>
      	* python/transform.py
      	- If CLI file parameters are passed then do not take files from the AMI
      	  tag itself (prevents double input file defintions of different types)
      	* python/trfGraph.py
      	- Print input and output data types at INFO level
      	* test/PyJobTransforms_TestConfiguration.xml
      	- Add trfSubstepIntegration test to RTT
      	* test/test_trfAMI.py
      	- Fix q120 test as redefined by PROC
      	* test/test_trfSubstepIntegration.py
      	- Test that Reco_tf and FullChain_tf can import substeps correctly
      	* Tag PyJobTransforms-02-03-43
      ...
      (Long ChangeLog diff - truncated)
      2936e122
  6. Jan 19, 2015
    • Graeme Stewart's avatar
      Improved JSON transform signatures; added "group" field; more robust script. · 37af8d63
      Graeme Stewart authored
      New way to elevate std::bad_alloc message to "CATASTROPHE" (level = fatal + 10)
      Improved GetTfCommand.py
      Wider test coverage
      * Tag PyJobTransforms-02-03-37 (PyJobTransforms-02-03-37)
      
      	* Tag PyJobTransforms-02-03-37
      
      2014-12-18  Azzah Alshehri <azzah.aziz.alshehri@cern.ch>
      	* python/trfArgClasses.py
      	- Convert the member variable self._desc in prodsysDescription method to be a local variable.
      	- Pass the argument groups into the json signature so that the AMI can use it to layout arguments sensibly when defining 	tags
      
      2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
      	* python/trfArgs.py
      	- Add new argument group 'Metadata'
      	- Add task, job, attempt arguments in metadata group
      	- Make metadata arguments a part of standard transform arguments
      	- Fix missing group for athenaMPMergeTargetSize
      	* test/test_trfValidation.py
      	- Clean up all test logfiles
      
      2014-12-17 Graeme Stewart <graeme.andrew.stewart@cern.ch>
      	* python/transform.py
      	- Small change in startup message
      	* python/trfExe.py
      ...
      (Long ChangeLog diff - truncated)
      37af8d63
  7. Nov 05, 2014
    • Graeme Stewart's avatar
      * trfExe.py · b90945b7
      Graeme Stewart authored
      	- Change default option in hybridMerge to be "slow" merging, due
      	to persistent issues with fast merged metadata
      	* trfReports.py
      	- Fix bad reference to _maxMsgLen in python report generator
      	* trfValidation.py
      	- Add file count validation for DAOD merging transform
      	* test_AODMerge_tf.py
      	- Account for metadata 'compression' in metadata value test case
      	* Tag PyJobTransforms-02-03-29 (PyJobTransforms-02-03-29)
      
      	* trfExe.py
      	- Change default option in hybridMerge to be "slow" merging, due
      	to persistent issues with fast merged metadata
      	* trfReports.py
      	- Fix bad reference to _maxMsgLen in python report generator
      	* trfValidation.py
      	- Add file count validation for DAOD merging transform
      	* test_AODMerge_tf.py
      	- Account for metadata 'compression' in metadata value test case
      	* Tag PyJobTransforms-02-03-29
      
      2014-09-19 Graeme Stewart <graeme.andrew.stewart@cern.ch>
          * trfAMI.py
          - Introduce test for 'new transform' and make sure that command line
          arguments are prefixed by "--"
          * trfReport.py
          - Truncate very long error messages in exitMsg
          * trfArgClasses.py
          - Refactor EOS globbing code to improve robustness and error messages
          * Tag PyJobTransforms-02-03-28
      ...
      (Long ChangeLog diff - truncated)
      b90945b7
  8. Sep 19, 2014
    • Graeme Stewart's avatar
      * trfExe.py · e7c117ca
      Graeme Stewart authored
              - Change format of DAOD output filename to match
                grid expectations
          * Tag PyJobTransforms-02-03-26 (PyJobTransforms-02-03-26)
      e7c117ca
Loading