This branch adds a few python modules and classes to support the
reporting of additional metadata produced in event generation. It
is also needed to fix a bug reported in AGENE-1586


Former-commit-id: a5aed9f264b0efd1c1887253754608b2f0258d7b

Former-commit-id: 20cf8aebf12692d816aa03e99c5c32dbc3d087fd

Former-commit-id: 4863837fd41550b07cea1a5ee5097996c085a883

Former-commit-id: b3f5a0ec35ca31991113b5a74be1b380698a5bc8

Given that the zip binary may not be available at all grid sites, this
branch replaces the zip binary from the back-end of the Archiving
transform with python's native zip capability. It can handle ZIP files
that use the ZIP64 extensions, that is ZIP files that are more than
4 GByte in size


Former-commit-id: 6d177b697cbcbd981a948f5ab30b115d87cd4097

Former-commit-id: d8019edbc3b8f247d5d77089a5be5686fec2b27b

It has been demonstrated that the checkpoint-restart technique is not
bringing us any benefits. On contrary, by keeping dmtcp in atlasexternals
we are causing troubles to building our software on PowerPC and MacOS


Former-commit-id: 9edfd0c81e75b27eede3cc3712c1c460467b037e

The implementation of trfJobReport, had assumption of generating one
jobReport per job. This is not valid on grid jobs, since a text report
is automatically generated by default in addition to JSON report.
This caused summing of dbTime and dbTotal, each time a report is
generated. This fix separates logic for report data generation from
that of report output for each format. The data dictionary is generated
only once and reused for each report format. The fix eliminates
the assumption of one report per job.


Former-commit-id: de0586891b691be8dbedfc187dc2be89527bef31

For jobs running on grid, PyJobTransforms detects the non-interactive
environment and enables text report. The debug message would help
detecting potential issues on grid jobs which otherwise would be
difficult to reproduce in interactive mode.


Former-commit-id: 43b31f69c027fff8120c241c154dc00774ff0bd3

This branch provides support for activating the Event Service mode
of AthenaMP for general transforms, specifically in reconstruction.
It also updates the support to simulation transforms by refactoring
the existing code.


Former-commit-id: 8695d7f118345e1c2e6ea3c753df0991aea283db

simplejson was an extra module required for very old python versions
(<2.6). It is now present in all python versions actively used
as the "json" module.


Former-commit-id: c80ebbc2dba16674017e4510a229c6ec169698aa

This new class will support substep aware configurations through
string arguments


Former-commit-id: b73da89df2df655516065011188391f2e11f827f

Former-commit-id: cbf41cd4d5e686e1254c566e1f9d2448d4ca0605

Ensuring that stopped particle TRs can be written alone (without a hit
file), and that we can filter TRs


Former-commit-id: f032c5b899765db73b1db487632cd712e1c768d6

This branch provides transform support for WebDAV TURLs. Remote input
files may be specified with davs://


Former-commit-id: 9c79201435d3d8c535acf02b4a0f94057ed24d10

This branch provides transform support for WebDAV TURLs. Remote input
files may be specified with davs://


Former-commit-id: 6ff72308ca080fbf0530cf62d9e5db17f35dd443

This branch fixes the incorrect parsing of service names from a sub-step's
log file during log file validation. Long service names are sometimes
listed in the job summary section of the log file without a space
between the name and the character '|'. Before the fix the log parser
was identifying these lines from the summary as additional errors. With
the fix in this branch the parser ignores such lines.


Former-commit-id: a7d5cc4313ddaa11ff788ec1c75f689027124443

This reverts commit e2e310cf [formerly ec540e4f].


Former-commit-id: 8bfe511a

This reverts commit ff0b4f24 [formerly 6b05cc7355ee685c06b2d3cbd7f6850ec12b99c6], reversing
changes made to 446c33ac [formerly 22969e11386ce422c4f8503c02b318b4add52c96].


Former-commit-id: ec540e4f5d111d010ca27b26069ff7a2c8168e9d

trfValidation.py, trfUtils.py, trfExe.py:
Provide an option in trfUtils such that callers can request substep
names to be removed from log lines, before parsing.
The presence of substep name 'in some log files', caused log scanner
to skip those lines since the log scaner simply looks for standard log
pattern SERVICE-LEVEL-MESSAGE.
With this update, scan behavior becomes independent of presence
(or absence) of substep names in log files.

test_trfValidation.py:
Add unit tests for log scan with Serial and MP use cases.


Former-commit-id: 9cf03f23a9801288a695764ab7e407d2bbf51319

The archiveExecutor uses the zip command to archive files with a default
compression level of -6. In this branch the compression level is set
to -0 (no compression)


Former-commit-id: c3cc93e4d602ad954b211c4ecf39bb82c58cd9e6

The introduction of a constructor method (__init__) in
archiveExecutor was not backward compatible with an implementation of
archiveExecutor in PATJobTransforms. In this branch the constructor
is deleted and its intended functionality is implemented in a way that
makes the constructor backward compatible


Former-commit-id: d060494c3b150f74e4594a4ac84b960d4691f00b

Fix the typo in function call of LogScanExecutor

Redefine timestamp charactor list to include charactors from timestamps
added by AthenaMP. The inconsistency in log scan results was due to
different timestamp formats in log files generated by serial jobs
and AthenaMP jobs. That additional timestamp in case of AthenaMP is
a debug feature. AthenaMP adds it during concatenation of worker logs.

Add unit test to validate correct worst error, when the log file
contains a combination of two time stamps as described above.


Former-commit-id: e5ae683f2a661435bf39cbfaa6a695d36f1c0428

This branch provides support for handling the output of AthenaMP in sharedWriter
mode. AthenaMP typically produces an output file per core, which are then merged
after processing. However, in sharedWriter mode, AthenaMP produces a single output
file and no merging is necessary. A new argument flag was added (--sharedWriter True)
to instruct the transforms to properly handle the single output file.


Former-commit-id: c3e85a35c7b09fbc7b864910198d08bf0e6c1988

This brach fixes the output of athenaMP when this is activated with
only one processor. In this particular case, an output file with the
correct name was not produced. The single-core worker output was
symlinked correctly in the top directory, but it was not self merged
(or moved) and renamed to the output file name provided by the
user. In this branch, it was chosen to move the single worker output
file to the top directory and rename it with the output file name
given as an argument to the transform.


Former-commit-id: 4a427bfb3063401c7ee219009ae9844302d2fb44

The functionality of Archive_tf.py has been expanded, not only
to archive files into a tar format (with or without compression),
but also to zip files. This functionality was requested by ADC to
manage large numbers of small outputs, which need to be stored for
metadata purposes. The extended functionality allows to zip together
the small output files before storage.


Former-commit-id: 24d16b6ef51f5a04ac23c3adebaeac5ff5f08dbe

Former-commit-id: db3badf481fc8b5dad01d412a6f01dafdbb24339

steering


Former-commit-id: 0731180deb51f18a0ade3a5ace50a536cbef5beb

Former-commit-id: e1cb969e45803203f6e07653948c54acdc862f73

After restarting from a checkpoint image, AthenaMP first updates some of
the configuration parameters (e.g. input file, output file, max events)
from runargs file in the master run directory and only after that
proceeds with forking worker processes.

At this point implemented only the mechanism that allows changing the number
of workers. Transform updated such that the number of workers gets written
out to runargs file, and AthenaMP picks this number from there.


Former-commit-id: 1385252f5c74f4554a929ffc5496f61257ad454f

1. Introduced new utility in trfUtils.py which selects available
port for dmtcp (both checkpoint and restart)

2. Transform uses the value of the --restart command-line option as
a full path to the checkpoint image tarball


Former-commit-id: 7d61f09946ddf272df298eaa4e295a22dd332883

the run directory


Former-commit-id: 26b4245e496f968faf6fc7e74fd9c766bafb9e8c

PyJobTransforms: introduced two new command-line options --checkpoint (bool)
and --restart (string). It is yet to be defined what value we should be passing
to the --restart option

AthenaMP: added code to checkpoint itself just before forking and then to
exit immediately. The code cannot yet be released because it depends on private
installation of DMTCP 2.4.5. Pending the appearance of DMTCP as Athena external.

Former-commit-id: 72d35ff6c66c6671cb88ad32cd771f5f2a380f0d

The time reporting info includes cpu and wall times used during
transformSetup, inFileValidation, outFileValidation, and
reportGeneration.
All time values which were integers in earlier versions of jobReport,
are now reported as floating point values rounded to 3 decimal places.
Move utility methods calcCpuTime & calcWallTime from trfExe to trfUtils.


Former-commit-id: 7c036b7f41b370f82b3e322859c6321c6e6ac5df

Takashi Yamanaka authored 7 years ago and Marcelo Vogel committed 7 years ago

Added a limit to the number of detail messages in the job report from G4Exceptions (ATLASJT-357)

See merge request !1045

Former-commit-id: 698c8b2e698157b3e54d2479090a91f4c8df2ec8

Takashi Yamanaka authored 7 years ago and Marcelo Vogel committed 7 years ago

Changed default pool merging executable in the transforms to latest version (ATLASJT-353)

See merge request !722

Former-commit-id: a24b32ad30cfedf0a5a560ce4395b0e9720befaf

	* python/transform.py
	- Added properties for cpu and walltime for transformSetup, inFileValidation
	  and outFileValidation.
	* python/trfReports
	- Added time reporting key-value pairs for transformSetup, inFileValidation
	  and outFileValidation.
	* python/trfUtils
	- Added utility methods calcCpuTime and CalcWallTime.
	* Tagging PyJobTransforms-03-00-34  [ATLASJT-291]


Former-commit-id: ecb568dbd078b855d6a005861339a3c089c43cdb

2017-01-13 Marcelo Vogel <mavogel@cern.ch>
	* python/trfExe.py
	- AthenaMP output files do not get suffix _000 in eventService mode
	* Tagging PyJobTransforms-03-00-33

2017-01-10 Marcelo Vogel <mavogel@cern.ch>
	* python/trfExe.py
	- Exclude metadata container MetaDataHdrForm in event fast merge. [ATLASDPD-844]
	* Tagging PyJobTransforms-03-00-32


Former-commit-id: 997eb1958b2b7fe6a01674b18016eacccb3baba7

Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346] (PyJobTransforms-03-00-31)

	* python/trfUtils.py
	- Added utility method isInteractiveEnv()
	* python/trfExe.py
	- Refactored checks for interactive environment using new method in trfUtils.
	* python/transform.py
	- Enabled default dump of environment into env.txt for non-interactive jobs. [ATLASJT-346]
	* Tagging PyJobTransforms-03-00-31

2016-11-17 Marcelo Vogel <mavogel@cern.ch>
	* scripts/makeTrfJSONSignatures.py
	- Replace script by updated version provided by AMI team [ATLASJT-284]
	* Tagging PyJobTransforms-03-00-30

2016-11-14 Marcelo Vogel <mavogel@cern.ch>
	* python/transform.py
	- Add property trfPredata to be set from env variable at trf init
	* python/trfReports.py
	- Add trfPredata to job report under key 'trfPredata'
	- Report version 2.0.7
	* Tagging PyJobTransforms-03-00-29
...
(Long ChangeLog diff - truncated)


Former-commit-id: 0db01cfba03324f8b6939d2f979bc9b02f861660

Remove = after pre/postInclude in output of GetTfCommand.py [ATLASAMITG-99] (PyJobTransforms-03-00-24)

	* scripts/GetTfCommand.py
	- Fix bug preventing correct output for option --printOnlyCmdLine
	* python/trfAMI.py
	- Remove '=' after pre/postInclude in _argsToString() [ATLASAMITG-99]
	* Tagging PyJobTransforms-03-00-24

2016-09-09 Marcelo Vogel <mavogel@cern.ch>
	* python/transform.py
	- Configure Event Service job via a special argument
	  eventService=True, which expands to other parameters [ATLASJT-320]
	* python/trfMPTools.py
	- Add switch to turn on/off output file checks in athenaMPOutputHandler()
	* python/trfExe.py
	- Turn on/off output file checks in athenaMP if eventService=False/True
	* test/test_trfMPTools.py
	- Update interface of athenaMPOutputHandler() in unittest
	* Tagging PyJobTransforms-03-00-23

2016-08-29 Umesh Worlikar <umesh.worlikar@cern.ch>
        * python/trfArgClasses.py
...
(Long ChangeLog diff - truncated)


Former-commit-id: e4f14ea8