This branch adds a few python modules and classes to support the
reporting of additional metadata produced in event generation. It
is also needed to fix a bug reported in AGENE-1586


Former-commit-id: a5aed9f264b0efd1c1887253754608b2f0258d7b

Former-commit-id: 20cf8aebf12692d816aa03e99c5c32dbc3d087fd

Former-commit-id: 4863837fd41550b07cea1a5ee5097996c085a883

Former-commit-id: b3f5a0ec35ca31991113b5a74be1b380698a5bc8

Given that the zip binary may not be available at all grid sites, this
branch replaces the zip binary from the back-end of the Archiving
transform with python's native zip capability. It can handle ZIP files
that use the ZIP64 extensions, that is ZIP files that are more than
4 GByte in size


Former-commit-id: 6d177b697cbcbd981a948f5ab30b115d87cd4097

The python modules in SimuJobTransforms are not available in
AthDerivation, and Merge_tf crashes while attempting to import
them for its HITSMerge substep. This branch makes the import
of these modules conditional to their availability in the release


Former-commit-id: 4557e6035658e3db7f09ef700a837936a91f7a03

Former-commit-id: d8019edbc3b8f247d5d77089a5be5686fec2b27b

Former-commit-id: b0745ecb885911a18dcd3064ee126ad1a260161d

This new transform implements the merging of any data type: RAW, ESD,
AOD, TAG, HIST, HITS, RDO, EVNT and NTUP. Any combination of input
data types can be used in the same job, e.g.,

Merge_tf.py --inputESDFile <inputESDFile> --inputAODFile <inputAODFile> ...


Former-commit-id: df061c6b0c5a1adbeec739d65311c1bc720a9eae

Former-commit-id: ec18aba959688e2179ea093ae237b1290306a89e

The transform "Merge_tf.py" takes any data type as input


Former-commit-id: 670af611ff9dccedb65a0468e86905398e4a8993

It has been demonstrated that the checkpoint-restart technique is not
bringing us any benefits. On contrary, by keeping dmtcp in atlasexternals
we are causing troubles to building our software on PowerPC and MacOS


Former-commit-id: 9edfd0c8

Former-commit-id: 3e40b830a733bf49794e14d40c736b0fa65877b7

The implementation of trfJobReport, had assumption of generating one
jobReport per job. This is not valid on grid jobs, since a text report
is automatically generated by default in addition to JSON report.
This caused summing of dbTime and dbTotal, each time a report is
generated. This fix separates logic for report data generation from
that of report output for each format. The data dictionary is generated
only once and reused for each report format. The fix eliminates
the assumption of one report per job.


Former-commit-id: de0586891b691be8dbedfc187dc2be89527bef31

For jobs running on grid, PyJobTransforms detects the non-interactive
environment and enables text report. The debug message would help
detecting potential issues on grid jobs which otherwise would be
difficult to reproduce in interactive mode.


Former-commit-id: 43b31f69c027fff8120c241c154dc00774ff0bd3

This branch provides support for activating the Event Service mode
of AthenaMP for general transforms, specifically in reconstruction.
It also updates the support to simulation transforms by refactoring
the existing code.


Former-commit-id: 8695d7f118345e1c2e6ea3c753df0991aea283db

simplejson was an extra module required for very old python versions
(<2.6). It is now present in all python versions actively used
as the "json" module.


Former-commit-id: c80ebbc2dba16674017e4510a229c6ec169698aa

This new class will support substep aware configurations through
string arguments


Former-commit-id: b73da89df2df655516065011188391f2e11f827f

Former-commit-id: cbf41cd4d5e686e1254c566e1f9d2448d4ca0605

Ensuring that stopped particle TRs can be written alone (without a hit
file), and that we can filter TRs


Former-commit-id: f032c5b899765db73b1db487632cd712e1c768d6

This branch provides transform support for WebDAV TURLs. Remote input
files may be specified with davs://


Former-commit-id: 9c79201435d3d8c535acf02b4a0f94057ed24d10

This branch provides transform support for WebDAV TURLs. Remote input
files may be specified with davs://


Former-commit-id: 6ff72308ca080fbf0530cf62d9e5db17f35dd443

This branch fixes the incorrect parsing of service names from a sub-step's
log file during log file validation. Long service names are sometimes
listed in the job summary section of the log file without a space
between the name and the character '|'. Before the fix the log parser
was identifying these lines from the summary as additional errors. With
the fix in this branch the parser ignores such lines.


Former-commit-id: a7d5cc4313ddaa11ff788ec1c75f689027124443

This reverts commit e2e310cf [formerly ec540e4f5d111d010ca27b26069ff7a2c8168e9d].


Former-commit-id: 8bfe511a86519c2db64c9d2179782fbc1359784e

This reverts commit ff0b4f24 [formerly 6b05cc7355ee685c06b2d3cbd7f6850ec12b99c6], reversing
changes made to 446c33ac [formerly 22969e11386ce422c4f8503c02b318b4add52c96].


Former-commit-id: ec540e4f5d111d010ca27b26069ff7a2c8168e9d

trfValidation.py, trfUtils.py, trfExe.py:
Provide an option in trfUtils such that callers can request substep
names to be removed from log lines, before parsing.
The presence of substep name 'in some log files', caused log scanner
to skip those lines since the log scaner simply looks for standard log
pattern SERVICE-LEVEL-MESSAGE.
With this update, scan behavior becomes independent of presence
(or absence) of substep names in log files.

test_trfValidation.py:
Add unit tests for log scan with Serial and MP use cases.


Former-commit-id: 9cf03f23a9801288a695764ab7e407d2bbf51319

The archiveExecutor uses the zip command to archive files with a default
compression level of -6. In this branch the compression level is set
to -0 (no compression)


Former-commit-id: c3cc93e4d602ad954b211c4ecf39bb82c58cd9e6

The introduction of a constructor method (__init__) in
archiveExecutor was not backward compatible with an implementation of
archiveExecutor in PATJobTransforms. In this branch the constructor
is deleted and its intended functionality is implemented in a way that
makes the constructor backward compatible


Former-commit-id: d060494c3b150f74e4594a4ac84b960d4691f00b

Fix the typo in function call of LogScanExecutor

Redefine timestamp charactor list to include charactors from timestamps
added by AthenaMP. The inconsistency in log scan results was due to
different timestamp formats in log files generated by serial jobs
and AthenaMP jobs. That additional timestamp in case of AthenaMP is
a debug feature. AthenaMP adds it during concatenation of worker logs.

Add unit test to validate correct worst error, when the log file
contains a combination of two time stamps as described above.


Former-commit-id: e5ae683f2a661435bf39cbfaa6a695d36f1c0428

This branch provides support for handling the output of AthenaMP in sharedWriter
mode. AthenaMP typically produces an output file per core, which are then merged
after processing. However, in sharedWriter mode, AthenaMP produces a single output
file and no merging is necessary. A new argument flag was added (--sharedWriter True)
to instruct the transforms to properly handle the single output file.


Former-commit-id: c3e85a35c7b09fbc7b864910198d08bf0e6c1988

This brach fixes the output of athenaMP when this is activated with
only one processor. In this particular case, an output file with the
correct name was not produced. The single-core worker output was
symlinked correctly in the top directory, but it was not self merged
(or moved) and renamed to the output file name provided by the
user. In this branch, it was chosen to move the single worker output
file to the top directory and rename it with the output file name
given as an argument to the transform.


Former-commit-id: 4a427bfb3063401c7ee219009ae9844302d2fb44

The functionality of Archive_tf.py has been expanded, not only
to archive files into a tar format (with or without compression),
but also to zip files. This functionality was requested by ADC to
manage large numbers of small outputs, which need to be stored for
metadata purposes. The extended functionality allows to zip together
the small output files before storage.


Former-commit-id: 24d16b6ef51f5a04ac23c3adebaeac5ff5f08dbe

Former-commit-id: db3badf481fc8b5dad01d412a6f01dafdbb24339

steering


Former-commit-id: 0731180deb51f18a0ade3a5ace50a536cbef5beb

Former-commit-id: e1cb969e45803203f6e07653948c54acdc862f73

After restarting from a checkpoint image, AthenaMP first updates some of
the configuration parameters (e.g. input file, output file, max events)
from runargs file in the master run directory and only after that
proceeds with forking worker processes.

At this point implemented only the mechanism that allows changing the number
of workers. Transform updated such that the number of workers gets written
out to runargs file, and AthenaMP picks this number from there.


Former-commit-id: 1385252f5c74f4554a929ffc5496f61257ad454f

1. Introduced new utility in trfUtils.py which selects available
port for dmtcp (both checkpoint and restart)

2. Transform uses the value of the --restart command-line option as
a full path to the checkpoint image tarball


Former-commit-id: 7d61f09946ddf272df298eaa4e295a22dd332883

the run directory


Former-commit-id: 26b4245e496f968faf6fc7e74fd9c766bafb9e8c

PyJobTransforms: introduced two new command-line options --checkpoint (bool)
and --restart (string). It is yet to be defined what value we should be passing
to the --restart option

AthenaMP: added code to checkpoint itself just before forking and then to
exit immediately. The code cannot yet be released because it depends on private
installation of DMTCP 2.4.5. Pending the appearance of DMTCP as Athena external.

Former-commit-id: 72d35ff6c66c6671cb88ad32cd771f5f2a380f0d

The time reporting info includes cpu and wall times used during
transformSetup, inFileValidation, outFileValidation, and
reportGeneration.
All time values which were integers in earlier versions of jobReport,
are now reported as floating point values rounded to 3 decimal places.
Move utility methods calcCpuTime & calcWallTime from trfExe to trfUtils.


Former-commit-id: 7c036b7f41b370f82b3e322859c6321c6e6ac5df