Event format meta data

Picking up from the discussion at !33803 (closed)

@akraszna implemented a very nice start to handling xAOD::EventFormat metadata in athenaMT. @gemmeren Pointed out that we should adopt a uniform approach to handling EventStreamInfo and xAOD::EventFormat. This leaves some approaches:

• Write multithreaded service maintaining one shared EventStreamInfo object in a thread-safe manner.
• Maintain a xAOD::EventFormat object for each output stream.
• Something we have not yet thought of Here we choose the second, i.e. maintain a xAOD::EventFormat object for each output stream. That means the key of the xAOD::EventFormat object now is a function of the stream name. Trying to adopt the EventStreamInfo style I went for something like:

AthenaOutputStream(
    name=name,
    HelperTools=[
        CfgMgr.xAODMaker__EventFormatStreamHelperTool(
            name="EventFormatStreamHelper"
            Key='Format{}'.format(name),
        ),
    ],
)

Of course any client looking up xAOD::EventFormat by key will have to adjust for that, as you can see in the read test. How much of a nightmare will this be?

I have not yet included recording and retrieving xAOD::EventFormat though the MetaDataSvc as implemented for EventStreamInfo. Is that desirable @mnowak

As you can see by the hacky approach and the title this is still a construction site and needs cleaning up.

Thoughts?

@tsulaia

Could MetaCont<> keyed by the output file be used to avoid the naming issue?

Hi All, I am in favor of following the EventStreamInfo approach, that is:

• maintaining separate EventFormat per output stream with a modified key that contains the streamname
• change clients to access EventFormat through some MetaDataSvc interface, so that the name change is transparent
• but how will we know which stream EventFormat do we want for every access? That was clear for ESI, but I have no idea about EventFormat.

There is also the question if we want to write EventInfo under the new name or the old original name. If we want the old name, then the OutputStream could do a trick with creating a temporary object when writing.

I think the question if we want to use a different key or a MetaCont is secondary here - both need to know the streamname and that is the bigger issue I think. Additionally MetaCont changes the object type in SG, which is an additional pain. So I prefer jsut a key change.

Attila is probably the best candidate to consider all possible places where EventFormat is accessed and give us some advice, right?

Cheers, Marcin

added 188 commits

• 51ea74bd...7542e2fd - 182 commits from branch atlas:master
• 4e1e514a - Enabled the unit test in xAODEventFormat.
• c9a3f9de - Re-designed how the xAOD::EventFormat metadata is created in Athena.
• 218da0b3 - Streamlined xAODMaker::EventFormatStreamHelperTool.
• 5f9d7ce6 - Introduced code for testing the xAOD::EventFormat creator infrastructure.
• fd703f7e - Maintain one EventFormat object for each stream
• ce3f2fad - Re-add locks in case this ever does run MT

Compare with previous version

added 1055 commits

• ce3f2fad...2b58a449 - 1045 commits from branch atlas:master
• e6edcde4 - Enabled the unit test in xAODEventFormat.
• 85efc346 - Re-designed how the xAOD::EventFormat metadata is created in Athena.
• fa4340f4 - Streamlined xAODMaker::EventFormatStreamHelperTool.
• 941cd35d - Introduced code for testing the xAOD::EventFormat creator infrastructure.
• 6744ebc6 - Maintain one EventFormat object for each stream
• d497cdbb - Re-add locks in case this ever does run MT
• 560210d8 - Add new interface component
• 703bf5b3 - Add EventFormat stream helper during POOL stream config
• 7d7822e2 - Format the name string of the HelperTool
• 013a0544 - Always read all xAOD::EventFormat Keys from input

Compare with previous version

The re-base fixed some linking issues that occurred linking the old fork to the latest master.

added 3 commits

• 92b2df56 - Test reading and writing of xAOD::EventFormat
• 54e385d0 - linter cleanup
• 62b9d40f - MetaDataStop waits for threads writing to MetaDataStore

Compare with previous version

added 1 commit

• b093b08e - Remove unused MetaDataMaker and Svc

Compare with previous version

added 738 commits

• b093b08e...e8b07cb9 - 724 commits from branch atlas:master
• 61d7a706 - Enabled the unit test in xAODEventFormat.
• 118052dd - Re-designed how the xAOD::EventFormat metadata is created in Athena.
• ff0415cb - Streamlined xAODMaker::EventFormatStreamHelperTool.
• d9f12013 - Introduced code for testing the xAOD::EventFormat creator infrastructure.
• 3bf924f3 - Maintain one EventFormat object for each stream
• d2212d56 - Re-add locks in case this ever does run MT
• 2d69ec74 - Add new interface component
• 36e47f2d - Add EventFormat stream helper during POOL stream config
• 7e9b4db8 - Format the name string of the HelperTool
• a9ec2377 - Always read all xAOD::EventFormat Keys from input
• 71010617 - Test reading and writing of xAOD::EventFormat
• cc1cf493 - linter cleanup
• bb87fdc0 - MetaDataStop waits for threads writing to MetaDataStore
• 00d2213a - Remove unused MetaDataMaker and Svc

Compare with previous version

added 2 commits

• 51778899 - Test write and read with component accumulator
• f672d816 - Deal with depreciation messages

Compare with previous version

added 1 commit

• b506a562 - Add documentation and clean up comments

Compare with previous version

added 218 commits

• b506a562...71c4ab81 - 201 commits from branch atlas:master
• 85878d10 - Enabled the unit test in xAODEventFormat.
• a045a8fc - Re-designed how the xAOD::EventFormat metadata is created in Athena.
• b1f45aa6 - Streamlined xAODMaker::EventFormatStreamHelperTool.
• 3d105e62 - Introduced code for testing the xAOD::EventFormat creator infrastructure.
• 3106b665 - Maintain one EventFormat object for each stream
• 87385ee4 - Re-add locks in case this ever does run MT
• 0091850f - Add new interface component
• 237db3d0 - Add EventFormat stream helper during POOL stream config
• 95cac798 - Format the name string of the HelperTool
• 2f433153 - Always read all xAOD::EventFormat Keys from input
• c4f6965d - Test reading and writing of xAOD::EventFormat
• 1fd3a9f2 - linter cleanup
• 77771136 - MetaDataStop waits for threads writing to MetaDataStore
• 96b68580 - Remove unused MetaDataMaker and Svc
• 3e1f8474 - Test write and read with component accumulator
• 16a9ae51 - Deal with depreciation messages
• 49afd057 - Add documentation and clean up comments

Compare with previous version

Should I set up the EventFormatMetaDataTool when the input stream is being configured?

An addition I wanted to highlight since Marcin and I were talking about it in separate emails:

The MetaDataTool locks a Mutex marking it busy for writing when starting to handle a beginInputFile incident, releasing the lock when done. The metaDataStop waits for such locks to be released. That should take care of two issues:

1. A second beginInputFile is fired while the first is busy
2. metaDataStop prepares the MetaDataStore for write out while a brginInputFile is still being handled For the first case one could be much more granular, but I am not sure this serves much purpose and complicates the code.

Thoughts?

added 77 commits

• 49afd057...c1468b61 - 59 commits from branch atlas:master
• 5d340821 - Enabled the unit test in xAODEventFormat.
• 7d13b970 - Re-designed how the xAOD::EventFormat metadata is created in Athena.
• a58a2e8e - Streamlined xAODMaker::EventFormatStreamHelperTool.
• 177895d2 - Introduced code for testing the xAOD::EventFormat creator infrastructure.
• a9d67443 - Maintain one EventFormat object for each stream
• 196583a7 - Re-add locks in case this ever does run MT
• 16df402b - Add new interface component
• 3d91f643 - Add EventFormat stream helper during POOL stream config
• bd352ae9 - Format the name string of the HelperTool
• a0e30dca - Always read all xAOD::EventFormat Keys from input
• 30c2fffb - Test reading and writing of xAOD::EventFormat
• d2d90e57 - linter cleanup
• 924dfa8c - MetaDataStop waits for threads writing to MetaDataStore
• bab894ec - Remove unused MetaDataMaker and Svc
• 35850d04 - Test write and read with component accumulator
• 194c5787 - Deal with depreciation messages
• bb0c1b09 - Add documentation and clean up comments
• b776f672 - Use MetaDataSvc interface to record EventFormat

Compare with previous version

This add the usage of the StoreGate interface @mnowak added to the MetaDataSvc to store (and retrieve) the xAOD::EventFormat. Still have to look through all other clients to make them work.

added 2 commits

• ee934a4e - Use MetaDataSvc interface to record EventFormat
• 179f86cc - Use first branch containing EventFormat in name

Compare with previous version

added 1 commit

• b57b0287 - Test deriving from files containing xAOD::EventFormat

Compare with previous version

added 364 commits

• b57b0287...80b4d51d - 363 commits from branch atlas:master
• d3bc35fe - Merge branch 'master' into 'EventFormatMetaData'

Compare with previous version

unmarked as a Work In Progress

WARNING: big files (>100K) are found in the changeset

136K in file Control/xAODRootAccess/Root/TEvent.cxx