[QEE] Turn on Inclusive Jet Lines. Requires Allen!1372

Inclusive jet HLT2 lines need to be taken off passthrough and require the new HLT1 decisions made with Allen!1372 (merged)

• Removes Hlt2QEE_IncJet10GeVFull
• Adds Hlt2QEE_IncJet(15|25|35|45)GeVFull
  • filtering on new Hlt1 Jet lines.
  • calo_clusters=True
• Removes Hlt2QEE_IncDiJet10GeVFull
• Add new prescales to Hlt2QEE_IncDiJet(15|20|25|30)GeVFull
  • benefit from the new filtering on Hlt1 Jet Lines
  • they already have calo_clusters=True
• Removes SpruceQEE_SingleJet10
• Adds `SpruceQEE_SingleJet(15|25|35|45)
  • calo_clusters=True
  • persistreco=True
  • filtering on new Hlt2QEE_IncJet()GeVFull
• Updates SpruceQEE_Trijets
  • filtering on new Hlt1 Jet lines.
• Removes `SpruceQEE_Dijets1010
• Updates SpruceQEE_Dijets()() (Except DiTopo)
  • calo_clusters=True
  • persistreco=True

10 GeV lines are removed because HLT1 filter starts at 15 GeV

Trijet lines are filtered now on HLT1 as these target primarily alpha_s (3jet vs 2jet) measurements. Since the inclusive lines will use the HLT1 filter, it makes sense also the trijet inclusive line one does as well.

Di-Topo lines are not included with persistreco and calo_clusters as these are targeting the same physics as SVtags. The nominal candidates should be enough for comparison studies, and in future, can add these back in. This saves ~50 MB/s over including them.

N.B. : BW here is calculated using unfiltered, full 30 MHz HLT1 ran minbias simulation 300k_HLT1_log.log

With Changes:

Ran over 300,000 events. 300k_HLT2_withIncJets.log

Fullstream filesize = 10.6 MB. Fullstream Events fired = 84

Fullstream Rate = 30 MHz * 84 / 300,000 = 8.4 kHz

Fullstream BW = 30 MHz * 10.6 MB / 300,000 = 1060 MB/s

Turbo filesize = 5.2 MB. 1091 events passed (overlap 13)

Exclusive Sprucing BW = 8.4 kHz * 5.8 MB / 84 = 580 MB/s

Without Inclusive Jets (Changes):

Ran over 300,000 events with inclusive jet lines prescaled to 0 300k_HLT2_NoIncJets.log

Fullstream filesize = 8.8 MB. Fullstream Events fired = 69 Fullstream Rate = 30 MHz * 69 / 300,000 = 6.9 kHz
Fullstream BW = 30 MHz * 8.8 MB / 300,000 = 880 MB/s

Turbo filesize = 1.2 MB. 1091 events passed (overlap 11)

Exclusive Sprucing BW From !3151 (merged) = 274 MB/s (Does not include Wcharm)

added 2024data-selections qee labels

assigned to @ngrieser

changed the description

changed the description

mentioned in merge request !3151 (merged)

added WIP dependency label

changed title from [QEE] Turn on Inclusive Jet Lines to [QEE] Turn on Inclusive Jet Lines{+. Requires Add inclusive jet line (Allen!1372 - merged)}

mentioned in merge request !3217 (merged)

added 1 commit

• 478916dd - First tuning round.

Compare with previous version

changed the description

@rjhunter Rough values for starting (including prescales) are now here. Will double the input stats tonight to get a bit more granularity, but it seems such values starting are reasonable.

changed the description

changed the description

changed the description

resolved all threads

changed the description

@rjhunter The rates have been significantly reduced, from 300 MB/s added estimate to 180 MB/s added by adding the OWNPV matching to dijet lines which comes with !3217 (merged) .

mentioned in merge request Allen!1372 (merged)

added 2 commits

• 7fe39186 - Format stuff.
• 825b080d - Add sprucing lines.

Compare with previous version

changed the description

added 1 commit

• 5b76f77f - Fixed formatting

Compare with previous version

This is now done from proponent side.

requested review from @rjhunter and @lugrazet

requested review from @isanders and @dzuliani

added 22 commits

• 5b76f77f...6542cdc9 - 21 commits from branch qee_upgrade_24
• 066feab1 - Merge branch 'qee_upgrade_24' into 'QEE_DiJetLinesOn'

Compare with previous version

added 2 commits

• df5a7d15 - Add persistreco to spruces that were missing.
• 7c2c28ab - Merge branch 'QEE_DiJetLinesOn' of ssh://gitlab.cern.ch:7999/lhcb/Moore into QEE_DiJetLinesOn

Compare with previous version

So to summarise my understanding of these changes: Change the hlt1_filter_jet_codes.

• Removes Hlt2QEE_IncJet10GeVFull
• Adds Hlt2QEE_IncJet(15|25|35|45)GeVFull
  • filtering on new Hlt1 Jet lines.
  • calo_clusters=True
• Removes Hlt2QEE_IncDiJet10GeVFull
• Add new prescales to Hlt2QEE_IncDiJet(15|20|25|30)GeVFull
  • benefit from the new filtering on Hlt1 Jet Lines
  • they already have calo_clusters=True
• Removes SpruceQEE_SingleJet10
• Adds `SpruceQEE_SingleJet(15|25|35|45)
  • calo_clusters=True
  • persistreco=True
  • filtering on new Hlt2QEE_IncJet()GeVFull
• Updates SpruceQEE_Trijets
  • filtering on new Hlt1 Jet lines.
• Removes `SpruceQEE_Dijets1010
• Updates SpruceQEE_Dijets()()
  • calo_clusters=True
  • persistreco=True

@ngrieser Could you confirm the above please and update the description accordingly?

The calo_clusters changes seem consistent to me .

Re: removal of 10GeV jet lines:
Can you elaborate as to why (and add it to the description)? This might be obvious but I'm not sure.

Re: SpruceQEE_Trijets:
a SpruceLine with an hlt1_filter_code but no hlt2_filter_code is definitely interesting and this is the first example I have seen.
Could you elaborate for me on the purpose of this line/filtering choice (and add it to the description). Thanks

Thanks all for the hard work here. Here's a review I have a few comments/and clarifying questions on the line changes here.

added 1 commit

• e7e4ba7d - Turn off persistreco and clusters for di-topo sprucing.

Compare with previous version

changed the description

added 1 commit

• a5ddde65 - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

added 1 commit

• 67295eb7 - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

added 1 commit

• 969ef134 - Fixed formatting

Compare with previous version

requested review from @abertoli

added Spruce_April24 label

Hello @abertoli, I noticed this MR that affects Sprucing. Are you looking into this sort of update? I added you as a reviewer for Sprucing. I also added the Spruce_April24 label but let's check. Thanks.

removed Spruce_April24 label

removed review request for @abertoli

approved this merge request

approved this merge request

approved this merge request

resolved all threads

merged

mentioned in commit 11bef38c

mentioned in merge request !3238 (closed)

@lugrazet Allen!1372 (merged) isn't merged yet. I guess that's fine, but now the larger QEE MR is blocked by this MR. I guess just something to keep in mind if for some reason this Allen one gets blocked.

I guess that everything at HLT2 works fine without the Allen MR, just that the lines filtering on the new HLT1 line will have zero rate, right? In that sense they aren't actually dependent, right?

This is correct, but if it doesn’t go in these should go back to passthrough and tuned prescales accordingly

I appreciate the reminder, from the looks of things, Allen!1372 (merged) is progressing well but just in case:
I would appreciate a clear and concise plan agreed on by both the Jet team and the QEE trigger team relatively soon in the situation where this doesn't get in in-time.
Perhaps we can organise a meeting on short notice tomorrow. -> Threw together this https://newdle.cern.ch/newdle/9YKyMHsV to agree on a time! Thanks in advance

I.e. If possible by end-of-day tomorrow if possible, as I (and of course more broadly our QEE trigger team) will need to very shortly be discussing/representing these changes to RTA/DPA maintainers/shifters/coordinators and these lines are very likely to be the centre of conversation given the very large increases.

The expected rate of HLT1PassThrough was 100 Hz. The new lines are feeding at 10 kHz or so. Since none of the inclusive prescales are at .01, you should be able to simply switch back to passthrough decision and remove all prescales on inclusive lines. Passthrough will be less efficient but will at least be unbiased. This should always be a net loss of bandwidth and should cause no issues on operations.

Thanks @lugrazet, but I don't think we need a discussion on this - I can make my stance clear. If the Allen line doesn't get in, then since this was merged then these lines don't fire. It's the responsibility of the line authors to make sure the Allen MR gets in.

Since our QEE deadline has lapsed, then any additional work for Luke and I (such as a) reverting this ourselves, or b) handling/reviewing an MR to master by others that reverts it ) goes well down the priority list. Today's deadline and our workloads must be respected.

I completely agree, we will work to get the Allen MR in

If the Allen line doesn't get in, then since this was merged then these lines don't fire.

I need to wholeheartedly disagree with Davide here. These lines feed at least 4 analyses clearly defined, which will not happen if the lines "simply do not fire." The contingency is a simple string replacement (thanks to Luke's code review), with well understood impacts on bandwidth. If there is simply not enough resources from QEE team, I will take full responsibility of the MR targeting master and find suitable reviewers beyond the group if resources are not available. Simply not having these lines is not an acceptable solution.

While I fully support the sentiment of "get the line in", it is now primarily out of our hands, and uncertainty from RTA leadership should be prepared for, and this simple contingency, as Luke asked for, was offered. @hyin @ausachov Please be aware of this situation, as I do not think a unilateral decision with impact of this nature, is meant to be made within the QEE group.

To be clearer, I meant that is obvious to me that the Jet team should do everything to get the Allen MR ready, as both Luke and Ross have done a tremendous job (as all of us). Afaik it’ll be up to management to decide if the changes we introduce in the Allen MR are fine or not, so this is not completely on us (I believe @thboettc is following this very closely). I think that we should try to find a solution where:

• We don’t place any more burden to Luke and Ross (as said by Ross)
• We still get our inclusive lines to work

I’m happy to join Nate if more work is needed during next week.

@ngrieser let me offer a compromise to you.

I think 99% of your available effort should be directed towards getting that Allen MR in: you asked us to merge this MR and we have had extensive discussions about this MR all based on the assumption that the Allen MR would get in. If you get to a point where it looks like it won't get in (and please don't let that point be Thursday or Friday), then:

• one of you make the MR to master that reverts the changes,
• AT me and I will glance over it and approve
• in parallel: handle the CI testing with the shifter yourselves (be aware this will take at least a day) and move it towards merge. Do not get it merged before !3151 (merged) and please do not slow down !3151 (merged).

!3151 (merged) will proceed on the assumption that the Allen MR gets in. If it doesn't then any other situation means less bandwidth from QEE, so I think no one from RTA/DPA will complain at that and we can have a productive discussion on the assumption that the Allen MR gets in.

Sounds good to me

mentioned in merge request !3315 (merged)