Skip to content
Snippets Groups Projects

Fix bugs in MET trigger reco sequences

Merged Fix bugs in MET trigger reco sequences
ggallard/athena:METSeqFix into master
Merged Gabriel Gallardo requested to merge ggallard/athena:METSeqFix into master

@khoo discovered that the MET trigger sequences were being attached to HLTFSCellMakerRecoSequence, HLTFSTopoRecoSequence, and jetRecoSequence. The effect of this was that the MET trigger sequences were run every time these three sequences were.

This bug is addressed here by creating new MET trigger sequences, and adding the three sequences above as steps in the new sequences.

Sep 17: At the moment the generateMenuMT test is failing due to a timeout resulting from some multi-threading bug. This is not related to this merge request.

Edited by Gabriel Gallardo

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • ATLAS Robot
    ATLAS Robot @atlasbot ·
    Developer

    This merge request affects 1 package:

    • Trigger/TriggerCommon/TriggerMenuMT
  • ATLAS Robot added Trigger TriggerMenu master review-pending-level-1 labels

    added Trigger TriggerMenu master review-pending-level-1 labels

  • Teng Jian Khoo
    Teng Jian Khoo @khoo ·
    Developer

    Hi Gabriel. Do you have a printout of the updated sequence structure? I might have expected this to change the logs wrt the reference, although maybe it's not in the segment of the log that is actually checked.

  • Gabriel Gallardo
    Gabriel Gallardo @ggallard ·
    Author

    Hi TJ,

    The part that is checked by the reference is unchanged. grepping for EFMET in the sequence structure, it shows up only in this bit:

    +-- AthSequencer/Step1_1met_cell
  +-- RoRSeqFilter/FStep1_1met_cell
  +-- AthSequencer/Step1_1met_cell_view
    +-- AthSequencer/Step1_1met_cell_reco
      +-- AthSequencer/MetCellAthSequence
        +-- InputMakerForRoI/IMclusterFS
        +-- AthSequencer/metCellRecoSequence
          +-- AthSequencer/ClusterRecoSequenceFS
            +-- HLTCaloCellMaker/HLTCaloCellMakerFS
          +-- EFMissingETAlgMT/EFMET_cell
    +-- TrigMissingETHypoAlgMT/METHypoAlg_cell
+-- AthSequencer/Step1_1met_jets
  +-- RoRSeqFilter/FStep1_1met_jets
  +-- AthSequencer/Step1_1met_jets_view
    +-- AthSequencer/Step1_1met_jets_reco
      +-- AthSequencer/MetJetAthSequence
        +-- InputMakerForRoI/IMclusterFS
        +-- AthSequencer/metJetRecoSequence
          +-- AthSequencer/JetRecSeq_a4_tc_em_subjes
            +-- AthSequencer/TopoClusterRecoSequenceFS
              +-- HLTCaloCellMaker/HLTCaloCellMakerFS
              +-- TrigCaloClusterMakerMT/TrigCaloClusterMakerMT_topoFS
                +-- CaloTopoClusterMaker/TrigTopoMaker
                +-- CaloTopoClusterSplitter/TrigTopoSplitter
                +-- CaloClusterMomentsMaker/TrigTopoMoments
            +-- PseudoJetAlgorithm/pjgalg_EMTopo
            +-- EventDensityAthAlg/HLT_EventDensity_Kt4EMTopoAlg
            +-- JetAlgorithm/jetalg_HLT_AntiKt4EMTopoJets_subjes
          +-- EFMissingETAlgMT/EFMET_mht
    +-- TrigMissingETHypoAlgMT/METHypoAlg_mht
+-- AthSequencer/Step1_1met_clusterpufit
  +-- RoRSeqFilter/FStep1_1met_clusterpufit
  +-- AthSequencer/Step1_1met_clusterpufit_view
    +-- AthSequencer/Step1_1met_clusterpufit_reco
      +-- AthSequencer/MetClusterPufitAthSequence
        +-- InputMakerForRoI/IMclusterFS
        +-- AthSequencer/metClusterPufitRecoSequence
          +-- AthSequencer/TopoClusterRecoSequenceFS
            +-- HLTCaloCellMaker/HLTCaloCellMakerFS
            +-- TrigCaloClusterMakerMT/TrigCaloClusterMakerMT_topoFS
              +-- CaloTopoClusterMaker/TrigTopoMaker
              +-- CaloTopoClusterSplitter/TrigTopoSplitter
              +-- CaloClusterMomentsMaker/TrigTopoMoments
          +-- EFMissingETAlgMT/EFMET_tcPufit
    +-- TrigMissingETHypoAlgMT/METHypoAlg_tcpufit
+-- AthSequencer/merged_Step1_1met_cell_Step1_1met_clusterpufit
  +-- RoRSeqFilter/Fmerged_Step1_1met_cell_Step1_1met_clusterpufit
  +-- AthSequencer/merged_Step1_1met_cell_Step1_1met_clusterpufit_view
    +-- AthSequencer/merged_Step1_1met_cell_Step1_1met_clusterpufit_reco
      +-- AthSequencer/MetCellAthSequence
        +-- InputMakerForRoI/IMclusterFS
        +-- AthSequencer/metCellRecoSequence
          +-- AthSequencer/ClusterRecoSequenceFS
            +-- HLTCaloCellMaker/HLTCaloCellMakerFS
          +-- EFMissingETAlgMT/EFMET_cell
      +-- AthSequencer/MetClusterPufitAthSequence
        +-- InputMakerForRoI/IMclusterFS
        +-- AthSequencer/metClusterPufitRecoSequence
          +-- AthSequencer/TopoClusterRecoSequenceFS
            +-- HLTCaloCellMaker/HLTCaloCellMakerFS
            +-- TrigCaloClusterMakerMT/TrigCaloClusterMakerMT_topoFS
              +-- CaloTopoClusterMaker/TrigTopoMaker
              +-- CaloTopoClusterSplitter/TrigTopoSplitter
              +-- CaloClusterMomentsMaker/TrigTopoMoments
          +-- EFMissingETAlgMT/EFMET_tcPufit
    +-- TrigMissingETHypoAlgMT/METHypoAlg_cell1_for_merged_Step1_1met_cell_Step1_1met_clusterpufit
    +-- TrigMissingETHypoAlgMT/METHypoAlg_tcpufit1_for_merged_Step1_1met_cell_Step1_1met_clusterpufit
    +-- ComboHypo/ComboHypo_merged_Step1_1met_cell_Step1_1met_clusterpufit
+-- AthSequencer/Step1_1noalg
  +-- RoRSeqFilter/FStep1_1noalg
  +-- AthSequencer/Step1_1noalg_view
    +-- AthSequencer/Step1_1noalg_reco
      +-- AthSequencer/MetCellAthSequence
        +-- InputMakerForRoI/IMclusterFS
        +-- AthSequencer/metCellRecoSequence
          +-- AthSequencer/ClusterRecoSequenceFS
            +-- HLTCaloCellMaker/HLTCaloCellMakerFS
          +-- EFMissingETAlgMT/EFMET_cell
    +-- TrigStreamerHypoAlgMT/StreamerHypoAlg

    It's been removed from the jet sequences:

    +-- AthSequencer/jetAthSequence_a4_tc_em_subjesIS
  +-- InputMakerForRoI/IMclusterFS
  +-- AthSequencer/JetRecSeq_a4_tc_em_subjesIS
    +-- AthSequencer/TopoClusterRecoSequenceFS
      +-- HLTCaloCellMaker/HLTCaloCellMakerFS
      +-- TrigCaloClusterMakerMT/TrigCaloClusterMakerMT_topoFS
        +-- CaloTopoClusterMaker/TrigTopoMaker
        +-- CaloTopoClusterSplitter/TrigTopoSplitter
        +-- CaloClusterMomentsMaker/TrigTopoMoments
    +-- PseudoJetAlgorithm/pjgalg_EMTopo
    +-- EventDensityAthAlg/HLT_EventDensity_Kt4EMTopoAlg
    +-- JetAlgorithm/jetalg_HLT_AntiKt4EMTopoJets_subjesIS

    and all the other (b-jet, clustering) sequences to which the MET algorithms have been mistakenly attached.

  • Teng Jian Khoo
    Teng Jian Khoo @khoo ·
    Developer

    Thanks! Yes, it looks good!

  • ATLAS Robot
    ATLAS Robot @atlasbot ·
    Developer

    :negative_squared_cross_mark: CI Result FAILURE

    Athena AthSimulation
    externals :white_check_mark: :white_check_mark:
    cmake :white_check_mark: :white_check_mark:
    make :white_check_mark: :white_check_mark:
    required tests :o: :white_check_mark:
    optional tests :white_check_mark: :white_check_mark:

    Full details available at NICOS MR-26495-2019-09-11-15-55
    :warning: Athena: number of compilation errors 0, warnings 2
    :warning: AthSimulation: number of compilation errors 0, warnings 1
    :pencil: For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 3179]

  • M
    Chaowaroj Wanotayaroj @maxcw ·
    Developer

    An error and a warning. Does not look like a CI problem. Please have a look.--MR L1

  • Chaowaroj Wanotayaroj added review-user-action-required label and removed review-pending-level-1 label

    added review-user-action-required label and removed review-pending-level-1 label

  • Gabriel Gallardo
    Gabriel Gallardo @ggallard ·
    Author

    One test timed out and the other couldn't find the ref file on EOS.

  • Gabriel Gallardo
    Gabriel Gallardo @ggallard ·
    Author

    Jenkins please retry a build

  • ATLAS Robot
    ATLAS Robot @atlasbot ·
    Developer

    This merge request affects 1 package:

    • Trigger/TriggerCommon/TriggerMenuMT
  • ATLAS Robot added review-pending-level-1 label and removed review-user-action-required label

    added review-pending-level-1 label and removed review-user-action-required label

  • ATLAS Robot
    ATLAS Robot @atlasbot ·
    Developer

    :negative_squared_cross_mark: CI Result FAILURE

    Athena AthSimulation
    externals :white_check_mark: :white_check_mark:
    cmake :white_check_mark: :white_check_mark:
    make :white_check_mark: :white_check_mark:
    required tests :o: :white_check_mark:
    optional tests :white_check_mark: :white_check_mark:

    Full details available at NICOS MR-26495-2019-09-11-23-06
    :warning: Athena: number of compilation errors 0, warnings 1
    :warning: AthSimulation: number of compilation errors 0, warnings 1
    :pencil: For experts only: Jenkins output [CI-MERGE-REQUEST-CC7 3192]

  • Gabriel Gallardo
    Gabriel Gallardo @ggallard ·
    Author

    The generateMenuMT test timed out again with this log tail:

    THistSvc                                6   0     DEBUG updateFiles()
IncidentProcAlg2                        6   0     DEBUG Calling 'CoreDumpSvc' for incident [EndEvent]
IncidentProcAlg2                        6   0     DEBUG Calling 'AlgContextSvc' for incident [EndEvent]
IncidentProcAlg2                        6   0     DEBUG Calling 'ToolSvc.MuonLayerHoughTool' for incident [EndEvent]
ToolSvc.MuonLayerHoughTool              6   0     DEBUG  clearing cache at end of event 
IncidentProcAlg2                        6   0     DEBUG Calling 'ToolSvc.MuPatHitTool' for incident [EndEvent]
ToolSvc.MuPatHitTool                    6   0     DEBUG  clearing cache at end of event 
IncidentProcAlg2                        6   0     DEBUG Calling 'ToolSvc.TrigMuonTrackSteering.MuPatCandidateTool' for incident [EndEvent]
IncidentProcAlg2                        6   0     DEBUG Calling 'ToolSvc.MuPatCandidateTool' for incident [EndEvent]
IncidentProcAlg2                        6   0     DEBUG Calling 'TrigTimerSvc' for incident [EndEvent]
IncidentProcAlg2                        6   0     DEBUG Calling 'Athena::RCUSvc' for incident [EndEvent]
IncidentSvc                             6   0     DEBUG Collecting listeners fired on context s: 0  e: 6
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
:
:
:
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset
TrigSignatureMoniMT                               DEBUG Publishing Rate Histogram and Reset


75% tests passed, 1 tests failed out of 4

Label Time Summary:
TriggerMenuMT    = 4028.41 sec*proc (4 tests)

Total Test time (real) = 4062.75 sec

The following tests FAILED:
	1522 - TriggerMenuMT_generateMenuMT_ctest (Timeout)
[ERROR_MESSAGE]
Errors while running CTest

Info: unit tests completed for project Athena, name and date: unit tests, exit code: 8

    This is reproducible in a local test.

    @bernius are you the right person to ask help from for this? I don't immediately see how my changes could have caused this.

  • Catrin Bernius
    Catrin Bernius @bernius ·
    Developer

    Hi @ggallard, it took me several attempts to see the full log file....why is the debug output turned on, do you know? This shouldn't be the default as you get lots/too many of printouts (tag to @tbold, maybe he knows?). I see that lines are truncated,

    .....first 40000 lines, bottom 35000 lines .....as well as lines around error messsage are displayed

    Looking through I noticed a typeerror:

    Py:TriggerMenuMT.HLTMenuConfig.Menu.GenerateMenuMT DEBUG Finished with retrieving chain configuration for chain ChainProp(name='HLT_e3_etcut_L1EM3', l1SeedThresholds=[], stream=['Main'], groups=['RATE:SingleElectron', 'BW:Electron'], mergingStrategy='parallel', mergingOrder=[], mergingOffset=-1, topoStartFrom=False) Traceback (most recent call last): File "/cvmfs/sft.cern.ch/lcg/releases/LCG_96/Python/2.7.16/x86_64-centos7-gcc8-opt/lib/python2.7/logging/init.py", line 868, in emit msg = self.format(record) File "/cvmfs/sft.cern.ch/lcg/releases/LCG_96/Python/2.7.16/x86_64-centos7-gcc8-opt/lib/python2.7/logging/init.py", line 741, in format return fmt.format(record) File "/cvmfs/sft.cern.ch/lcg/releases/LCG_96/Python/2.7.16/x86_64-centos7-gcc8-opt/lib/python2.7/logging/init.py", line 465, in format record.message = record.getMessage() File "/cvmfs/sft.cern.ch/lcg/releases/LCG_96/Python/2.7.16/x86_64-centos7-gcc8-opt/lib/python2.7/logging/init.py", line 329, in getMessage msg = msg % self.args TypeError: not all arguments converted during string formatting Logged from file TriggerConfigHLT.py, line 36

    which comes from from a print out statement here: https://gitlab.cern.ch/atlas/athena/blob/master/Trigger/TriggerCommon/TriggerMenuMT/python/HLTMenuConfig/Menu/TriggerConfigHLT.py#L36:

            log.debug("Registered chain ", chainConfig.name )

    But this shouldn't be the reason for your problem (though not 100% sure), it should be fixed though at some point though.

    Have you tried running the test by hand in your local area? Does it run ok there? I've looked through the log file but it's really hard to see if there are any other errors listed, and the search/print/download function currently don't work for me. Cheers, Catrin

  • Catrin Bernius
    Catrin Bernius @bernius ·
    Developer

    Downloading the 722 (!) pages of log file, I found this error message:

    InDetPixelClusterization_MuonFS ERROR SG::ExcNullReadHandle: Dereference of read handle failed: IMEFCBFS_view_1+Pixel AlgoExecutionTask ERROR SG::ExcNullReadHandle: Dereference of read handle failed: IMEFCBFS_view_1+Pixel

    Though I also doubt that this comes from your changes/only flagged up for your changes.

    I would say that the DEBUG printout needs to get sorted as well as the log.debug("Registered chain ", chainConfig.name ), and then the CI should be restarted.

  • Gabriel Gallardo
    Gabriel Gallardo @ggallard ·
    Author

    Thanks for looking at this @bernius!

    The TypeError seems to be from the egamma set up sequences. @fernando , are you the right person to tag here?

    As for the excessive DEBUG output, @tbold is there a switch I can turn off somewhere?

    When I run the test locally, I just get repeated outputs of this line:
    TrigSignatureMoniMT DEBUG Publishing Rate Histogram and Reset

    Gabriel

    Edited by Gabriel Gallardo
  • Catrin Bernius
    Catrin Bernius @bernius ·
    Developer

    Are you sure the type error it's from the egamma setup? I think it really comes from this line log.debug("Registered chain ", chainConfig.name ) which is here: https://gitlab.cern.ch/atlas/athena/blob/master/Trigger/TriggerCommon/TriggerMenuMT/python/HLTMenuConfig/Menu/TriggerConfigHLT.py#L36 It also says below the line of the type error:

    Logged from file TriggerConfigHLT.py, line 36 There are 45 occurrences of this error in the log file, also for jet chains. So @ferando, this is not related to you.

    @ggallard, could you please comment out this line for now or find a fix for it?

  • Gabriel Gallardo added 45 commits

    added 45 commits

    Compare with previous version

  • Gabriel Gallardo added 1 commit

    added 1 commit

    • bd810d5d - Comment out error-prone debug statement

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
Please register or sign in to reply