Skip to content

Add extra_params option to set process affinity

Rafal Bielski requested to merge rbielski/HLTMPPU:affinity into master

Add a new extra param affinityCoresPerThread to set process affinity. The affinity is set immediately after fork to a range of CPUs with the size of N threads * affinityCoresPerThread. The default value is 0, meaning no affinity settings are changed.

Tested with commands like:

athenaHLT.py --cfgdict='{"HLTMPPU":{"extra_params":["affinityCoresPerThread=1"]}}' --imf --threads=2 --concurrent-events=2 --nprocs=2 --number-of-events=80 --file=/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/TrigP1Test/data18_13TeV.00360026.physics_EnhancedBias.merge.RAW._lb0151._SFO-1._0001.1 --file=/cvmfs/atlas-nightlies.cern.ch/repo/data/data-art/TrigP1Test/data18_13TeV.00360026.physics_EnhancedBias.merge.RAW._lb0151._SFO-1._0002.1 AthExHelloWorld/HelloWorldOptions.py >athenaHLT.log 2>&1 &`

Example result for threads=2 and nprocs=2:

2022-Jan-10 19:35:59,291 LOG [HLTMPPU::forkChildren(...) at /workspaces/tdaq/HLTMPPU/src/HLTMPPU.cxx:1022] Forked child "athenaHLT-46083-01" with pid= 46159
2022-Jan-10 19:35:59,292 INFO [HLTMPPU::forkChildren(...) at /workspaces/tdaq/HLTMPPU/src/HLTMPPU.cxx:1051] Set process affinity for child # 1 to cores: 0, 1
(...)
2022-Jan-10 19:35:59,302 LOG [HLTMPPU::forkChildren(...) at /workspaces/tdaq/HLTMPPU/src/HLTMPPU.cxx:1022] Forked child "athenaHLT-46083-02" with pid= 46161
2022-Jan-10 19:35:59,302 INFO [HLTMPPU::forkChildren(...) at /workspaces/tdaq/HLTMPPU/src/HLTMPPU.cxx:1051] Set process affinity for child # 2 to cores: 2, 3

Note that affinity is inherited by any thread started by the process, so all downstream threads of a given fork (event loop, scheduler, TBB workers) will have the same setting. Verified this by checking with taskset on LWP (thread ID) while the job was running.

AFAIK no privileges are required for a process to set its own affinity, so this works without any sudo rules.

cc @fwinkl, @wiedenma, @ssottoco
Jira: ATR-24548

Edited by Frank Winklmeier

Merge request reports