VecMem Based Examples, main branch (2024.01.10.) (!68103) · Merge requests · atlas / athena

Attila Krasznahorkay requested to merge akraszna/athena:VecMemExamples-main-20240110 into main Jan 10, 2024

This is a replacement for !67932 (closed), now without the changes that got merged in with !67924 (merged).

These are some examples of how to make use of vecmem in Athena.

In AthExCUDA I wrote some code that demonstrates (in a for now fairly awkward way) how data from an xAOD container could be transferred back and forth with a VecMem SoA container;
- This would definitely be done through some helper code later on, this is only a very first example. @ssnyder had some good suggestions on how this sort of copying could be done between an auxiliary container and a VecMem container.
- This is mainly meant for @fwinkl and @dossantn at this point. (Though @bwynne should also ping the FastTrackFinder developers here, as I'll also want to get them on board with this...)
In AthExSYCL I updated the build setup to make use of the CMake code developed for the Acts R&D projects for building SYCL source files.
- Theis part of mainly for @tbold, and his group's developments. Note that with the updated recipe that I put into that package's README, I was able to run this MR's code successfully on top of our current x86_64-el9-gcc13-opt nightly. 😄

To be even a bit more explicit here, since the oneAPI setup is a bit fragile with our current environment setup, you can tell CMake to build "Intel binaries" with oneAPI 2024.0 like the following:

setupATLAS
asetup Athena,main,latest
source /cvmfs/projects.cern.ch/intelsw/oneAPI/linux/24-all-setup.sh
export SYCLCXX="`which icpx` -fsycl --gcc-toolchain=/cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9"

This results in output like the following (if you have the right sort of hardware... 😛):

[bash][pcadp04]:build-workdir > ./CMakeFiles/atlas_build_run.sh athena.py --stdcmalloc AthExSYCL/LinearTransformExample_jobOptions.py
Tue Dec 19 14:50:57 CET 2023
Py:Athena            INFO using release [WorkDir-24.0.20] [x86_64-el9-gcc13-opt] [VecMemExamples-main-20231219/2e45db7f0d4] -- built on [2023-12-19T1444]
Py:Athena            INFO including file "AthenaCommon/Preparation.py"
Py:Athena            INFO including file "AthenaCommon/Atlas.UnixStandardJob.py"
Py:Athena            INFO executing ROOT6Setup
Py:Athena            INFO including file "AthenaCommon/Execution.py"
Py:Athena            INFO including file "AthExSYCL/LinearTransformExample_jobOptions.py"
Py:Athena            INFO including file "AthenaCommon/runbatch.py"
Py:ConfigurableDb    INFO Read module info for 5257 configurables from 4 genConfDb files
Py:ConfigurableDb    INFO No duplicates have been found: that's good !
ApplicationMgr    SUCCESS 
====================================================================================================================================
                                                   Welcome to ApplicationMgr (GaudiCoreSvc v37r2)
                                          running on pcadp04 on Tue Dec 19 14:53:56 2023
====================================================================================================================================
ApplicationMgr       INFO Application Manager Configured successfully
AthDictLoaderSvc     INFO in initialize...
AthDictLoaderSvc     INFO acquired Dso-registry
ClassIDSvc           INFO getRegistryEntries: read 3092 CLIDRegistry entries for module ALL
CoreDumpSvc          INFO Handling signals: 11(Segmentation fault) 7(Bus error) 4(Illegal instruction) 8(Floating point exception) 14(Alarm clock) 
AthenaEventLoopMgr   INFO Initializing AthenaEventLoopMgr
AthSYCL::Linear...  DEBUG Property update for OutputLevel : new value = 2
AthSYCL::Linear...  DEBUG input handles: 0
AthSYCL::Linear...  DEBUG output handles: 0
ApplicationMgr       INFO Application Manager Initialized successfully
ApplicationMgr       INFO Application Manager Started successfully
AthenaEventLoopMgr   INFO   ===>>>  start of run 1    <<<===
AthenaEventLoopMgr   INFO   ===>>>  start processing event #1, run #1 0 events processed so far  <<<===
AthSYCL::Linear...  DEBUG Using device Intel(R) Data Center GPU Flex 170 (1.3)
AthenaEventLoopMgr   INFO   ===>>>  done processing event #1, run #1 1 events processed so far  <<<===
AthenaEventLoopMgr   INFO   ===>>>  start processing event #2, run #1 1 events processed so far  <<<===
AthSYCL::Linear...  DEBUG Using device Intel(R) Data Center GPU Flex 170 (1.3)
AthenaEventLoopMgr   INFO   ===>>>  done processing event #2, run #1 2 events processed so far  <<<===
...

The perhaps more interesting part is that you can do the following to build "NVIDIA binaries" on top of x86_64-el9-gcc13-opt at the moment:

setupATLAS
asetup Athena,main,latest
source /cvmfs/projects.cern.ch/intelsw/oneAPI/linux/24-all-setup.sh --include-intel-llvm
export SYCLCXX="`which clang++` -fsycl -fsycl-targets=nvidia_gpu_sm_75 --gcc-toolchain=/cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9 --cuda-path=/cvmfs/projects.cern.ch/lcg/releases/cuda/12.3/x86_64-linux"

Resulting in output like:

[bash][atspot01]:build > ./CMakeFiles/atlas_build_run.sh athena.py AthExSYCL/LinearTransformExample_jobOptions.py
Wed Jan 10 15:14:13 CET 2024
Preloading tcmalloc_minimal.so
Py:Athena            INFO using release [WorkDir-24.0.21] [x86_64-el9-gcc13-opt] [VecMemExamples-main-20240110/fb994517bb0] -- built on [2024-01-10T1513]
Py:Athena            INFO including file "AthenaCommon/Preparation.py"
Py:Athena            INFO including file "AthenaCommon/Atlas.UnixStandardJob.py"
Py:Athena            INFO executing ROOT6Setup
Py:Athena            INFO including file "AthenaCommon/Execution.py"
Py:Athena            INFO including file "AthExSYCL/LinearTransformExample_jobOptions.py"
Py:Athena            INFO including file "AthenaCommon/runbatch.py"
Py:ConfigurableDb    INFO Read module info for 5226 configurables from 4 genConfDb files
Py:ConfigurableDb    INFO No duplicates have been found: that's good !
# setting LC_ALL to "C"
ApplicationMgr    SUCCESS 
====================================================================================================================================
                                                   Welcome to ApplicationMgr (GaudiCoreSvc v37r2)
                                          running on atspot01 on Wed Jan 10 15:18:01 2024
====================================================================================================================================
ApplicationMgr       INFO Application Manager Configured successfully
AthDictLoaderSvc     INFO in initialize...
AthDictLoaderSvc     INFO acquired Dso-registry
ClassIDSvc           INFO getRegistryEntries: read 3092 CLIDRegistry entries for module ALL
CoreDumpSvc          INFO Handling signals: 11(Segmentation fault) 7(Bus error) 4(Illegal instruction) 8(Floating point exception) 14(Alarm clock) 
AthenaEventLoopMgr   INFO Initializing AthenaEventLoopMgr
AthSYCL::Linear...  DEBUG Property update for OutputLevel : new value = 2
AthSYCL::Linear...  DEBUG input handles: 0
AthSYCL::Linear...  DEBUG output handles: 0
ApplicationMgr       INFO Application Manager Initialized successfully
ApplicationMgr       INFO Application Manager Started successfully
AthenaEventLoopMgr   INFO   ===>>>  start of run 1    <<<===
AthenaEventLoopMgr   INFO   ===>>>  start processing event #1, run #1 0 events processed so far  <<<===
AthSYCL::Linear...  DEBUG Using device NVIDIA GeForce RTX 2060 (7.5)
AthenaEventLoopMgr   INFO   ===>>>  done processing event #1, run #1 1 events processed so far  <<<===
AthenaEventLoopMgr   INFO   ===>>>  start processing event #2, run #1 1 events processed so far  <<<===
AthSYCL::Linear...  DEBUG Using device NVIDIA GeForce RTX 2060 (7.5)
AthenaEventLoopMgr   INFO   ===>>>  done processing event #2, run #1 2 events processed so far  <<<===
...

VecMem Based Examples, main branch (2024.01.10.)

Merge request reports