VecMem Based Examples, main branch (2024.01.10.)
This is a replacement for !67932 (closed), now without the changes that got merged in with !67924 (merged).
These are some examples of how to make use of vecmem in Athena.
- In AthExCUDA I wrote some code that demonstrates (in a for now fairly awkward way) how data from an xAOD container could be transferred back and forth with a VecMem SoA container;
- This would definitely be done through some helper code later on, this is only a very first example. @ssnyder had some good suggestions on how this sort of copying could be done between an auxiliary container and a VecMem container.
- This is mainly meant for @fwinkl and @dossantn at this point. (Though @bwynne should also ping the FastTrackFinder developers here, as I'll also want to get them on board with this...)
- In AthExSYCL I updated the build setup to make use of the CMake code developed for the Acts R&D projects for building SYCL source files.
- Theis part of mainly for @tbold, and his group's developments. Note that with the updated recipe that I put into that package's README, I was able to run this MR's code successfully on top of our current
x86_64-el9-gcc13-optnightly.😄
- Theis part of mainly for @tbold, and his group's developments. Note that with the updated recipe that I put into that package's README, I was able to run this MR's code successfully on top of our current
To be even a bit more explicit here, since the oneAPI setup is a bit fragile with our current environment setup, you can tell CMake to build "Intel binaries" with oneAPI 2024.0 like the following:
setupATLAS
asetup Athena,main,latest
source /cvmfs/projects.cern.ch/intelsw/oneAPI/linux/24-all-setup.sh
export SYCLCXX="`which icpx` -fsycl --gcc-toolchain=/cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9"This results in output like the following (if you have the right sort of hardware...
[bash][pcadp04]:build-workdir > ./CMakeFiles/atlas_build_run.sh athena.py --stdcmalloc AthExSYCL/LinearTransformExample_jobOptions.py
Tue Dec 19 14:50:57 CET 2023
Py:Athena INFO using release [WorkDir-24.0.20] [x86_64-el9-gcc13-opt] [VecMemExamples-main-20231219/2e45db7f0d4] -- built on [2023-12-19T1444]
Py:Athena INFO including file "AthenaCommon/Preparation.py"
Py:Athena INFO including file "AthenaCommon/Atlas.UnixStandardJob.py"
Py:Athena INFO executing ROOT6Setup
Py:Athena INFO including file "AthenaCommon/Execution.py"
Py:Athena INFO including file "AthExSYCL/LinearTransformExample_jobOptions.py"
Py:Athena INFO including file "AthenaCommon/runbatch.py"
Py:ConfigurableDb INFO Read module info for 5257 configurables from 4 genConfDb files
Py:ConfigurableDb INFO No duplicates have been found: that's good !
ApplicationMgr SUCCESS
====================================================================================================================================
Welcome to ApplicationMgr (GaudiCoreSvc v37r2)
running on pcadp04 on Tue Dec 19 14:53:56 2023
====================================================================================================================================
ApplicationMgr INFO Application Manager Configured successfully
AthDictLoaderSvc INFO in initialize...
AthDictLoaderSvc INFO acquired Dso-registry
ClassIDSvc INFO getRegistryEntries: read 3092 CLIDRegistry entries for module ALL
CoreDumpSvc INFO Handling signals: 11(Segmentation fault) 7(Bus error) 4(Illegal instruction) 8(Floating point exception) 14(Alarm clock)
AthenaEventLoopMgr INFO Initializing AthenaEventLoopMgr
AthSYCL::Linear... DEBUG Property update for OutputLevel : new value = 2
AthSYCL::Linear... DEBUG input handles: 0
AthSYCL::Linear... DEBUG output handles: 0
ApplicationMgr INFO Application Manager Initialized successfully
ApplicationMgr INFO Application Manager Started successfully
AthenaEventLoopMgr INFO ===>>> start of run 1 <<<===
AthenaEventLoopMgr INFO ===>>> start processing event #1, run #1 0 events processed so far <<<===
AthSYCL::Linear... DEBUG Using device Intel(R) Data Center GPU Flex 170 (1.3)
AthenaEventLoopMgr INFO ===>>> done processing event #1, run #1 1 events processed so far <<<===
AthenaEventLoopMgr INFO ===>>> start processing event #2, run #1 1 events processed so far <<<===
AthSYCL::Linear... DEBUG Using device Intel(R) Data Center GPU Flex 170 (1.3)
AthenaEventLoopMgr INFO ===>>> done processing event #2, run #1 2 events processed so far <<<===
...The perhaps more interesting part is that you can do the following to build "NVIDIA binaries" on top of x86_64-el9-gcc13-opt at the moment:
setupATLAS
asetup Athena,main,latest
source /cvmfs/projects.cern.ch/intelsw/oneAPI/linux/24-all-setup.sh --include-intel-llvm
export SYCLCXX="`which clang++` -fsycl -fsycl-targets=nvidia_gpu_sm_75 --gcc-toolchain=/cvmfs/sft.cern.ch/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9 --cuda-path=/cvmfs/projects.cern.ch/lcg/releases/cuda/12.3/x86_64-linux"Resulting in output like:
[bash][atspot01]:build > ./CMakeFiles/atlas_build_run.sh athena.py AthExSYCL/LinearTransformExample_jobOptions.py
Wed Jan 10 15:14:13 CET 2024
Preloading tcmalloc_minimal.so
Py:Athena INFO using release [WorkDir-24.0.21] [x86_64-el9-gcc13-opt] [VecMemExamples-main-20240110/fb994517bb0] -- built on [2024-01-10T1513]
Py:Athena INFO including file "AthenaCommon/Preparation.py"
Py:Athena INFO including file "AthenaCommon/Atlas.UnixStandardJob.py"
Py:Athena INFO executing ROOT6Setup
Py:Athena INFO including file "AthenaCommon/Execution.py"
Py:Athena INFO including file "AthExSYCL/LinearTransformExample_jobOptions.py"
Py:Athena INFO including file "AthenaCommon/runbatch.py"
Py:ConfigurableDb INFO Read module info for 5226 configurables from 4 genConfDb files
Py:ConfigurableDb INFO No duplicates have been found: that's good !
# setting LC_ALL to "C"
ApplicationMgr SUCCESS
====================================================================================================================================
Welcome to ApplicationMgr (GaudiCoreSvc v37r2)
running on atspot01 on Wed Jan 10 15:18:01 2024
====================================================================================================================================
ApplicationMgr INFO Application Manager Configured successfully
AthDictLoaderSvc INFO in initialize...
AthDictLoaderSvc INFO acquired Dso-registry
ClassIDSvc INFO getRegistryEntries: read 3092 CLIDRegistry entries for module ALL
CoreDumpSvc INFO Handling signals: 11(Segmentation fault) 7(Bus error) 4(Illegal instruction) 8(Floating point exception) 14(Alarm clock)
AthenaEventLoopMgr INFO Initializing AthenaEventLoopMgr
AthSYCL::Linear... DEBUG Property update for OutputLevel : new value = 2
AthSYCL::Linear... DEBUG input handles: 0
AthSYCL::Linear... DEBUG output handles: 0
ApplicationMgr INFO Application Manager Initialized successfully
ApplicationMgr INFO Application Manager Started successfully
AthenaEventLoopMgr INFO ===>>> start of run 1 <<<===
AthenaEventLoopMgr INFO ===>>> start processing event #1, run #1 0 events processed so far <<<===
AthSYCL::Linear... DEBUG Using device NVIDIA GeForce RTX 2060 (7.5)
AthenaEventLoopMgr INFO ===>>> done processing event #1, run #1 1 events processed so far <<<===
AthenaEventLoopMgr INFO ===>>> start processing event #2, run #1 1 events processed so far <<<===
AthSYCL::Linear... DEBUG Using device NVIDIA GeForce RTX 2060 (7.5)
AthenaEventLoopMgr INFO ===>>> done processing event #2, run #1 2 events processed so far <<<===
...