Skip to content

Add CUDA support for AvalancheMicroscopic

Tom Neep requested to merge tneep/garfieldpp:cuda into master

Hi @hschindl,

Here is the long promised merge request to add CUDA support for AvalancheMicroscopic. Sorry it has taken so long to get to this stage, our updates ended up quite far behind the master and it took a big effort to catch up. I think it is in a place where it would be good to start the discussion/review now.

As you can see, this is a pretty huge MR so here is a bit of a breakdown of where the changes are.

❯  git diff --dirstat master | sort -nr
  51.8% GPU/
  25.2% Source/
  17.8% Include/Garfield/
   3.9% Examples/GemGPU/

Over 50% of the changes are new additions in the GPU directory (and an example). This is new code that won't affect anything else. I'm more worried about the changes to Source and Include/Garfield since we don't want to break any existing functionality.

❯ git diff --numstat master -- Source/ Include/Garfield/ | sort -nr
256	24	Source/ComponentFieldMap.cc
255	59	Source/AvalancheMicroscopic.cc
159	13	Include/Garfield/ComponentFieldMap.hh
138	14	Source/MediumMagboltz.cc
97	14	Include/Garfield/Medium.hh
92	0	Include/Garfield/AvalancheMicroscopic.hh
88	28	Include/Garfield/TetrahedralTree.hh
87	8	Include/Garfield/MediumMagboltz.hh
86	8	Include/Garfield/Component.hh
63	6	Include/Garfield/Sensor.hh
47	6	Source/TetrahedralTree.cc
34	6	Source/Sensor.cc
18	1	Include/Garfield/MediumGas.hh
15	0	Include/Garfield/MultiProcessInterface.hh
12	1	Include/Garfield/ComponentAnsys123.hh
8	0	Source/ComponentAnsys123.cc
3	0	Source/MediumGas.cc
3	0	Source/Medium.cc
3	0	Source/Component.cc
3	0	Include/Garfield/RandomEngineRoot.hh
3	0	Include/Garfield/RandomEngine.hh

Let me know how you'd like to proceed. We can maybe find some areas where we can break things up into smaller chunks.

I'm also planning on investigating where we can use the CUDA C++ standard library a bit more. This looks to have come along a fair way since we started and might allow us to remove some of the changes that were needed. It depends if we want to set a minimum version of CUDA to use. We've been testing on 11.5 but it looks like there might be newer features that would clean up some parts of the code. That can come later.

I should probably say here that the GPU code should probably be considered "experimental" and is certainly not feature complete. However, It would be great to gather some feedback of what features people need.

Testing

  1. Login to a GPU enabled node (e.g. lxplus-gpu)
  2. Setup Garfield++ as normal but add the -DUSEGPU=ON option when doing cmake
  3. See Examples/GemGPU/ for an example, run with ./gem_gpu --gpu (for gpu, and without the flag for cpu)
  4. The example might run slower on GPU than CPU depending on the GPU specs. On the Tesla T4 available on lxplus-gpu speeds might be similar, but gains are clear with newer GPUs or a larger initial number of electrons.

Merge request reports