Move to VTune 2020 and associated Problems
Thanks to @maszyman the nightlies will now run a fixed version(2019) of VTune.
The automatic change to the 2020 version broke the tests because the attaching to a job causes the vtune process to segfault and leave the Moore job just paused.
I manually reproduced this in a dual terminal setup.
Terminal 1 was running the job and upon attaching prints this to stdout
AMPLXE_TPSSCOLLECTOR: pytrace_tracewriter:341: (jitWriterReady) :
Assertion failed: pytrace_tracewriter:341: (jitWriterReady) : . Please contact the technical support.
*** Break *** segmentation violation
While terminal 2 from where I attached vtune prints this:
vtune: Warning: Function 'PyEval_EvalFrameEx' can be analyzed incorrectly because it uses indirect branch instructions.
vtune: Error: Assertion failed: pytrace_tracewriter:341: (jitWriterReady) : . Please contact the technical support.
vtune: Collection started. To stop the collection, either press CTRL-C or enter from another console window: vtune -r /home/chasse/lhcb-benchmark-scripts/profile_out -command stop.
Thanks to suggestions from @ahennequ I also investigated simply starting vtune directly with the job together using -start-paused -resume-after 60 -d 120
which pauses vtune in the beginning, starts it after 60 seconds and then stops after an additional 60 seconds (-d 120 is stop after 120s from the launch).
I did observe a throughput reduction O(30%) when running vtune like this. Is this a problem for the accuracy of the actual measurement?? I don't know, I suggest you ask your favorite priest...
I'm attaching the flamegraph from this run.
Note that for some reason there are many more stacks and more detail (good! classname::operator()
which makes parsing for the FlameBars
a pain.
So for now I suggest we continue with the 2019 version that works and requires no further intervention right now.
How do we proceed in the future? Who knows...
I still think if we want to get more serious about these kinds of measurements we need a stack build with framepointers