Do not run by default the Yandex MVAs
Currently by default the Yandex PID tunes are run by default in the ANNPID tuple tool. This was added in a recent update. Unfortunately at the time I did not personnally check the runtime performance.
Running some tests just now it seems these new MVAs are very slow to initialise, and use a lot of memory. In my tests, running a standard davinci job, they increased the resident size of my job by almost 1GB and added about 30secs to the initialisation time of the application. As such it is inappropriate for them to be run by default for all users. Users who are interested in running them can manually request them, by adding them to the tunes used. This update turns off their running by default to save CPU and memory for those who do not need them, and do not know they need to turn them off.
As an aside, the current cpu and memory usage of these MVAs will certainly I believe prevent any use of them in production, where the increase in memory usage is simply not acceptable. If any future use of these is to be made, in either the HLT or offline stripping, the memory issue will need to be addressed.
Merge request reports
Activity
- [2017-07-25 22:10] Validation started with lhcb-gaudi-merge#174
- [2017-07-26 00:03] Validation started with lhcb-old-stripping-tests#476
- [2017-07-26 00:08] Validation started with lhcb-gaudi-head-noavx#81
- [2017-07-26 00:08] Validation started with lhcb-head#1547
- [2017-07-26 00:08] Validation started with lhcb-2017-patches#90
- [2017-07-26 00:09] Validation started with lhcb-clang-test#654
- [2017-07-26 00:09] Validation started with lhcb-gaudi-head-noavx2#62
- [2017-07-26 00:09] Validation started with lhcb-lcg-dev3#231
- [2017-07-26 00:10] Validation started with lhcb-lcg-dev4#231
- [2017-07-26 00:10] Validation started with lhcb-gaudi-head#1544
- [2017-07-26 15:53] Validation started with lhcb-gaudi-merge#175
- [2017-07-27 00:03] Validation started with lhcb-old-stripping-tests#477
- [2017-07-27 00:09] Validation started with lhcb-gaudi-head-noavx#82
- [2017-07-27 00:09] Validation started with lhcb-gaudi-head-noavx2#63
- [2017-07-27 00:09] Validation started with lhcb-2017-patches#91
- [2017-07-27 00:09] Validation started with lhcb-head#1548
- [2017-07-27 00:09] Validation started with lhcb-gaudi-head#1545
- [2017-07-27 00:09] Validation started with lhcb-clang-test#655
- [2017-07-27 00:10] Validation started with lhcb-lcg-dev3#232
- [2017-07-27 00:10] Validation started with lhcb-lcg-dev4#232
- [2017-07-27 10:03] Validation started with lhcb-clang-test#656
- [2017-07-28 00:03] Validation started with lhcb-old-stripping-tests#478
- [2017-07-28 00:08] Validation started with lhcb-head#1549
- [2017-07-28 00:08] Validation started with lhcb-gaudi-head-noavx#83
- [2017-07-28 00:08] Validation started with lhcb-lcg-dev4#233
- [2017-07-28 00:08] Validation started with lhcb-clang-test#657
- [2017-07-28 00:09] Validation started with lhcb-gaudi-head-noavx2#64
- [2017-07-28 00:09] Validation started with lhcb-lcg-dev3#233
- [2017-07-28 00:09] Validation started with lhcb-gaudi-head#1546
- [2017-07-28 00:09] Validation started with lhcb-2017-patches#92
- [2017-07-28 08:08] Validation started with lhcb-gaudi-merge#176
- [2017-07-28 08:09] Validation started with lhcb-gaudi-merge#177
- [2017-07-29 00:03] Validation started with lhcb-old-stripping-tests#479
- [2017-07-29 00:08] Validation started with lhcb-head#1550
- [2017-07-29 00:08] Validation started with lhcb-gaudi-head-noavx#84
- [2017-07-29 00:08] Validation started with lhcb-gaudi-head-noavx2#65
- [2017-07-29 00:08] Validation started with lhcb-lcg-dev4#234
- [2017-07-29 00:08] Validation started with lhcb-clang-test#658
- [2017-07-29 00:08] Validation started with lhcb-lcg-dev3#234
- [2017-07-29 00:09] Validation started with lhcb-2017-patches#93
- [2017-07-29 00:10] Validation started with lhcb-gaudi-head#1547
- [2017-07-29 10:32] Validation started with lhcb-clang-test#658
- [2017-07-30 00:02] Validation started with lhcb-old-stripping-tests#480
- [2017-07-30 00:08] Validation started with lhcb-gaudi-head-noavx2#66
- [2017-07-30 00:08] Validation started with lhcb-2017-patches#94
- [2017-07-30 00:08] Validation started with lhcb-clang-test#659
- [2017-07-30 00:08] Validation started with lhcb-gaudi-head-noavx#85
- [2017-07-30 00:08] Validation started with lhcb-head#1551
- [2017-07-30 00:09] Validation started with lhcb-lcg-dev4#235
- [2017-07-30 00:10] Validation started with lhcb-gaudi-head#1548
- [2017-07-30 00:10] Validation started with lhcb-lcg-dev3#235
- [2017-07-30 08:09] Validation started with lhcb-gaudi-merge#178
- [2017-07-30 11:27] Validation started with lhcb-gaudi-merge#179
- [2017-07-30 15:32] Validation started with lhcb-gaudi-merge#180
- [2017-07-31 00:03] Validation started with lhcb-old-stripping-tests#481
- [2017-07-31 00:07] Validation started with lhcb-gaudi-head-noavx#86
- [2017-07-31 00:07] Validation started with lhcb-lcg-dev3#236
- [2017-07-31 00:07] Validation started with lhcb-gaudi-head-noavx2#67
- [2017-07-31 00:07] Validation started with lhcb-head#1552
- [2017-07-31 00:07] Validation started with lhcb-clang-test#660
- [2017-07-31 00:08] Validation started with lhcb-lcg-dev4#236
- [2017-07-31 00:09] Validation started with lhcb-2017-patches#95
- [2017-07-31 00:10] Validation started with lhcb-gaudi-head#1549
- [2017-07-31 10:54] Validation started with lhcb-gaudi-head#1550
Edited by Software for LHCb@jonrob Thanks for the notification. The task will be open again to address this issue. Will this go in a next DV release? The PIDCalib options where the tool is included have been merged yesterday in a new Erasmus version based on DV v42r5. That should be released by today, and the production will start tomorrow so in principle I cannot change anything.
@jonrob Thanks for letting us know.
We do have ideas of speeding up the code and reducing the memory consumption (connected, for example, to the comment about vector of predictions: Rec!602 (comment 256288) ) , however, this was not in the scope of the first implementation. Can you (or anyone) give the target memory consumption and running time (for offline and online)? Also, how many events did you run to receive 30 seconds delay?
@mfontana sorry, I am a bit confused. Which data samples will have these yPID variables?
@derkach The 2015-2016 PID calibration samples
@derkach I think your target should be to be similar in memory and execution time to the current TMVA based MVAs. Any more than this will be an issue.
Note the CPU as I said was not the per event CPU, but the time to initialise the tools. I did not get as far as timing the per event time, but again, your target if you want these MVAs to be able to be used in production is to be no worse than the current TMVA based MVAs.
mentioned in commit f314450a