Commit 833df00a authored by Remi Mommsen's avatar Remi Mommsen Committed by Dainius Simelevicius
Browse files

references #180: add instructions on how to measure FEDbuilders

parent be8b4f94
......@@ -318,7 +318,7 @@ and stop the launchers separately:
The simple scans described above used an identical fragment size for
all FEDs. It is also possible to run a configuration using
parameterized FED sizes. The FED sizes are parametrized with a
polynom 2nd order a+bx+cx^2+dx^3. The parameters need to be specified
polynom 3rd order a+bx+cx^2+dx^3. The parameters need to be specified
in a csv file which lists for each FED id the following parameters:
fedId,a,b,c,d,rms
The rms for the log-normal distribution is specified relative to the
......@@ -334,9 +334,12 @@ absolute values. The full command to a full scan looks like:
-o $outDir --logDir $logDir \
--relsize 1.0 1.25 1.5 2.0 2.5 0 0.25 0.5 0.75 3.0 3.5 4.0
The FED-size parameter files for pp and HI used for the initial
estimate for run 3 can be found at
https://gitlab.cern.ch/cms-daq/measurements/evb/daq3val/-/tree/master/configs/run3FB
There are a couple more options which might be helpful to understand
problems:
There are a couple more options which might be helpful:
--numa disables any NUMA settings.
In line 185ff of test/Configuration.py you find some commented
......@@ -350,6 +353,9 @@ problems:
--maxTriggerRate allows to limited the trigger rate in Hz. This is
done by throttling the rate on the EVM by sleeping in the
event-assignment thread.
--waitForStateTransitions requires a key stroke before each state
transition is initiated. This is helpful to investigate problems
during state transitions.
......@@ -383,6 +389,141 @@ takes one or more dat files as argument. A typical command looks like
--totalThroughput --minx 620 --maxx 2580 --maxy 320 --ratemax 280 --nologx
FEDbuilder Performance
----------------------
The FEDbuilders in the production system do not use the simplistic
approach of a fixed number of FEROLs per RU. The fragment sizes for
different FEDs/subsystems are vastly different. Therefore, the
FEDbuilders contain different number of FEDs. In addition, FEDbuilders
only combine FEDs belonging to the same subsystem. The design of the
FEDbuilders needs to take into account the throughput capability of
the RU, too. This depends on the number of streams and the FED
sizes. Typically, the RU can handle a higher rate of small FED sizes
from more streams and less streams from FEDs with large fragments.
There are two methods to assess the throughput of the FEDbuilders:
estimate the throughput based on the FBset created by the RCMS
configurator in the HWCfg DB, and to measure the actual performance
using the test bed, e.g. daq3val.
Throughput based on the FBset:
(1) Use the DAQ configurator to display the FBset structure.
(2) Copy the text into a text file. An example of the run 2 FBset
definition can be found at
https://gitlab.cern.ch/cms-daq/measurements/evb/daq3val/-/blob/master/configs/run3FB/run2FB.txt
(3) Use the FED size parameter file to calculate the expected
throughput:
checkFBthroughput.py --plain run2FB.txt fedSizes-run3-pp.csv 1
checkFBthroughput.py contains different models for the maximum
throughput for different number of FEROL streams. These parameters can
be obtained by using the '--fit' option in plotScans.py. Note that
there are two fit models used: a linear and quadratic plateau model as
defined in FitModels.py. You need to decide case by case which of the
models yield a more accurate description. In any case, these can
only give an approximate estimate if the FED builder would be able to
cope with the FED sizes.
checkFBthroughput.py has additional parameters:
--triggerRate sets the expected L1 trigger rate. The default is
100 kHz, which needs to be lowered in case of heavy-ion FED sizes.
--printAll shows the throughput of all FEDbuilders, not only of
those which have a too high throughput
--plain is used to read the flat text file. Without this option,
the input must be a full DAQ configuration in XML format. (This
option has not been tested with configurations with a folded RUBU.)
Actual performance measurement in daq3val:
There are not enough FEROLs available in the testbed to measure all
FED builders simultaneously. Therefore, the idea is to create separate
configurations for each FEDbuilder, which contains only the EVM on a
separate host and the RUBU handling the given FEDbuilder. The aim is
to measure the achievable throughput and L1 trigger rate for the
expected fragment sizes. The performance of the RUBU node is
influenced by the BU: a single BU in the system has to digest a much
higher event rate than in standard running, where the rate is
distributed over several BUs. However, having multiple BUs in the
system causes the BU on the RUBU node to request less events than its
less loaded neighbors. Therefore, it is advisable to measure the
performance with one and two BUs. The truth lies most likely somewhere
between these two settings. The results of the test run in July 2020
can be found at https://twiki.cern.ch/twiki/bin/view/CMS/DAQ3fedbuilders
(1) Manually create two XML templates of the full system with the
maximum number of FEDs foreseen in a FED builder, once with one BU on
the RUBU node and once with an additional BU on the EVM node. Note
that the FED ids are replaced by FEDID01..FEDID55.
An example (not containing the latest settings from the SWT) can be found at
https://gitlab.cern.ch/cms-daq/measurements/evb/daq3val/-/tree/master/configs/run3FB
(2) Edit makeFBconfigs.sh in above URL to define the fbDef file,
which contains the plain text FB set definitions, and the confDir,
which is used to store the output.
(3) Run makeFBconfigs.sh, which will create the XML configurations for
all FEDbuilders.
(4) Perform the measurements by editing and executing doFBscans.sh in
the daq3val directory. Note that the it takes several days to run the
measurements for all FEDbuilders. Therefore, it might be useful to
grep for certain FEDbuilder names (c.f. the code in doFBscans.sh)
(5) The plots shown on the twiki page are generated using
https://gitlab.cern.ch/cms-daq/measurements/evb/daq3val/-/blob/master/results/FEDbuilders/plots.sh
Setting FED sizes with RCMS
---------------------------
The DAQ FM can configure individual FED sizes, too. This is useful to
assess the full DAQ performance in emulator mode. However, it is not
possible to run scans with different FED sizes in an automated
way.
To run with individual FED sizes, you need to set the exported
global variable GET_FED_FRAGMENT_SIZES_FROM_FILE to 'true' in the
.cmsOneStep configurator. In this case, the DAQ FM sets the FED sizes
according to the file /nfshome0/daqoncall/fedFragmentSizes.csv.
In September 2020, this is a symlink to
fedFragmentSizes/fedFragmentSizes_HI2018.csv, which has the best
estimate of the HI fragment sizes from 2018. You need to re-configure
DAQ whenever you changed the symlink to another file to pick up the
changes.
The format of the file read by the DAQ FM is different from the
parameter file used by the EvB scripts which contains the parameters
of a 3rd-degree polynominal (c.f. above). The file read by the DAQ FM
has the format
fedId,meanSize,rmsSize
where the mean size is in Bytes and the rms size gives the width of
the log-normal distribution in Bytes. There might be further columns,
which are ignored by the DAQ FM.
There is a script test/scripts/createFedSizeFile.py in the EvB
repository which allows to create the FED-size file for the DAQ FM
from the parameter file used for the EvB measurements:
createFedSizeFile.py paramFile relEventSize outputFile
paramFile is the file which contains the fit parameters.
It corresponds to the parameter 'calculateFedSizesFromFile' of
runScans.py.
relEventSize specifies the relative FED sizes to use
It corresponds to one of the 'relsize' of runScans.py.
outputFile specifies the file name which is can be read by the DAQ FM.
EvB Benchmark
-------------
This benchmark uses the standard EvB code and protocols to measure
......@@ -415,7 +556,7 @@ To test a configuration for a single fragment, you can use
--logDir should point to a directory with enough quota
(e.g. /globalscratch/$USER) as it will contain the detailled
debug output of all XDAQ processes.
--nbMeasurements 0' is a backdoor to inhibit the scan. In this
--nbMeasurements 0 is a backdoor to inhibit the scan. In this
case, the script waits for the return being pressed before
proceeding. It is possible to specify multiple sizes in which case
the script reconfigures the system for the next fragment size and
......@@ -433,4 +574,4 @@ To test a configuration for a single fragment, you can use
are requested on the command line. RUBUs are treated as RUs.
To run a full scan, the same parameters can be used as for the
standard performance scan (see above.)
standard performance scan (see above).
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment