warning: output: Repeated slices with same orbit number (not strictly increasing)

The below warning infrequently pops up in bursts during CALO runs, which causes new data to be potentially skipped:

"Apr 25 16:55:27 scoutrubu-c2e37-29-01 runSCdaq.sh[400576]: OutputByOrbitStream::OutputFixedOrbits: First orbit number in slice is not strictly increasing: received 13220014, but previous seen slice had 13220014; not writing this slice."

Version: https://gitlab.cern.ch/scouting-demonstrator/scdaq/-/tags/v2.1.1

Status: investigate

Offending code:

OutputFixedOrbits(Slice&), in OutputByOrbit.cc:32

if (!dev_looping_orbits_ &&
static_cast<long long int>(first_orbit_number_in_slice) <= last_seen_orbit_) {
// Orbit processing is strictly increasing; slices with out of order orbits must be dropped.
// The dev_looping_orbits_ boolean exempts cases where the input filter loops over a file,
// since this means the orbit number can jump backward during normal execution.
// TODO: Input filter is a better fit for source data validation.
// TODO: The below solution does not (and cannot) fully validate slices with multiple orbits!
// NOTE: Consider writing faulty slices to an error file if DEBUG level is selected
LOG(ERROR) << "First orbit number in slice is not strictly increasing: received "
<< first_orbit_number_in_slice << ", but previous seen slice had "
<< last_seen_orbit_ << "; not writing this slice.";
out.reset();
}

Possible causes:

• orbit repetition from source board
• SCDAQ mislabels the slice due to bug / early exit.
• SCDAQ does not progress to the next slice pointer due to bug (as each slice contains a single orbit)