Skip to content

NXCALS-3871 - Simple buffering in Ingestion

Guys,

this is a replacement for !998 (closed) without a single change :) It's done however under a different task to implement a simple buffering and nothing else. It is meant to address use cases where data is sent in batches and there might be a single logical partition in the batch that is different than the currently active one. Without buffering the first data point that gets to the service will close the entity's history window with its stamp t1 resulting in all records that belong to the same batch and have t < t1 being rejected.
This model is common for datasources like WinnCC OA and REMUS - they have a constant schema and partition is taken from metadata which remains unchanged within the batch of data; it may change though between batches. Since batches are sent sequentially such buffering (being done long enough) solves the problem.

This implementation goes a bit further and paves the way for the more sophisticated buffering where the decision about data points being flushed from the buffer will depend on the history of changes of their logical partition with respect to some predefined time.

@acc-logging-team - please have a look for approval.

Closes NXCALS-3871

Merge request reports