NXCALS-7436 Race condition in Spark scripts

Merge request template

COMPATIBILITY NOTE

Please make sure that this change is BACKWARDS COMPATIBLE and a new server will work correctly with clients using old APIs versions.

It may be that data sent in the new format will break the old clients.

In case of any potential incompatibility we have to inform our clients and schedule release & deployment with them.

---

@acc-logging-team - please have a look for approval.

Please describe the merge request here.

Closes NXCALS-7436