New check to avoid MR where DSID is modified

The root cause seems to be this bug which was fixed: !3468 (merged)

the following can happen

  1. we produce a sample like mc.123456.physics1.py
  2. someone makes a request to change 123456 -> 456789
  3. the system considers that 123456 is free so when a new MR is made there is a chance that a new sample will be assigned that DSID like mc.123456.physics2.py
  4. ProdSys doesn't check if a DSID is previously produced so we end up with mc.123456.physics1.py mc.123456.physics2.py

To avoid this we would need one of these options:

  1. keep a list of all DSIDs that have ever been assigned. If something gets obsoleted we do not remove it from the list. I don't think this is going to work well (or at least it's not that trivial to implement) because now we look at the DSIDs from all open remote branches, and I think it is perfectly valid to delete a branch (because we no longer want to merge it) and free up the DSIDs for other use. (least preferred option since it goes against the whole design)
  2. before the assignment of DSIDs we do a rucio search to see if samples with these DSIDs already exist (this is fairly easy to implement in the CI, the downside is it will consume a lot of cpu resources for edge cases, since this happens with a rate of <0.1 %
  3. this type of MR f3debef9 should already be avoided by the notify CI job. So in principle we don't need to implement anything new.
  4. given that the approval of the jO is more focused on getting the pipeline green, I wonder if we should remove approval rights to everyone but the PMG conveners

My preferred solutions: 3+4

See ATLMCPROD-11647

Another instance is here: !3151 (merged)

Edited by Spyros Argyropoulos