Review of the seed functionality
Context
The seed for the pseudo-random number generator is one of the arguments supplied by a user in each simulation. User-defined value is used not as a seed for the entire simulation campaign started from the file but as the first seed in the list of as many iterations as are defined by the "number of simulations" value. In other words: when "number of simulations" value is set to 3 and "seed" to 0, we get one iteration using the generator initialised with 0, another one with 1 and the last one with 2.
The situation gets more complicated in the distributed setting: the seeds are used to decide on the number of jobs created on HTCondor. Users are required to provide as many seeds as there will jobs - and they are required to have enough numbers in between them so that each generators is initialised with a different number. Recently, there has been the issue #161 (closed) to add a warning preventing users from accidentally running same sets of seeds.
However, it still seems unnecessarily convoluted. Let's try to fix it!
Problems
- Danger of letting users inadvertently run the exact same simulations by specifying SEED in sensitivity analysis to a list of, e.g., consecutive numbers.
- Furthermore, we need to review the current take on using same seeds in simulations with different parameters. This could potentially trigger a review of how parameters are combined in the sensitivity analysis file (potential for another issue).
- Setting the number of jobs using the size of the seeds field in the sensitivity analysis is not optimal, especially given that we want to make "None" value the default way to run things.
- Check in the
sensitivity_analysis.py
is implementing checks on seeds intervals. In case of any changes, it will need to be further accommodated. If it is to stay in the same form, we should consider @dlizan suggestion to either add intervaltree or implement the required functionality subset ourselves.
At the very least, we need to expand warnings in this regard in the user guide and in the instructions.