Open requested to merge fbury/inference:exclusion_interpolation into master
The problem I faced recently was when running 2D exclusion limits with a very large number of points
kl,-25,25,51:kt,-5,5,51 which prompted the following memory error
[...] File "/afs/cern.ch/user/f/fbury/work/inference/dhi/plots/exclusion.py", line 477, in plot_exclusion_and_bestfit_2d interpolation=interpolation_method, File "/afs/cern.ch/user/f/fbury/work/inference/dhi/plots/util.py", line 413, in get_contours fill_hist_from_points(h, x_values, y_values, z_values, **kwargs) File "/afs/cern.ch/user/f/fbury/work/inference/dhi/plots/util.py", line 296, in fill_hist_from_points interp = scipy.interpolate.Rbf(x_values, y_values, z_values, **rbf_args) File "/cvmfs/cms.cern.ch/slc7_amd64_gcc700/external/py2-scipy/1.2.1-pafccj/lib/python2.7/site-packages/scipy/interpolate/rbf.py", line 241, in __init__ self.nodes = linalg.solve(self.A, self.di) File "/cvmfs/cms.cern.ch/slc7_amd64_gcc700/external/py2-scipy/1.2.1-pafccj/lib/python2.7/site-packages/scipy/interpolate/rbf.py", line 247, in A r = squareform(pdist(self.xi.T, self.norm)) # Pairwise norm File "/cvmfs/cms.cern.ch/slc7_amd64_gcc700/external/py2-scipy/1.2.1-pafccj/lib/python2.7/site-packages/scipy/spatial/distance.py", line 2159, in squareform M = np.zeros((d, d), dtype=X.dtype) MemoryError
Turns out the lists
x_values, etc, had a length of
50601, squared in the
rbf method and my script exceeded 10 GB of RAM then was killed on lxplus.
I do not have a fix for the
rbf choice (maybe a smart dropping of some values in the array ?), but the
TGraph2D option (=
root argument) works well enough and in a few seconds. On the other hand, the scipy
linear option takes a much longer time and provides similar result.
Therefore, I think it makes sense that the user can select the interpolation method, and I copied the argument that as in the