[surface model] fix some kTolerance usages for mixed precision

This small MR fixes a few usages of the old kTolerance and introduces the usage of the precision based kTolerance.

Note that there are more compile-time kTolerances used, however, I propose to only change them gradually when investigating failing rays and not all at once, since when fully compiling with single precision, many traverse errors were observed.