Skip to content

Small fix to make the code compatible with new Spark3 version

Sergio Blanco Fernandez requested to merge (removed):master into master

Recently, the version of Spark in the analytix cluster has been updated from Spark2 to Spark3 (https://cern.service-now.com/service-portal?id=outage&n=OTG0076547). Here, it's proposed a small fix to the code to be able to run with the new version.

In principle, the new Spark3 version does not affect directly the code. However, we need to load a newer LCG machine in env.sh to be consistent with this new version of Spark. Since the LCG release that was loaded previously was quite old, some libraries have been also updated to more recent versions.

In our case, we are mostly affected by the upgrade of uproot from 3.X to 4.X. This update is not a minor one, since there are some definitions that have changed and uproot_methods (used to load histograms from numpy arrays in our code) is not yet available for uproot4. Furthermore, uproot_methods is not compatible with uproot4.

First, this fix modifies the env.sh file to use a newer LCG machine. Second, in cases where uproot_methods is used, we are now using uproot3 and uproot3_methods. The parts of the code that don't need uproot_methods are using uproot4, with an updated syntax.

Merge request reports