WIP: Java imports tracing
Cmmnbuild_dep_manager Manager class injects jpype module to be able to track which java packages were used and report it to remote server. Implementation supports jpype versions 0.6.3 and >0.7. For the purpose of jpype imports testing, multiprocess tests execution was added using pytest-xdist library. Old tests were adjusted to work in distributed mode
Merge request reports
Activity
17 17 - conda install -c conda-forge --yes --quiet Jpype1${JPYPE_VN} openjdk 18 18 19 19 - pip install -e . 20 - pip install pytest-cov pytest 20 - pip install pytest-cov pytest pytest-xdist - cmmnbuild_dep_manager/tracing.py 0 → 100644
1 from cmw_tracing import ConfigTracer -
I don't really want to expose CERN tracing details into cmmnbuild-dep-manager, and would much prefer we used a common interface instead. For example, can we use the Python logging system, and simply register a CMW tracing handler as part of the Acc-Py distribution? Thereby removing any need for
cmmnbuild-dep-managerto know about CMW.
127 126 return_value=[mock_resolver]) 128 127 129 128 resolver = mock_resolver 130 with mock_resolvers, no_pre_installed_modules(): 129 with mock_resolvers: 131 130 manager = cbdm.Manager() 131 # Overwrites modules.json save paths so distributed tests can be performed 107 def _inject_jpype_for_import_reporting(self, jpype): 108 if jpype.__version__ == "0.6.3": 109 java_obj = jpype._jclass._JavaClass 110 name_extractor = complex_name_extractor 111 elif jpype.__version__ >= "0.7": 112 java_obj = jpype._jpype._JObject 113 name_extractor = simple_name_extractor 114 else: 115 self.log.info("JPype version {} not suitable for tracking injection".format(jpype.__version__)) 116 return 117 118 def tracking_wrapper(func): 119 def wrapped_getattr(__self, item): 120 try: 121 name = name_extractor(__self) 122 self._tracker(name) 113 name_extractor = simple_name_extractor 114 else: 115 self.log.info("JPype version {} not suitable for tracking injection".format(jpype.__version__)) 116 return 117 118 def tracking_wrapper(func): 119 def wrapped_getattr(__self, item): 120 try: 121 name = name_extractor(__self) 122 self._tracker(name) 123 except Exception as e: 124 self.log.warning("Tracker logging raised exception: {}".format(e)) 125 return func(__self, item) 126 return wrapped_getattr 127 128 java_obj.__getattribute__ = tracking_wrapper(java_obj.__getattribute__) -
So this one is a bit too far-reaching and intrusive for my liking - I think we need to find a way that makes the tracing sympathetic to those who don't want it enabled, and to allow it to continue looking like an open-source-able codebase instead of a corporately-controlled library (e.g. open-source libraries can add tracing, but don't hard-code the tracing destination such that different users can have different tracing). In terms of the implementation, I also think there is a little bit too much connectivity to be something that we would adopt in
cmmnbuild-dep-manageras it stands.However, in terms of what you've achieved here, I think it is a really nice start and (like your other MR) has most of the bits that we can pick up and use.
Let's get together and make sure that we are aligned on the direction, and then iterate some more on this one before merging.