Add test that checks whether all known configurables can be instantiated

Following up on a discussion in Alignment!139 (merged), tests should be created in all projects to check whether all known configurables can be instantiated. @cattanem suggested an implementation that does not use a reference list, but instead just fails if any configurable fails to instantiate. That will make the maintenance of the test much easier.

added testing label

mentioned in merge request Alignment!139 (merged)

It turns out that the real obsolete code is in https://gitlab.cern.ch/lhcb/LHCb/-/blob/master/GaudiConf/python/GaudiConf/QMTest/ConfigurablesTest.py, which uses svnProjectDeps on line 104. IMHO this test should be drastically overhauled (basically to remove all the functionality that checks that all expected modules/configurables have been checked out and built, and concentrate on checking that they can all be instantiated). Once done, the per project configurables test can be resurrected. @rmatev @clemenci what do you think? Is it useful?

Probably it's useful to understand in what scenario the instantiation will fail.

0. The most frequent issue I recall is getting into this case when an upstream build failed. I'm not sure we need a test for this, a failed build is as good a signal as it gets.
1. I assume Gaudi and it's tests guarantees that C++ ("genconf") configurables can be instantiated and so we don't need to test these.
2. ConfigurableUsers may fail (I don't remember what happens when Gaudi collects these and what is effectively tested) but we're probably phasing these out anyway.

In general, any python we commit should probably be checked for syntax errors, but also even imported if it's in a python package directory (much like the test for including public headers). That doesn't guarantee the instantiation of something is going to work but it's probably going to cover a lot.

The main reason for this test was that when we were using SVN the checkout of a project required multiple round-trips to SVN (one per package), and the probability of a failure (non detected) in one of the requests was non negligible.

This test is meant to make sure that all packages were actually checked out, by checking that the matching component library was compiled.

As our test coverage is not so good, this test was very important to make sure we didn't get corrupted releases.

With the switch from SVN to Git, and in particular moving from many small projects to monolithic projects, we do not have to check that the checkout was complete.

OTOH, it's a good idea to check that ConfigurableUser specializations do not contain errors. Just importing the module should be enough to rule out syntax errors... and that's already done at compile time, so we do not need a test for it. Proper unit/integration tests would be better, but they cannot be automated.

I suggest we just drop this test.

@clemenci, @rmatev and I had the same discussion and came to the same conclusion. I will drop the test and close this issue...

mentioned in merge request !2764 (merged)

Test being removed in !2764 (merged) (should be backported also to all legacy branches)

mentioned in merge request !2766 (merged)

mentioned in merge request !2767 (merged)

mentioned in merge request !2769 (merged)

mentioned in merge request !2771 (merged)

mentioned in merge request !2773 (merged)

mentioned in merge request !2774 (merged)

mentioned in merge request !2775 (merged)

mentioned in merge request !2776 (merged)

mentioned in merge request !2777 (merged)

mentioned in merge request !2778 (merged)

mentioned in merge request !2779 (merged)

closed