GaudiConfigurables/cmt/requirements

+package GaudiConfigurables
+version v0r0
+author "Marco Clemencic"
+
+branches cmt doc python
+
+# Needed for the parsers
+use GaudiKernel v*
+
+use Python v* LCG_Interfaces -no_auto_imports
+
+#############################################################################
+private
+
+apply_pattern install_python_modules
+
+apply_pattern pyd_boost_module module=test_validator files=python/test_validator.cpp
+
+end_private

GaudiConfigurables/doc/Notes.txt

+2010-05-31 08:47:06
+Features to add:
+ * "Map" special property value (as for Vector)
+ * implementation of class aliases (e.g. EvtDataSvc -> EventDataSvc)
+   Probably it is enough to just subclass the actual class with the new name
+ * support for namespaces (how?)
+ * tools: may be with a property 'parent' that reflects itself in the name used
+   in Configurable._instances (it might not be needed to touch the actual name),
+   then the change of name of the parent must update the _instances map for the
+   private tools (do we want the tools to be accessible from the parent?)
+ * validators for components: a string property is often a place-holder for a
+   component type/name. The property should accept strings or instances and
+   validate (at least) the type of component (Algorithm, AlgTool, Service) and
+   possibly the requested interfaces.
+   When requested, the property should instantiate the proper configurable instance
+   and return it (implement the same semantic as Vector and Map?)
+ * we need a way of selecting the right validator from the C++.
+   One possibility is to pass to "genconf" the module with custom validators,
+   which must contain a map "C++ typeinfo" -> "validator instance" (or update the
+   main map in the main modules)
+ * Database of configurables.
+   It must be a text database, instead of a module, a la rootmap:
+     C++ class -> Python configurable class
+   It could be generated from the Python modules in post-process step (like the
+   rootmap or the configurable user).
+   (remember to treat correctly the zipped python directories)
+
+2010-05-31 10:46:50
+Added base classes for Service, Algorithm and AlgTool.
+
+2010-05-31 12:06:52
+Instead of the selection of the validator, we need to select the property according
+to the C++ typeinfo.
+
+2010-05-31 14:23:15
+Since the instantiation of an object with a name already used is not possible, an
+easy way of replacing the existing instance with an instance of the new type is
+needed. Or better, we need a way to copy all the set properties from one
+instance to another and the function to get the existing instance.
+
+2010-05-31 16:14:10
+Warning: ctypes-based floating point number validator not yet implemented.
+
+2010-10-28 15:30:36
+A simple extension to the Property base class allow genconf to print the C++ type of
+the Property itself.
+The template arguments of the property class can be used as template argument of
+a small helper function to produce an effective validator (using Boost Python).
+E.g.:
+// in the C++ python module
+BOOST_PYTHON_MODULE(validators)
+{
+  using namespace boost::python;
+  def("SimpleProperty<int,BoundedVerifier<int> >",
+      check<int,BoundedVerifier<int> >);
+}
+# in Python
+Property(name = "MyInt",
+         validator = findValidator('SimpleProperty<int,BoundedVerifier<int> >'),
+         default = 123,
+         doc = "An integer property")
+
+An open question is how to effectively avoid duplications. For small projects it
+may be OK to have one validation module per component library, but we should not
+need more than a handful of validators modules (we have only a limited set of
+properties).
+
+2010-10-28 18:34:20
+It has been sort of easy to modify genconf and GaudiPolicy to allow for the creation
+of the binary python validators (still with duplicates).
+I also added the generation of a python module for the configurable classes. The
+description of the components is done with a dictionary, passed to a generic function
+that translates the descriptions in classes in the namespace of the module (it
+should reduce the memory footprint of the components modules without funny deletes
+of functions).
+The database of configurables and of validators still has to be added.
+
+2010-10-28 19:25:32
+I cannot compile the validators for GaudiExamples, but I managed to get the components
+module right (at least a preliminary version).
+The requirements in GaudiPolicy may need tuning to correctly re-generate the
+configurables.
+I should also improve the documentation of the properties adding type and default
+value.
+
+2010-10-29 08:23:21
+One interesting possibility for the configurables database is to use a BerkleyDB.
+It will efficiently handle automatically merge and locking (no need for 2 steps).
+The problem is that it cannot be stored in a zip file, so either we use the binary
+Python modules directory or PATH or LD_LIBRARY_PATH. Alternatively, we can add yet
+one more environment variable.
+
+2010-10-29 15:54:47
+Several improvements in the creation, documentation of the properties.
+Tried to run the tests, but need to be fixed.
+GaudiExamples compiles and mainly works (Handles must be tested).
+
+2010-10-29 20:01:11
+To play with BDM, probably it is best to start with the validators. For the configurables
+themselves I may directly store their description directly in a DB and be done
+with it.
+
+2010-10-31 14:52:40
+The tricky part of using a db of the validators is to build the validators module
+only if needed.
+Possible solutions are:
+1) collect all the validators in a single module for the whole
+project (similar to the "merged" confDb or to the python.zip (build delayed to the
+container?).
+2) put the code to build the validator module in the genconfig fragment, so that
+it can be triggered conditionally by the presence of the .cpp file.
+
+2010-11-01 19:10:51
+This is an interesting idea about how to deal with the validators database.
+We can use a single validators module per projects (option '1' above), but we do
+not need a database, because the database of configurables is enough.
+This is how we can do it:
+ - genconf just updates the database with the configurables description
+   (it has to remove everything for the current library and re-add it)
+ - in a second step (triggered like the zipping of python modules) we can go
+   through the database of configurables to generate the python binary module
+   with the validators
+In this scheme there is no need for an actual python module with the configurables,
+but the "Configurables" module can just look-up in the database the descriptions
+and create the actual classes on demand.
+
+2010-11-02 14:03:13
+Created the meta-module GaudiConfigurables.Components which dynamically retrieve
+the component description from the database(s) and generate the code for the
+configurable class. It is interesting to not a couple of nice implementation
+details that could be used in the old Gaudi.Configurables:
+ - __all__ is a property of the instance instead of being a special case in the
+   __getattr__ method (same for __path__ and __name__)
+ - the classes that are accessed through the module become attributes of the
+   instance, so that there is no need to go through __getattr__ a second time
+The main limitations of this preliminary implementation are that the databases
+are open when the module is loaded and never closed. It should be better to open
+the databases only if we need to look into them and close them immediately after
+or, at least, when we have generated all the classes that they contain.
+I also disabled (temporarily) the generation of the validation modules.
+
+2010-11-02 17:18:15
+Generation of validator modules re-enabled (we still have duplications, but less).
+Proper discovery of validator functions for a given type. To be noted the
+interesting use of the generator function in ValidatorsDB.
+
+2010-11-03 09:31:52
+I decided, for the moment, to stick to the original plan of one validator library
+per component library.
+At least I removed the duplicated checkers leveraging on the database. I may
+have to find a way to reduce the memory usage due to the empty modules that get
+loaded (may be by flagging them somehow and not loading them).
+I also need to change genconfig to take into account all the databases to avoid
+duplicates across projects. (problem: how can I tell the difference between the
+database I have to write into and the others? "realpath" would be good, but I
+have to check if Boost has something like that)

GaudiConfigurables/doc/release.notes

+Package : GaudiKernel
+Package manager : Marco Clemencic
+

GaudiConfigurables/python/GaudiConfigurables/Components.py

+import os, sys, anydbm
+
+class _ComponentsModule(object):
+    """Module facade class to wrap the database of configurables.
+    """
+    __path__ = None
+    __name__ = __name__
+
+    def __init__(self):
+        """Initialize the instance.
+        """
+        # If set to true, does not raise an AttributeError if the configurable is not found.
+        self.ignoreMissingConfigurables = False
+        # list of databases found in the LD_LIBRARY_PATH (or PATH on Win32)
+        if sys.platform.startswith("win"): # Win
+            search_path = os.environ.get("PATH", "")
+        elif sys.platform == "darwin":
+            search_path = os.environ.get("DYLD_LIBRARY_PATH", "")
+        else: # Linux
+            search_path = os.environ.get("LD_LIBRARY_PATH", "")
+        # look for file called _components.db in all the entries of the path
+        self._databases = filter(lambda db: os.path.exists(db),
+                                 map(lambda d: os.path.join(d, "_configurables.db"),
+                                     search_path.split(os.pathsep)))
+        # open all the databases
+        self._databases = map(anydbm.open, self._databases)
+        # Classes already retrieved from the databases
+        self._classes = {}
+        import string
+        ## Translation table for _pythonizeName()
+        self._makeConfTransTable = string.maketrans("<>&*,: ().","__rp__s___")
+        ## Template for the configurable class code
+        self._makeConfTemplate = """from GaudiConfigurables import *
+from GaudiConfigurables.Properties import *
+class %(name)s(%(base)s):
+\t__properties__=(%(props)s)
+\t__cpp_type__=%(type)r"""
+        import logging
+        self._log = logging.getLogger(self.__name__)
+
+    @property
+    def __all__(self):
+        """List all the available classes in the databases"""
+        data = set()
+        for db in self._databases:
+            # Get all the entries in the database that do not have the special
+            # sequence "->" (used for meta-data)
+            data.update([x for x in db.keys() if "->" not in x])
+        # the keys in the databases are the actual C++ class names,
+        # so we have to translate them to Python identifiers
+        return sorted(map(self._pythonizeName, data))
+
+    def _pythonizeName(self, name):
+        """Convert a C++ class name to a valid Python identifier."""
+        import string
+        return string.translate(name, self._makeConfTransTable)
+
+    def _makeClass(self, name, props):
+        """Generate the configurable class 'name' from the property description.
+        @param name: name of the C++ class
+        @param props: list of property descriptions
+                      [("Prop1Name","Prop1C++Type",default,"Prop1Type",doc), ...]
+        """
+        pyname = self._pythonizeName(name)
+        self._log.debug("Generating code for class %s (%s)", pyname, name)
+        # The elements of props are tuples that match the arguments of the
+        # constructor of Property
+        props = [ "Property" + repr(p) for p in props ]
+        base = "Configurable"
+        code = self._makeConfTemplate % {"name": pyname,
+                                         "type": name,
+                                         "base": base,
+                                         "props": ",".join(props)}
+        locals = {} # temporary name space
+        try:
+            exec code in locals
+        except:
+            print "Exception in:"
+            print code
+            print "locals:", locals.keys()
+            raise
+        # Extract the new class from the temporary name space and put it in the
+        # instance' one.
+        self.__dict__[pyname] = locals[pyname]
+        if name != pyname:
+            self.__dict__[name] = self.__dict__[pyname]
+
+    def _getCompDesc(self, name):
+        """Get the description of a component class from the databases.
+        @param name: name of the component (C++ or Python)
+        @return: a tuple with the actual name and the string description or None if not found
+        """
+        for db in self._databases:
+            if db.has_key(name): # this works if name is C++ or the pythonized one is equal
+                return (name, db[name])
+            else:
+                for k in db.keys():
+                    if name == self._pythonizeName(k):
+                        return (k, db[k])
+        return None
+
+    def __getattr__(self, name):
+        """Retrieve the requested class from the database.
+        """
+        # FIXME: This is needed for properties of type handle.
+        exec "from GaudiKernel.GaudiHandles import *"
+        self._log.debug("Requested attribute %s", name)
+        nd = self._getCompDesc(name)
+        if nd is None:
+            if self.ignoreMissingConfigurables:
+                self._log.warning('Configurable class %s not in database', name)
+                return None
+            else:
+                raise AttributeError("module '%s' does not have attribute '%s'" % (self.__name__, name))
+
+        name, desc = nd
+        self._makeClass(name, eval(desc))
+        return self.__dict__[name]
+
+sys.modules[__name__] = _ComponentsModule()

GaudiConfigurables/python/GaudiConfigurables/Properties.py

+'''
+Created on 28/mag/2010
+
+@author: Marco Clemencic
+'''
+import os, sys, logging
+
+# FIXME: This is needed for properties of type handle.
+from GaudiKernel.GaudiHandles import * # pylint: disable-msg=F0401
+
+# Number of times to indent output
+# A list is used to force access by reference
+__report_indent = [0]
+def report(fn):
+    """Decorator to print information about a function
+    call for use while debugging.
+    Prints function name, arguments, and call number
+    when the function is called. Prints this information
+    again along with the return value when the function
+    returns.
+    """
+    def wrap(*params,**kwargs):
+        # pylint: disable-msg=E1101
+        # pylint: disable-msg=W0612
+        call = wrap.callcount = wrap.callcount + 1
+        # pylint: enable-msg=E1101
+        # pylint: enable-msg=W0612
+        indent = ' ' * __report_indent[0]
+        fc = "%s(%s)" % (fn.__name__, ', '.join(
+            [a.__repr__() for a in params] +
+            ["%s = %s" % (a, repr(b)) for a,b in kwargs.items()]
+        ))
+        print "%s%s called [#%s]" % (indent, fc, call)
+        __report_indent[0] += 1
+        ret = fn(*params,**kwargs)
+        __report_indent[0] -= 1
+        print "%s%s returned %s [#%s]" % (indent, fc, repr(ret), call)
+        return ret
+    wrap.callcount = 0
+    return wrap
+
+def _defaultValidator(_):
+    """Default no-op validation function."""
+    return True
+
+class ValidatorsDB(object):
+    """Class to dynamically load the validator modules.
+    """
+    def __init__(self):
+        self._log = logging.getLogger("ValidatorsDB")
+        self._modIter = self._modules()
+
+    def _modules(self):
+        """Iterator over the list of validator modules.
+        """
+        # collect the list of validator modules
+        suff = "_validators."
+        if sys.platform.startswith("win"):
+            suff += "pyd"
+        else:
+            suff += "so"
+        # actual loop
+        for d in sys.path:
+            if not d: d = '.' # empty name means '.'
+            elif not os.path.isdir(d): # skip non-directories (elif because '' is a directory)
+                continue
+            self._log.debug("Scanning directory %s", d)
+            for m in ( os.path.splitext(f)[0]
+                       for f in os.listdir(d)
+                       if f.endswith(suff) ):
+                yield m
+
+    def __getattr__(self, name):
+        """Import the validator modules until the validator for the requested
+        type is found.
+        """
+        self._log.debug("Requested validator for unknown type %s", name)
+        # loop over the "other" modules
+        for m in self._modIter:
+            # import
+            self._log.debug("Importing %s", m)
+            m = __import__(m)
+            # add the validators to the instance scope
+            for v in dir(m):
+                if not v.startswith("_"):
+                    self._log.debug("Registering type %s", v)
+                    self.__dict__[v] = getattr(m, v)
+            # check if now we have what was requested
+            if name in self.__dict__:
+                self._log.debug("Found")
+                return self.__dict__[name]
+        self._log.debug("Not found, using default")
+        self.__dict__[name] = _defaultValidator
+        return _defaultValidator
+
+_validatorsDB = ValidatorsDB()
+
+# Property and Wrapper has to be considered as a "friend" of Configurable,
+# because they have to access some internal storage, so
+# pylint: disable-msg=W0212
+# (Access to a protected member X of a client class)
+class Property(object):
+    '''
+    Generic property class to handle
+    '''
+    def __init__(self, name,
+                 cppType, default,
+                 validator = _defaultValidator, doc = None):
+        '''
+        Constructor
+        '''
+        self.name = name
+        self.cppType = cppType
+        if type(validator) is str:
+            validator = getattr(_validatorsDB, validator)
+        self._validator = validator
+
+        if self._validator(repr(default)):
+            self.default = default
+        else:
+            raise TypeError("Bad default value %r: %s" %
+                            (default, self._validator.__doc__))
+
+        # The documentation of a property is:
+        #  - optional text (specified in C++)
+        #  - C++ type
+        #  - default value
+        docs = []
+        if doc:
+            docs.append(doc.strip())
+            docs.append("")
+        docs.append("type: " + self.cppType)
+        docs.append("default: " + repr(self.default))
+        self.doc = "\n".join(docs)
+
+    def get(self, owner):
+        if self.name in owner._propertyData:
+            return owner._propertyData[self.name]
+        else:
+            # return a clone of the default
+            return type(self.default)(self.default)
+            # raise AttributeError(self.name)
+
+    def set(self, owner, value):
+        if self._validator(repr(value)):
+            owner._propertyData[self.name] = value
+        else:
+            raise TypeError("Property '%s.%s' cannot accept value %r: %s" %
+                            (owner.__class__.__name__, self.name, value, self._validator.__doc__))
+
+    def delete(self, owner):
+        if self.name in owner._propertyData:
+            del owner._propertyData[self.name]
+        # raise RuntimeError("Cannot delete property %s" % self.name)
+
+class DefaultWrapper(object):
+    """
+    Wrapper for the default value of complex mutable properties, used by
+    VectorProperty to correctly implement the "property" semantics for vectors
+    and maps.
+    """
+    def __init__(self, owner, name, value):
+        """
+        Initialize the wrapper.
+        @param owner: instance of the configurable that this list belongs to
+        @param name: name of the property in the owner
+        @param object: object to wrap (value of the property)
+        """
+        class MethodWrapper(object):
+            """
+            Simple wrapper class to implement a simple and generic closure.
+            """
+            def __init__(self, owner, name, value, method):
+                """
+                Data for the closure.
+                """
+                self.owner = owner
+                self.name = name
+                self.value = value
+                self.method = method
+            def __call__(self, *args, **kwargs):
+                """
+                Call the wrapped method and fix the property value in its owner.
+                """
+                retval = apply(getattr(self.value, self.method), args, kwargs)
+                self.owner._propertyData[self.name] = self.value
+                return retval
+        # wrap the methods that need special treatment
+        wrapped = set(["__setitem__", "__setslice__",
+                       "__iadd__", "extend",
+                       "append", "insert"])
+        for method in wrapped:
+            if hasattr(value, method):
+                setattr(self, method, MethodWrapper(owner, name, value, method))
+        # forward the other non-defined methods (forcing __str__ and __repr__
+        # because they are always present)
+        for method in [ m for m in dir(value) ]:
+            if not hasattr(self, method):
+                setattr(self, method, getattr(value, method))
+        self._value = value
+    def __str__(self):
+        return str(self._value)
+    def __repr__(self):
+        return repr(self._value)

GaudiConfigurables/python/GaudiConfigurables/Tests.py

+'''
+Created on 29/mag/2010
+
+@author: Marco Clemencic
+'''
+import unittest
+
+from GaudiConfigurables import *
+from GaudiConfigurables.Properties import *
+import test_validator
+int_val = getattr(test_validator,
+                  "SimpleProperty<int,BoundedVerifier<int> >")
+str_val = getattr(test_validator,
+                  "SimplePropertyRef<std::string,NullVerifier<std::string> >")
+vec_int_val = getattr(test_validator,
+                      "SimplePropertyRef<std::vector<int,std::allocator<int> >,NullVerifier<std::vector<int,std::allocator<int> > > >")
+vec_str_val = getattr(test_validator,
+                      "SimplePropertyRef<std::vector<std::string,std::allocator<std::string> >,NullVerifier<std::vector<std::string,std::allocator<std::string> > > >")
+
+class TestAlgorithm(Algorithm):
+    __properties__ = (Property("OutputLevel",
+                               "int", 3,
+                               int_val,
+                               "something"),
+                      Property("Items",
+                               "vector<int>", [],
+                               vec_int_val,
+                               doc = "A vector of ints"),
+                      Property("Tools",
+                               "vector<string>", [],
+                               vec_str_val,
+                               doc = "A vector of strings"))
+
+class MyAlgorithm(TestAlgorithm):
+    __properties__ = (Property("OutputLevel",
+                               "string", "ciao",
+                               str_val,
+                               "something new"),)
+
+class Test(unittest.TestCase):
+
+
+    def setUp(self):
+        pass
+
+
+    def tearDown(self):
+        # Remove all instances
+        Configurable._instances.clear()
+
+    def test_000_defaultValidator(self):
+        "defaultValidator"
+        from GaudiConfigurables.Properties import _defaultValidator
+        self.assert_(_defaultValidator("anything"))
+
+#    def test_000_CppTypes(self):
+#        "Decoding of C++ types"
+#        test_strings = [
+#                        ("std::string",
+#                            Type("std::string")),
+#                        ("std ::  basic_string",
+#                            Type("std::basic_string")),
+#                        ("map<std::string , int>",
+#                            Type("map", ["std::string","int"])),
+#                        ("std::vector < std::pair<int, double> >",
+#                            Type("std::vector",[Type("std::pair", ["int","double"])]),
+#                        ("std::map < std::pair<int, double>, std:: basic_string<char> >",
+#                            Type("std::map", [Type("std::pair", ["int", "double"]),
+#                                              Type("std:: basic_string", "char")])),
+#                        ("string<int, pair<a,b>>",
+#                            Type("string", [Type("int"),
+#                                            Type("pair", ["a","b"])])),
+#                        ]
+
+
+    def test_000_validators(self):
+        "Validators"
+        self.assert_(int_val("-100"))
+        self.assert_(int_val("0"))
+        self.assert_(int_val("100"))
+        self.assertFalse(int_val(str(1 << 31)))
+        self.assertFalse(int_val("test"))
+        self.assertFalse(int_val("[]"))
+
+        #ui32 = CIntValidator(32, signed = False)
+        #self.failUnlessRaises(ValidatorError, ui32, -100)
+        #ui32(0)
+        #ui32(100)
+        #self.failUnlessRaises(ValidatorError, ui32, 1 << 32)
+
+        #fp = CFPValidator()
+        #fp(100)
+        #fp(1000L)
+        #fp(1e10)
+        #self.failUnlessRaises(ValidatorError, fp, [])
+        #self.failUnlessRaises(ValidatorError, fp, "test")
+
+        self.assert_(str_val("test"))
+        self.assertFalse(str_val("[]"))
+
+    def test_010_singleton(self):
+        "Named singleton"
+        a = TestAlgorithm("a")
+        b = TestAlgorithm("b")
+        self.assert_(a is TestAlgorithm("a"))
+        self.assert_(b is TestAlgorithm("b"))
+        self.assert_(a is not b)
+
+        # fail to get the singleton with the wrong type
+        self.failUnlessRaises(TypeError, MyAlgorithm, "a")
+
+    def test_020_simple_type_check(self):
+        "Simple type checking"
+        self.failUnlessRaises(TypeError, TestAlgorithm, OutputLevel = "abc")
+        self.failUnlessRaises(TypeError, TestAlgorithm, Items = ["abc"])
+        self.failUnlessRaises(TypeError, TestAlgorithm, Items = 10)
+        self.failUnlessRaises(TypeError, TestAlgorithm, Items = "abc")
+        self.failUnlessRaises(TypeError, TestAlgorithm, Tools = [10])
+        self.failUnlessRaises(TypeError, TestAlgorithm, Tools = 10)
+        self.failUnlessRaises(TypeError, TestAlgorithm, Tools = "abc")
+        a = TestAlgorithm()
+        a.OutputLevel = 100
+        a.Items = range(3)
+        a.Tools = ["one", "two", "three"]
+        self.assertEquals(a.OutputLevel, 100)
+        self.assertEquals(a.Items, range(3))
+        self.assertEquals(a.Tools, ["one", "two", "three"])
+
+    def test_021_const_class(self):
+        "Class constantness"
+        try:
+            TestAlgorithm.OutputLevel = 1
+            self.fail("modified class object")
+        except AttributeError:
+            pass
+
+    def test_022_instance_name(self):
+        "Instance name"
+        a = TestAlgorithm("name1")
+        self.assert_("name1" in Configurable._instances)
+        self.assert_(Configurable._instances["name1"] is a)
+        # rename
+        a.name = "name2"
+        self.assert_("name1" not in Configurable._instances)
+        self.assert_("name2" in Configurable._instances)
+        self.assert_(Configurable._instances["name2"] is a)
+
+        # rename to an already used name
+        TestAlgorithm("name3")
+        try:
+            a.name = "name3"
+            self.fail("I managed to rename an instance using an already used name")
+        except ValueError:
+            pass
+
+        # try to unset the name
+        try:
+            del a.name
+            self.fail("I managed unset the name of an instance")
+        except AttributeError:
+            pass
+
+    def test_025_default_assignment(self):
+        "Assignment to default"
+        a = TestAlgorithm()
+
+        self.assert_(not a.isSet("OutputLevel"))
+        a.OutputLevel = a.OutputLevel
+        self.assert_(a.isSet("OutputLevel"))
+
+        self.assert_(not a.isSet("Items"))
+        a.Items = a.Items
+        self.assert_(a.isSet("Items"))
+
+    def test_025_default_retrieval(self):
+        "Default retrieval"
+        a = TestAlgorithm()
+        self.assert_(not a.isSet("OutputLevel"))
+        self.assertEquals(a.getDefault("OutputLevel"), 3)
+        a.OutputLevel = 10
+        self.assert_(a.isSet("OutputLevel"))
+        self.assertEquals(a.getDefault("OutputLevel"), 3)
+        self.failUnlessRaises(AttributeError, a.getDefault, "Undefined")
+
+        self.assert_(not a.isSet("Items"))
+        self.assertEquals(a.getDefault("Items"), [])
+
+    def test_030_vector_semantics(self):
+        "Vector semantics"
+        a = TestAlgorithm("a")
+        self.assert_(not a.isSet("Items"))
+        a.Items += [1]
+        self.assert_(a.isSet("Items"))
+        self.assertEquals(a.Items, [1])
+
+        b = TestAlgorithm("b")
+        self.assert_(not b.isSet("Items"))
+        b.Items.append(2)
+        self.assert_(b.isSet("Items"))
+        self.assertEquals(b.Items, [2])
+
+    def test_040_properties(self):
+        "Properties"
+        a = TestAlgorithm()
+        self.assertEquals(set(a.propertyNames()), set(["OutputLevel", "Items", "Tools"]))
+        self.assertEquals(a.propertyDict(), {})
+        self.assertEquals(a.propertyDict(defaults = True), {"OutputLevel": 3,
+                                                            "Items": [],
+                                                            "Tools": []})
+        a.OutputLevel = 5
+        self.assertEquals(a.propertyDict(), {"OutputLevel": 5})
+        self.assertEquals(a.propertyDict(defaults = True), {"OutputLevel": 5,
+                                                            "Items": [],
+                                                            "Tools": []})
+        a.Items = range(3)
+        self.assertEquals(a.propertyDict(), {"OutputLevel": 5, "Items": range(3)})
+        self.assertEquals(a.propertyDict(defaults = True), {"OutputLevel": 5,
+                                                            "Items": range(3),
+                                                            "Tools": []})
+
+    def test_041_inherited_properties(self):
+        "Inherited properties"
+        a = MyAlgorithm()
+        self.assertEquals(set(a.propertyNames()), set(["OutputLevel", "Items", "Tools"]))
+        self.assertEquals(a.propertyDict(), {})
+        self.assertEquals(a.propertyDict(defaults = True), {"OutputLevel": "ciao",
+                                                            "Items": [],
+                                                            "Tools": []})
+        a.OutputLevel = "test"
+        self.assertEquals(a.propertyDict(), {"OutputLevel": "test"})
+        self.assertEquals(a.propertyDict(defaults = True), {"OutputLevel": "test",
+                                                            "Items": [],
+                                                            "Tools": []})
+        a.Items = range(3)
+        self.assertEquals(a.propertyDict(), {"OutputLevel": "test", "Items": range(3)})
+        self.assertEquals(a.propertyDict(defaults = True), {"OutputLevel": "test",
+                                                            "Items": range(3),
+                                                            "Tools": []})
+
+    def test_050_pickling(self):
+        "Pickling"
+        import pickle
+        TestAlgorithm(OutputLevel = 20, Items = range(5))
+        TestAlgorithm("Prova")
+        MyAlgorithm("ATest")
+        MyAlgorithm(OutputLevel = "ciao")
+        MyAlgorithm(Items = range(2))
+        before = dict(Configurable._instances)
+
+        s = pickle.dumps(Configurable._instances, 2)
+        Configurable._instances.clear()
+        pickle.loads(s)
+        after = dict(Configurable._instances)
+
+        bk = before.keys()
+        ak = after.keys()
+        bk.sort()
+        ak.sort()
+        self.assertEquals(ak, bk)
+        for k in bk:
+            self.assertEquals(before[k].propertyDict(), after[k].propertyDict())
+
+    def test_051_representation(self):
+        "Repr format"
+        TestAlgorithm(OutputLevel = 20, Items = range(5))
+        TestAlgorithm("Prova")
+        MyAlgorithm("ATest")
+        MyAlgorithm(OutputLevel = "ciao")
+        MyAlgorithm(Items = range(2))
+        before = dict(Configurable._instances)
+
+        s = "\n".join(map(repr, Configurable._instances.values()))
+        Configurable._instances.clear()
+        exec s
+        after = dict(Configurable._instances)
+
+        bk = before.keys()
+        ak = after.keys()
+        bk.sort()
+        ak.sort()
+        self.assertEquals(ak, bk)
+        for k in bk:
+            self.assertEquals(before[k].propertyDict(), after[k].propertyDict())
+
+    def test_080_changetype_of_instance(self):
+        "Change the type of an instance"
+        NewType = MyAlgorithm
+        old_instance = TestAlgorithm("test", Items = range(3))
+        # Quoted from the documentation
+        name = old_instance.name
+        old_instance.name = name + "_old"
+        new_instance = NewType(name, **old_instance.propertyDict())
+
+        self.assertEquals(new_instance.propertyDict(), {"Items": range(3)})
+
+    def _test_300_class_alias(self):
+        "Aliases for configurable classes"
+        class TestAlgAlias(TestAlgorithm):
+            pass
+        # I do not know what to expect yet
+
+if __name__ == "__main__":
+    #import sys;sys.argv = ['', 'Test.test_010_singleton']
+    unittest.main()

GaudiConfigurables/python/GaudiConfigurables/__init__.py

+"""
+This module contains the main classes for the Python counterparts of Gaudi C++
+components used for the configuration: the "Configurable"s.
+
+Configurables are "named singletons", in the sense that they are instantiated
+with a name (by default the class name) and only one instance can exist with the
+given name.
+Two calls to the constructor of a class with the same name will return the same
+instance (even if called from different modules in the same process).
+There cannot be instances of different types with the same name. Trying to
+instantiate an object of a type with the name of an already existing object of
+another type with raise an exception. A possible work-around is
+
+  name = old_instance.name
+  old_instance.name = name + "_old"
+  new_instance = NewType(name, **old_instance.propertyDict())
+
+
+The shape of objects and classes are unmutable. It is not possible to add data
+members or methods to the classes or instances outside the definition.
+
+Configurable classes must provide the special class member "__properties__" which
+must be an iterable containing instances of the Property class (in the submodule
+Properties).
+
+@author: Marco Clemencic <marco.clemencic@cern.ch>
+"""
+
+import Properties
+
+class MetaConfigurable(type):
+    """
+    Meta-class for Configurables.
+
+    Instrument the Configurable class with the properties defined in
+    __properties__.
+    The optional __cpp_type__ can be used to specify the corresponding C++ type
+    (if not present, the Python class name is used).
+    """
+    def __new__(cls, name, bases, dct):
+        """
+        Add the property data members to the generated class and fill the __slots__
+        data member accordingly.
+        """
+        # get the slots from the base class
+        slots = set(dct.get("__slots__", []))
+        # if there are no properties defined, add anyway the __properties__
+        # member
+        if "__properties__" not in dct:
+            dct["__properties__"] = []
+        # add the requested properties
+        for p in dct["__properties__"]:
+            dct[p.name] = property(p.get, p.set, p.delete, p.doc)
+            slots.add(p.name)
+        # update the slots
+        dct["__slots__"] = tuple(slots)
+        # Add the __cpp_type__ property if not defined
+        if "__cpp_type__" not in dct:
+            dct["__cpp_type__"] = name
+        # set the default instance name if not defined
+        if "__defaultInstanceName__" not in dct:
+            dct["__defaultInstanceName__"] = dct["__cpp_type__"]
+        # generate the class
+        return type.__new__(cls, name, bases, dct)
+
+    def __setattr__(self, name, value):
+        """
+        A __setattr__ declaration here forbids modifications of the class object.
+        """
+        raise AttributeError("'%s' class has no attribute '%s'" % (self.__name__, name))
+
+# The class Configurable has got several data members that are automatically
+# added by the meta-class.
+# pylint: disable-msg=E1101
+class Configurable(object):
+    """
+    Base class for configurables.
+    """
+    __metaclass__ = MetaConfigurable
+    __slots__ = ("_name", "_propertyData")
+    # All instantiated configurables
+    _instances = {}
+
+    def _getName(self):
+        """
+        Getter for property 'name'.
+        """
+        return self._name
+    def _setName(self, name):
+        """
+        Setter for property 'name'.
+        """
+        if name not in self._instances:
+            # rename allowed
+            # remove old entry
+            del self._instances[self._name] # pylint: disable-msg=E0203
+            # add self with the new name
+            self._name = name # pylint: disable-msg=W0201
+            self._instances[self._name] = self
+        else:
+            raise ValueError("cannot rename '%s' to '%s', name already used"
+                             % (self._name, name))
+    def _delName(self):
+        """
+        Delete handler for property 'name'.
+        """
+        raise AttributeError("cannot remove attribute 'name'")
+
+    name = property(_getName, _setName, _delName, "Name of the instance")
+
+    def __new__(cls, name = None, **kwargs):
+        # Instance default name
+        if name is None:
+            name = cls.__defaultInstanceName__
+        # check if the name is already used
+        if name not in cls._instances:
+            # a new instance is needed
+            instance = super(Configurable, cls).__new__(cls)
+            cls._instances[name] = instance
+            # Initialize data members of the instance
+            #  - name
+            instance._name = name
+            #  - storage for the properties that are set
+            instance._propertyData = {}
+        else:
+            # instance already present
+            instance = cls._instances[name]
+            # check if the type matches
+            if instance.__class__ is not cls:
+                raise TypeError("Instance called '%s' already exists of class '%s' ('%s' requested)"
+                                % (name, instance.__class__.__name__, cls.__name__))
+        # set the properties from keyword arguments
+        for attr, value in kwargs.items():
+            setattr(instance, attr, value)
+        return instance
+
+    def __repr__(self):
+        """
+        Return representation string for the configurable instance.
+
+        The string can be evaluated to obtain an equivalent instance.
+        """
+        retval = self.__class__.__name__ + "("
+        if self.name != self.__class__.__defaultInstanceName__:
+            pars = [repr(self.name)]
+        else:
+            pars = []
+        retval += ",".join(pars + [ "%s=%r" % i for i in self._propertyData.items()])
+        retval += ")"
+        return retval
+
+    def __str__(self):
+        """
+        String representation as <C++ class>/<instance name>.
+        """
+        return "/".join((self.__cpp_type__, self.name))
+
+    def isSet(self, prop):
+        """
+        Tell if the property has been explicitly set.
+        """
+        return prop in self._propertyData
+
+    @classmethod
+    def getDefault(cls, prop):
+        """
+        Return the default value of a property.
+        """
+        for p in cls.__properties__:
+            if p.name == prop:
+                return p.default
+        # If not in the class, try the bases
+        for base in [ b for b in cls.__bases__ if hasattr(b, "getDefault") ]:
+            try:
+                return base.getDefault(prop)
+            except AttributeError:
+                # hide errors from the base classes
+                pass
+        raise AttributeError("'%s' object has no property '%s'"
+                             % (cls.__name__, prop))
+
+    @classmethod
+    def propertyNames(cls):
+        """
+        Return a list with the names of all the properties.
+        """
+        names = [ p.name for p in cls.__properties__ ]
+        for base in [ b for b in cls.__bases__ if hasattr(b, "propertyNames") ]:
+            names += base.propertyNames()
+        names = list(set(names)) # uniquify
+        names.sort()
+        return names
+
+    def propertyDict(self, defaults = False):
+        """
+        Return a dictionary with the value of the properties.
+
+        @param defaults: if True, the properties that are not set are included
+        in the returned dictionary with the default value, if False, only the
+        properties that are explicitly set are returned.
+        """
+        d = dict(self._propertyData)
+        if defaults:
+            for prop in [ p for p in self.propertyNames() if p not in d ]:
+                d[prop] = self.getDefault(prop)
+        return d
+
+    def __getnewargs__(self):
+        """
+        Needed for pickling with protocol 2.
+        """
+        # return the instance name in a tuple, so that it is passed to the
+        # __new__ method.
+        return (self.name,)
+
+    def __getstate__(self):
+        """
+        Required for pickling with protocol 2.
+        (otherwise the pickling uses the __slots__, which implies that the defaults
+        get set)
+        """
+        return self.propertyDict()
+
+    def __setstate__(self, state):
+        """
+        Required for pickling with protocol 2 (to match __getstate__).
+        """
+        for prop, value in state.items():
+            setattr(self, prop, value)
+
+# pylint: enable-msg=E1101
+
+class Service(Configurable):
+    """
+    Base class for standard component type "Service".
+    """
+    pass
+
+class Algorithm(Configurable):
+    """
+    Base class for standard component type "Algorithm".
+    """
+    pass
+
+class AlgTool(Configurable):
+    """
+    Base class for standard component type "AlgTool".
+    """
+    pass
+
+class Auditor(Configurable):
+    """
+    Base class for standard component type "Auditor".
+    """
+    pass

GaudiConfigurables/src/checker.cpp

+// Compile with:
+// g++ --shared -fPIC -o libchecker.so checker.cpp
+
+#include <iostream>
+#include <string>
+extern "C"
+bool check(char *s) {
+  std::string data(s);
+  std::cout << data << std::endl;
+  return data.size() > 3;
+}
+
+/* python:
+from ctypes import CDLL, c_char_p, c_bool
+lib = CDLL("./libchecker.so")
+check = lib.check
+check.argtypes = [c_char_p]
+check.restype = c_bool
+check(1)
+check("ciao")
+*/

GaudiConfigurables/src/python/test_validator.cpp

+#include <boost/python.hpp>
+#include <GaudiKernel/Property.h>
+#define PYCONF_VALIDATOR_MODULE
+#include "GaudiKernel/PyConfValidators.h"
+
+BOOST_PYTHON_MODULE(test_validator)
+{
+  using namespace boost::python;
+  def("SimpleProperty<int,BoundedVerifier<int> >",
+      check<int,BoundedVerifier<int> >);
+  def("SimpleProperty<bool,BoundedVerifier<bool> >",
+      check<bool,BoundedVerifier<bool> >);
+  def("SimplePropertyRef<std::string,NullVerifier<std::string> >",
+      check<std::string,NullVerifier<std::string> >);
+  def("SimplePropertyRef<std::vector<int,std::allocator<int> >,NullVerifier<std::vector<int,std::allocator<int> > > >",
+      check<std::vector<int,std::allocator<int> >,NullVerifier<std::vector<int,std::allocator<int> > > >);
+  def("SimplePropertyRef<std::vector<std::string,std::allocator<std::string> >,NullVerifier<std::vector<std::string,std::allocator<std::string> > > >",
+      check<std::vector<std::string,std::allocator<std::string> >,NullVerifier<std::vector<std::string,std::allocator<std::string> > > >);
+  def("SimplePropertyRef<std::vector<long,std::allocator<long> >,NullVerifier<std::vector<long,std::allocator<long> > > >",
+      check<std::vector<long,std::allocator<long> >,NullVerifier<std::vector<long,std::allocator<long> > > >);
+}

GaudiExamples/cmt/requirements

@@ -92,6 +92,22 @@ private
 apply_pattern linker_library    library=GaudiExamplesLib
 apply_pattern component_library library=GaudiExamples
 
+# StringKeyEx.cpp
+macro_append genconfig_extra_options " --validator-include GaudiKernel/StringKey.h"
+# BoostArrayProperties.cpp
+macro_append genconfig_extra_options " --validator-include GaudiKernel/BoostArrayAsProperty.h"
+# ExtendedProperties2.cpp
+macro_append genconfig_extra_options " --validator-include GaudiKernel/AlgFactory.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/Point3DTypes.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/Point4DTypes.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/Vector3DTypes.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/Vector4DTypes.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/GenericVectorTypes.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/VectorsAsProperty.h"
+macro_append genconfig_extra_options " --validator-include GaudiKernel/SVectorAsProperty.h"
+# HistoProps.cpp
+macro_append genconfig_extra_options " --validator-include GaudiKernel/HistoProperty.h"
+
 macro_append ROOT_linkopts " -lHist " \
               target-winxp " libHist.lib " \
               target-mac104 " -lHist -lMatrix "
@@ -112,3 +128,5 @@ macro_remove PyExample_use_linkopts '$(GaudiExamples_linkopts)'
 apply_pattern QMTest
 path_prepend PYTHONPATH "" \
                  QMTest "${GAUDIEXAMPLESROOT}/tests/qmtest"
+
+end_private

GaudiKernel/GaudiKernel/Property.h

@@ -121,6 +121,9 @@ protected:
   mutable PropertyCallbackFunctor* m_readCallBack   ;
   // call back functor for update
   PropertyCallbackFunctor* m_updateCallBack ;
+public:
+  /// Return the demangled name of the current property class.
+  virtual std::string propertyClass() const;
 };
 // ============================================================================
 #include "GaudiKernel/PropertyCallbackFunctor.h"

GaudiKernel/GaudiKernel/PyConfValidators.h

+#ifndef PYCONF_VALIDATOR_MODULE
+#error This header is meant only to be used in the implementation of validators for configurables
+#endif
+// Checker functions
+template <class TYPE>
+bool check(const std::string &s) {
+  TYPE tmp ;
+  return Gaudi::Parsers::parse(tmp, s).isSuccess();
+}
+template <class TYPE, class VERIFIER>
+bool check(const std::string &s) {
+  TYPE tmp ;
+  return Gaudi::Parsers::parse(tmp, s).isSuccess()
+         && VERIFIER().isValid(&tmp);
+}

GaudiKernel/cmt/requirements

@@ -40,7 +40,8 @@ macro genconfig_cmd           $(GAUDIKERNELROOT)/$(tag)/genconf.exe \
 private
 
 application genconf -import=Boost Util/genconf.cpp
-macro_append genconflinkopts "$(GaudiKernel_linkopts) $(Boost_linkopts) $(Boost_linkopts_filesystem) $(Boost_linkopts_regex) $(Boost_linkopts_program_options)"
+macro_append genconflinkopts "$(GaudiKernel_linkopts) $(Boost_linkopts) $(Boost_linkopts_filesystem) $(Boost_linkopts_regex) $(Boost_linkopts_program_options) "
+macro_append genconflinkopts "-lgdbm " target-winxp ""
 macro genconf_dependencies GaudiKernel
 
 # FIXME: (MCl) I do not understand why genconf gets the CppUnit link options,

GaudiKernel/src/Util/GaudiDBM.h

+#ifndef GAUDI_DBM_H
+#define GAUDI_DBM_H
+
+#include <gdbm.h>
+#include <sys/stat.h>
+
+#include <string>
+#include <exception>
+
+namespace Gaudi {
+  /** Wrapper class for easy access to GDBM files.
+   */
+  class DBM {
+  public:
+    /// Read-Write modes for a GDBM file.
+    enum ReadWriteType {
+      Read = GDBM_READER,           ///< read-only
+      Write = GDBM_WRITER,          ///< read/write an existing file
+      WriteOrCreate = GDBM_WRCREAT, ///< read/write an existing file or create it
+      Create = GDBM_NEWDB           ///< create a new file
+    };
+
+    /** Constructor.
+     *
+     * @param path: path to the database file
+     * @param read_write: open mode for gdbm
+     * @param mode: create mode for the file (only if the file needs to be created)
+     * @param block_size: number of bytes to read in a chunk (only used in creation)
+     */
+    DBM(const std::string &path,
+        ReadWriteType read_write = Read,
+        mode_t mode = S_IRUSR|S_IWUSR|S_IRGRP|S_IROTH, int block_size = 4096):
+      m_path(path), m_bs(block_size), m_rw(read_write), m_mode(mode), m_file(0)
+    {}
+
+    /// Destructor.
+    ~DBM() {
+      if (m_file) {
+        gdbm_close(m_file);
+      }
+    }
+
+    /** Store data in the database.
+     */
+    bool store(const std::string &k, const std::string &v, bool replace = false) {
+      checkFile();
+      datum dk = {const_cast<char*>(k.c_str()), k.size()};
+      datum dv = {const_cast<char*>(v.c_str()), v.size()};
+      return gdbm_store(m_file, dk, dv, replace ? GDBM_REPLACE : GDBM_INSERT) == 0;
+    }
+
+    /** Retrieve data from the database.
+     */
+    std::string fetch(const std::string &k) const {
+      checkFile();
+      datum dk = {const_cast<char*>(k.c_str()), k.size()};
+      datum d = gdbm_fetch(m_file, dk);
+      if (!d.dptr) {
+        // throw exception("Key not found: " + k);
+        return "";
+      }
+      std::string res(d.dptr, d.dsize);
+      free(d.dptr);
+      return res;
+    }
+
+    /** Check for existence of a key in the database.
+     */
+    bool exists(const std::string &k) const {
+      checkFile();
+      datum dk = {const_cast<char*>(k.c_str()), k.size()};
+      return gdbm_exists(m_file, dk);
+    }
+
+    /** Remove an entry in the database.
+     */
+    bool remove(const std::string &k) {
+      checkFile();
+      datum dk = {const_cast<char*>(k.c_str()), k.size()};
+      return gdbm_delete(m_file, dk) == 0;
+    }
+
+  private:
+    /// Check if the database file has been opened and open it if not.
+    void checkFile() const {
+      if (!m_file){
+        m_file = gdbm_open(const_cast<char*>(m_path.c_str()), m_bs, m_rw, m_mode, 0);
+      }
+    }
+    /// Path to the database file.
+    std::string m_path;
+    int m_bs, m_rw, m_mode;
+    /// Internal gdbm file handle.
+    mutable GDBM_FILE m_file;
+  };
+}
+
+#endif /* GAUDI_DBM_H */

GaudiKernel/src/Util/genconf.cpp

@@ -33,6 +33,8 @@
 #include "boost/algorithm/string/trim.hpp"
 #include "boost/algorithm/string/case_conv.hpp"
 #include "boost/format.hpp"
+#include <boost/foreach.hpp>
+#include <boost/tokenizer.hpp>
 #include "boost/regex.hpp"
 
 #include "GaudiKernel/System.h"
@@ -65,11 +67,10 @@
 #include <set>
 #include <vector>
 
+#include "GaudiDBM.h"
 
 #include "DsoUtils.h"
 
-
-
 namespace po = boost::program_options;
 namespace fs = boost::filesystem;
 
@@ -100,6 +101,10 @@ class configGenerator
   /// buffer of auto-generated configurables
   stringstream m_pyBuf;
 
+  stringstream m_newConfBuf;
+  set<string> m_propertyTypes;
+  set<string> m_headers;
+
   /// switch to decide if the generated configurables need
   /// to import GaudiHandles (ie: if one of the components has a XyzHandle<T>)
   bool    m_importGaudiHandles;
@@ -117,17 +122,22 @@ class configGenerator
   ///  - Name of the configurable base class for the Service component
   GaudiUtils::HashMap<std::string, std::string> m_configurable;
 
+  /// gdbm file with the database of configurables.
+  Gaudi::DBM m_confDB;
+
 public:
   configGenerator( const string& pkgName,
-		   const string& outputDirName ) :
+		   const string& outputDirName,
+		   const string& confDB ) :
     m_pkgName           ( pkgName ),
     m_outputDirName     ( outputDirName ),
     m_pyBuf             ( ),
+    m_newConfBuf        ( ),
     m_importGaudiHandles( false ),
     m_dbBuf             ( ),
-    m_configurable      ( )
+    m_configurable      ( ),
+    m_confDB            (confDB, Gaudi::DBM::WriteOrCreate)
   {}
-
   /// main entry point of this class:
   ///  - iterate over all the modules (ie: library names)
   ///  - for each module extract component informations
@@ -170,6 +180,9 @@ public:
     m_configurable[ "Service" ] = cfgService;
     m_configurable[ "ApplicationMgr" ] = cfgService;
   }
+  void setValidatorHeaders(const Strings_t& hdrs) {
+    m_headers.insert(hdrs.begin(), hdrs.end());
+  }
 
 private:
   int genComponent( const std::string& libName,
@@ -183,6 +196,59 @@ private:
     pyOut << m_pyBuf.str() << flush;
     dbOut << m_dbBuf.str() << flush;
   }
+
+  void genValidatorSource(const string &dir, const string &lib) {
+    {
+      fstream data((fs::path(dir) / fs::path(lib + "_validators.cpp")).string().c_str(),
+          ios_base::out|ios_base::trunc);
+      data << "// --- Auto generated file: DO NOT EDIT ---\n\n"
+              "// This needs to be first to avoid warnings\n"
+              "#include <boost/python.hpp>\n"
+              "// Headers required by the validators\n"
+              "#include <GaudiKernel/Property.h>\n";
+      for (set<string>::iterator t = m_headers.begin();
+          t != m_headers.end(); ++t) {
+        data << "#include <" << *t << ">\n";
+      }
+      data << "// Common code for the validators (implementation of \"check\")\n"
+              "//   must be after the headers\n"
+              "#define PYCONF_VALIDATOR_MODULE\n"
+              "#include <GaudiKernel/PyConfValidators.h>\n"
+              "// Code of the Python module\n"
+           << "BOOST_PYTHON_MODULE(" << lib << "_validators)\n{\n"
+              "using namespace boost::python;\n";
+
+      for (set<string>::iterator t = m_propertyTypes.begin();
+          t != m_propertyTypes.end(); ++t) {
+        const string valKey = "validator->" + *t;
+        if (m_confDB.exists(valKey) && (m_confDB.fetch(valKey) != lib))
+          continue; // skip the property type if already present in another validator library
+
+        data << "def(\"" << *t << "\",\n"
+                "    check";
+        string::size_type p = t->find_first_of('<');
+        if (p != string::npos) {
+          data << t->substr(p);
+        } // Special cases
+        else if (*t == "GaudiHandleProperty") { // equivalent to a string
+          data << "<std::string,NullVerifier<std::string> >";
+        }
+        else if (*t == "GaudiHandleArrayProperty") { // equivalent to vector<string>
+          data << "<std::vector<std::string,std::allocator<std::string> >,NullVerifier<std::vector<std::string,std::allocator<std::string> > > >";
+        }
+        else { // If unknown, use the catch-all string type
+          data << "<std::string,NullVerifier<std::string> >";
+        }
+        data << ");\n";
+
+        // mark this property type as present for this library
+        m_confDB.store(valKey, lib, true);
+      }
+      data << "}\n";
+
+    }
+  }
+
   void genTrailer( std::ostream& pyOut,
 		   std::ostream& dbOut );
 
@@ -231,6 +297,12 @@ int main ( int argc, char** argv )
     ("load-library,l",
      po::value< Strings_t >()->composing(),
      "preloading library")
+    ("validator-include,I",
+     po::value<Strings_t>(),
+     "extra headers to be added to the validator C++/Python module")
+    ("configurables-db",
+     po::value<string>()->default_value("../genConf/_configurables.db"),
+     "gdbm file containing the database of configurables")
     ;
 
   // declare a group of options that will be allowed both on command line
@@ -385,13 +457,16 @@ int main ( int argc, char** argv )
   copy( libs.begin(), libs.end(), ostream_iterator<string>(cout, " ") );
   cout << "] ::::::" << endl;
 
-  configGenerator py( pkgName, out.string() );
+  configGenerator py( pkgName, out.string(),
+                      vm["configurables-db"].as<string>() );
   py.setConfigurableModule     (vm["configurable-module"].as<string>());
   py.setConfigurableDefaultName(vm["configurable-default-name"].as<string>());
   py.setConfigurableAlgorithm  (vm["configurable-algorithm"].as<string>());
   py.setConfigurableAlgTool    (vm["configurable-algtool"].as<string>());
   py.setConfigurableAuditor    (vm["configurable-auditor"].as<string>());
   py.setConfigurableService    (vm["configurable-service"].as<string>());
+  if (vm.count("validator-include"))
+    py.setValidatorHeaders(vm["validator-include"].as<Strings_t>());
 
   int sc = EXIT_FAILURE;
   try {
@@ -478,6 +553,8 @@ int configGenerator::genConfig( const Strings_t& libs )
     // reset state
     m_importGaudiHandles = false;
     m_pyBuf.str("");
+    m_newConfBuf.str("");
+    m_propertyTypes.clear();
     m_dbBuf.str("");
 
     // Scan the pluginSvc namespace and store the "background" of already
@@ -504,6 +581,21 @@ int configGenerator::genConfig( const Strings_t& libs )
       continue;
     }
 
+    const std::string compLibKey(*iLib + "->components");
+    {
+      // Remove the components belonging to this library from the database
+      std::string components = m_confDB.fetch(compLibKey);
+      typedef boost::char_separator<char> sep;
+      typedef boost::tokenizer<sep> tokenizer;
+      tokenizer tokens(components, sep(";"));
+      BOOST_FOREACH(std::string t, tokens)
+      {
+        m_confDB.remove(t);
+      }
+    }
+    // keep trace of the components in the current library, to store them in the
+    stringstream componentsInLib;
+    bool firstComponentInLib = true;
     for ( Member_Iterator it = factories.FunctionMember_Begin();
           it != factories.FunctionMember_End();
           ++it ) {
@@ -625,6 +717,12 @@ int configGenerator::genConfig( const Strings_t& libs )
           allGood = false;
         }
         prop->release();
+        if (!firstComponentInLib) {
+          componentsInLib << ';';
+        } else {
+          firstComponentInLib = false;
+        }
+        componentsInLib << name;
       } else {
         cout << "ERROR: could not cast IInterface* object to an IProperty* !\n"
              << "ERROR: return type from PluginSvc is [" << rtype << "]...\n"
@@ -635,6 +733,9 @@ int configGenerator::genConfig( const Strings_t& libs )
       }
     } //> end loop over factories
 
+    /// Store in the database the list of components that are in the library.
+    m_confDB.store(compLibKey, componentsInLib.str(), true);
+
     ///
     /// write-out files for this library
     ///
@@ -652,6 +753,25 @@ int configGenerator::genConfig( const Strings_t& libs )
     genBody   ( py, db );
     genTrailer( py, db );
 
+    // update the list of headers required by the validators
+    {
+      const string k(*iLib + "->headers");
+      if (!m_headers.empty()) {
+        stringstream hdrs;
+        for (set<string>::iterator h = m_headers.begin(); h != m_headers.end(); ++h) {
+          if (h != m_headers.begin())
+            hdrs << ';';
+          hdrs << *h;
+        }
+        m_confDB.store(k, hdrs.str(), true);
+      } else {
+        if (m_confDB.exists(k)) m_confDB.remove(k);
+      }
+    }
+
+    genValidatorSource((fs::path("..") / System::getEnv("CMTCONFIG")).string(),
+                       *iLib);
+
   } //> end loop over libraries
 
   dummySvc->release();
@@ -759,6 +879,8 @@ configGenerator::genComponent( const std::string& libName,
   string cname = componentName;
   pythonizeName(cname);
 
+  stringstream compDesc;
+
   typedef GaudiUtils::HashMap<std::string, std::string> PropertyDoc_t;
   PropertyDoc_t propDoc;
 
@@ -767,10 +889,17 @@ configGenerator::genComponent( const std::string& libName,
           << "( " << m_configurable[componentType] << " ) :"
           << "\n";
   m_pyBuf << "  __slots__ = { \n";
+
+
+  // The format of a component description is:
+  //   [<prop_desc>,...]
+  // The class name is the key in the database.
+  compDesc << '[';
   for ( vector<Property*>::const_iterator it = properties.begin() ;
         it != properties.end(); ++it ) {
 
     const string pname  = (*it)->name();
+    const string pclass = (*it)->propertyClass();
     // Validate property name (it must be a valid Python identifier)
     if (!boost::regex_match(pname, pythonIdentifier)) {
       std::cout << "ERROR: invalid property name \"" << pname
@@ -784,13 +913,31 @@ configGenerator::genComponent( const std::string& libName,
 
     string pvalue, ptype;
     pythonizeValue( (*it), pvalue, ptype );
+
+    m_pyBuf << "    # " << pclass << "\n";
     m_pyBuf << "    '" << pname << "' : " << pvalue <<", # " << ptype << "\n";
 
     if ( (*it)->documentation() != "none" ) {
       propDoc[pname] = (*it)->documentation();
     }
 
+    // Property description:
+    //   (name, cppType, default, propClass, doc)
+    compDesc << "('" << pname << "'"
+             << ",'" << System::typeinfoName(*(*it)->type_info()) << "'"
+             << ',' << pvalue
+             << ",'" << pclass << "',";
+    if ( (*it)->documentation() != "none" ) {
+      compDesc << "'''" << (*it)->documentation() << "'''";
+    } else {
+      compDesc << "None";
+    }
+    compDesc << "),";
+    m_propertyTypes.insert(pclass);
   }
+  compDesc << "]";
+  m_confDB.store(componentName, compDesc.str());
+
   m_pyBuf << "  }\n";
   m_pyBuf << "  _propertyDocDct = { \n";
   for ( PropertyDoc_t::const_iterator iProp = propDoc.begin();

GaudiPolicy/cmt/fragments/genconfig_header

@@ -34,7 +34,9 @@ $(outdir)/$(confpy) : $(conflib) $(outdir)
 	  --configurable-algorithm=$(confAlgorithm) \
 	  --configurable-algtool=$(confAlgTool) \
 	  --configurable-auditor=$(confAuditor) \
-          --configurable-service=$(confService) \
+	  --configurable-service=$(confService) \
+	  --configurables-db=$(confDB) \
+	  $(genconfig_extra_options) \
 	  -i ../$(tag)/$(library_prefix)$(library).$(shlibsuffix)
 
 $(outdir):

GaudiPolicy/cmt/requirements

@@ -270,12 +270,12 @@ pattern pyd_module \
   macro shlibsuffix "$(shlibsuffix)" target_<module>&target-winxp "pyd" ; \
   macro library_prefix "$(library_prefix)" target_<module> "" ; \
   macro <module>_install_dir "$(CMTINSTALLAREA)/$(python_bin_module_dir)" target-winxp "$(CMTINSTALLAREA)\$(python_bin_module_dir)" ; \
-  library <module> -no_static -import=Python -target_tag install_dir="$(<module>_install_dir)" <files> ; \
+  library <module> -no_static -group=<group> -import=Python -target_tag install_dir="$(<module>_install_dir)" <files> ; \
   end_private ; \
   macro_append <module>_cppflags " -ftemplate-depth-64" target-winxp "" ;
 # Boost version of the pattern
 pattern pyd_boost_module \
-  apply_pattern pyd_module module=<module> files=<files> deps=<deps> name=<name> ; \
+  apply_pattern pyd_module module=<module> files=<files> deps=<deps> name=<name> group=<group> ; \
   macro_append <module>_use_linkopts " $(Boost_linkopts_python)"
 
 cmtpath_pattern \
@@ -956,6 +956,7 @@ macro genconfig_configurableAlgorithm "ConfigurableAlgorithm"
 macro genconfig_configurableAlgTool "ConfigurableAlgTool"
 macro genconfig_configurableAuditor "ConfigurableAuditor"
 macro genconfig_configurableService "ConfigurableService"
+macro genconfig_configurablesDB "$(CMTINSTALLAREA)/$(tag)/lib/_configurables.db"
 
 ## add pattern and macro for library preloading for genconf
 
@@ -1021,7 +1022,7 @@ pattern do_real_genconfig \
   document genconfig <library>Conf -s=../$(tag) -group=<group> \
            outdir=$(<package>_root)$(genconfDir)<package> \
            library=<library> \
-	   library_preload=$(genconfig_preload) \
+           library_preload=$(genconfig_preload) \
            conf_destdir=$(genconfInstallDir) \
            confModuleName=$(genconfig_configurableModuleName) \
            confDefaultName=$(genconfig_configurableDefaultName) \
@@ -1029,9 +1030,12 @@ pattern do_real_genconfig \
            confAlgTool=$(genconfig_configurableAlgTool) \
            confAuditor=$(genconfig_configurableAuditor) \
            confService=$(genconfig_configurableService) \
+           confDB=$(genconfig_configurablesDB) \
            $(library_prefix)<library>.$(shlibsuffix) ; \
   macro_append <library>Conf_dependencies " <library> " ; \
   apply_pattern install_python_init package=<package> group=<group> ; \
+  apply_pattern pyd_boost_module module=<library>_validators group=<group> files=../$(tag)/<library>_validators.cpp ; \
+  macro_append <library>_validators_dependencies "<library>Conf" ; \
   macro_append <group><package>_python_init_dependencies " <library>Conf " ; \
   private ; \
     macro merge_genconfDb_tag "--do-merge"   \

GaudiPolicy/scripts/gen_validators_src.py

+#!/usr/bin/env python
+
+import os, sys, anydbm
+
+# FIXME: This is needed for properties of type handle.
+from GaudiKernel.GaudiHandles import *
+
+if os.path.exists(sys.argv[1]):
+    db = anydbm.open(sys.argv[1])
+else:
+    print "%s:WARNING:" % sys.argv[0], "cannot find database file", sys.argv[1]
+    db = {}
+keys = db.keys()
+
+cpp = ['// --- Auto generated file: DO NOT EDIT ---\n',
+       '// This needs to be first to avoid warnings',
+       '#include <boost/python.hpp>',
+       '// Headers required by the validators',
+       '#include <GaudiKernel/Property.h>']
+
+# collect extra headers
+hdrs = set()
+for k in [ k for k in keys if k.endswith("->headers") ]:
+    hdrs.update(db[k].split(";"))
+for h in sorted(hdrs):
+    cpp.append("#include <%s>" % h)
+
+module = os.path.splitext(os.path.basename(sys.argv[2]))[0]
+cpp += ['// Common code for the validators (implementation of "check")',
+        '#define PYCONF_VALIDATOR_MODULE',
+        '#include <GaudiKernel/PyConfValidators.h>',
+        '// Code of the Python module',
+        'BOOST_PYTHON_MODULE(%s)' % module,
+        '{',
+        'using namespace boost::python;']
+
+# collect all the validators
+all = set()
+for k in [ k for k in db.keys() if "->" not in k ]:
+    all.update([ x[3] for x in eval(db[k]) ])
+for p in sorted(all):
+    cpp.append('def("%s",' % p)
+    pos = p.find("<")
+    if pos >= 0:
+        t = p[pos:]
+    elif p == "GaudiHandleProperty": # equivalent to a string
+        t = "<std::string,NullVerifier<std::string> >"
+    elif p == "GaudiHandleArrayProperty": # equivalent to vector<string>
+        t = "<std::vector<std::string,std::allocator<std::string> >,NullVerifier<std::vector<std::string,std::allocator<std::string> > > >"
+    else: # If unknown, use the catch-all string type
+        t = "<std::string,NullVerifier<std::string> >"
+    cpp.append('    check%s);' % t)
+
+cpp.append("}")
+
+open(sys.argv[2], "w").write("\n".join(cpp) + "\n")