Commit 7d6b0ed6 authored by marcocle's avatar marcocle
Browse files

Restructuring of the repository.

See https://twiki.cern.ch/twiki/bin/view/Gaudi/GaudiSVNRepository


git-svn-id: svn+ssh://svn.cern.ch/reps/gaudi/Gaudi/trunk@6014 53ecefc3-a64d-0410-9bdf-b8581fa3287e
parents
package Gaudi
version v21r5
branches doc cmt
use GaudiSys v*
apply_pattern ProjectVersionHeader
apply_pattern install_scripts
apply_pattern install_python_modules
set GAUDIEXE $(GAUDIROOT)/$(tag)/Gaudi.exe \
GAUDI_with_installarea $(Gaudi_cmtpath)/$(GAUDI_installarea_prefix)/$(tag)/bin/Gaudi.exe
# don't need to put the full path for the executable
# they are on the path
alias Gaudi "Gaudi.exe"
alias GaudiRun "gaudirun.py"
private
application Gaudi main.cpp
path_remove JOBOPTSEARCHPATH "/Gaudi/" target-winxp "\Gaudi\"
path_prepend JOBOPTSEARCHPATH "$(GAUDIROOT)/options" target-winxp "$(GAUDIROOT)\options"
apply_pattern QMTest
path_prepend JOBOPTSEARCHPATH "" \
QMTest&target-winxp "$(GAUDIROOT)\tests\pyjobopts" \
QMTest "$(GAUDIROOT)/tests/pyjobopts"
path_prepend PYTHONPATH "" \
QMTest&target-winxp "$(GAUDIROOT)\tests\python" \
QMTest "$(GAUDIROOT)/tests/python"
Package : Gaudi
Package manager : Marco Clemencic
================ Gaudi v21r5 ====================================
! 2009-10-19 - Marco Clemencic
- Modified the python modules Gaudi, Gaudi.Main and the script guadirun.py
to avoid that an "import Gaudi" indirectly triggers a load of the configurables
database (it is necessary to import Gaudi.Configuration).
! 2009-10-12 - Marco Clemencic
- Normalization of white-spaces in Python files.
(using http://svn.python.org/projects/python/tags/r254/Tools/scripts/reindent.py)
! 2009-10-09 - Eoin Smith
- Patch #3297: Support for GaudiPython Parallel in gaudirun.py
Modified gaudirun.py to accept the command line option "--parallel" to
specify the number of workers to use.
================ Gaudi v21r4 ====================================
! 2009-07-22 - Marco Clemencic
- Modified the the code behind "from Configuables import MyConf" to match the
standard Python semantics: raise an exception if the configurable is not
found.
The meta-module "Configuables" should be deprecated in favor of
"Gaudi.Configurables", that can also be used as in:
import Gaudi.Configurables as cfgs
cfgs.MyConf
The exception for missing configurables can be disabled with
import Gaudi.Configurables
Gaudi.Configurables.ignoreMissingConfigurables = True
(done by default in genconfuser.py for backward compatibility).
! 2009-07-20 - Marco Clemencic
- Moved the use of tcmalloc from Gaudi to GaudiPolicy to make it available
everywhere.
================ Gaudi v21r3 ====================================
================ Gaudi v21r2 ====================================
! 2009-06-25 - Marco Clemencic
- Added support for tcmalloc (via LD_PRELOAD) in gaudirun.py (option -T/--tcmalloc).
! 2009-06-15 - Hubert Degaudenzi
- use the new LCG tag convention.
! 2009-05-25 - Marco Clemencic
- Fixed bug #45517: Units not taken into account in .opts files
Added the function importUnits() to populate from python the units used in a
.opts file.
Added a test for the new function.
================ Gaudi v21r1 ====================================
================ Gaudi v21r0 ====================================
! 2009-04-08 - Wouter Hulsbergen
- Added '--printsequence' to gaudirun.py to print the sequence of algorithms
after the configuration.
! 2009-01-29 - Marco Clemencic
- Symbol visibility
- Updated main.cpp to use the new GAUDI_IMPORT macro.
================ Gaudi v20r4 ====================================
! 2008-12-16 - Marco Clemencic
- Fixed bug #45392: gaudrun.py returns status code 0 even if the application
manager fails
The return code of AppMgr.run is now correctly propagated.
! 2008-12-15 - Marco Clemencic
- Added a test for the passive "depedency" of ConfigurableUser.
! 2008-12-05 - Marco Clemencic
- Added a test for the new logic of ConfigurableUser.propagateProperty.
================ Gaudi v20r3 ====================================
! 2008-11-04 - Marco Clemencic
- Improvements in gaudirun.py:
- Added command line option --post-option to accept (single line) options to
be executed after the apply of ConfigurableUser.
- Modified --option to allow execution of the option in the actual position
on the command line with respect to the option files.
- Added --debug to enable some debugging print-out in gaudirun.py and in the
Configurables.
! 2008-11-03 - Marco Clemencic
- Added the function Gaudi.Configuration.getConfigurable to simplify the
creation of configurable instances. If a configurable wit the give name
exists it is returned, otherwise it is created using the name as its type or
the specified type.
! 2008-10-30 - Marco Clemencic
- Extended gaudirun.py to use the automatic apply of ConfigurableUser.
- Adde the options:
--output (to replace -p and allow dump to file in python or old options)
--no-conf-user-apply (to disable the automatic apply of ConfigurableUser)
! 2008-10-23 - Marco Clemencic
- Added a test for bug #42428 (properties of clones are not cloned).
- Added the directory tests/python for modules needed for the tests
- added a fake configurable MyToolTest into the new directory
- Added a test for bug #43065 (Error when unpickling GaudiHandles) using the
fake configurable MyToolTest, within a more generic (in principle) test for
pickling (gaudi.pickle_write) and unpickling (gaudi.pickle_read).
! 2008-10-21 - Marco Clemencic
- Minor changes in gaudirun.py to make use of the improvements in
GaudiKernel.ProcessJobOptions.
! 2008-10-17 - Marco Clemencic
- Fixed bug #39846. gaudirun.py creates invalid entries in flattened options
- Use only the configurables that pass the filter "getNeededConfigurables"
(from GaudiKernel.Proxy.Configurable) to generate pickle file or dump.
- Added test (gaudi.bug_39846).
! 2008-08-06 - Marco Clemencic
- Added a test for bug #39809 (wrong configuration of public tools).
! 2008-08-05 - Marco Clemencic
- Fixed bug #39116. gaudirun.py export to old options prints '#' instead of '//'
The fix required a review of the way messages are printed across the whole
Configurables machinery. The Python standard logging module is now used and
the customization in GaudiKernel.Logging has been removed.
In printout in Python job option files should be done with the log instance
accessible from Gaudi.Configuration.
- Added a test for bug #39116 (gaudi.export_oldopts).
================ Gaudi v20r2 ====================================
================ Gaudi v20r1 ====================================
! 2008-06-25 - Marco Clemencic
- Added tests for:
- bug #38206: gaudirun.py fails to set a list property from a property
reference.
- bug #38194: gaudirun.py expands too many environment variables.
================ Gaudi v20r0 ====================================
! 2008-06-13 - Marco Clemencic
- Apply the pattern ProjectVersionHeader to generate the GAUDI_VERSION.h file.
Added the version.cmt file too, so that the pattern works without version
directory.
! 2008-06-05 - Marco Clemencic
- Added tests for:
- Bug #36742: Cannot pickle options using "PropertyReference"
- Bug #37479: gaudirun.py fails parsing old option lines with comments
and quotes.
- Bug #37488: importOptions does not expand environment variables.
- Bug #35347: Cannot import 2 Python option files with the same name.
- Bug #34769: Assigning a configurable to a list property does not fail.
- Added Windows batch wrapper for gaudirun.py (gaudirun.bat).
- Added to the command line option "--option" to gaudirun.py.
It allows to specify Python lines to be executed after the standard
configuration to modify it. E.g.:
gaudirun.py default.py --option "MessageSvc().OutputLevel = DEBUG"
A test has been added too (gaudi.cmdline_conf).
================ Gaudi v19r9 ====================================
! 2008-04-04 - Marco Clemencic
- Bug #35201. Python options parser '//' interpreted as comment in strings.
Added a test for the bug.
! 2008-03-07 - Marco Clemencic
- Fixed Bug #34046. False warning when mixing Python options and old options.
Improved the test on case sensitivity to expose the bug (if present).
! 2008-03-06 - Marco Clemencic
- Modified Gaudi.Main.gaudimain to use the environment variables GAUDIAPPNAME
and GAUDIAPPVERSION to set the default values for the ApplicationMgr
properties AppName and AppVersion.
================ Gaudi v19r8 ====================================
! 2008-02-11 - Marco Clemencic
- Added aliases for the configurables FileCatalog (Gaudi::MultiFileCatalog) and
IODataManager (Gaudi::IODataManager).
! 2008-02-08 - Marco Clemencic
- Modified python/Gaudi/CommonGaudiConfigurables.py to ignore missing packages,
essentially to allow the import of Gaudi.Configuration when GaudiPoolDb is
missing.
================ Gaudi v19r7 ====================================
! 2008-01-11 - Marco Clemencic
- Added a test to check the late-bound references in old options
(e.g. "MyAlg.MyProp = @AnotherAlg.AProp;").
- Added a simple test for properties with wrong cases in (old) job options.
! 2008-01-09 - Marco Clemencic
- Added a test to check the conversion between configurables and strings.
================ Gaudi v19r6 ====================================
! 2007-11-16 - Marco Clemencic
- Added a couple of tests for the correct overriding of mixed options (exposing
a bug fixed in GaudiKernel).
! 2007-11-14 - Marco Clemencic
- Added a test for the function GaudiKernel.Configurable.purge.
! 2007-11-12 - Marco Clemencic
- Moved the *.qmt files to the implicit suite (directory) "gaudi.qms".
! 2007-11-01 - Marco Clemencic
- Added the function Gaudi.Configuration.configurationDict to produce the
dictionary of configurations.
- Minor changes in the command line options for gaudirun.py:
- short option for "--dry-run" in now "-n"
- the configuration is printed as dictionary by default and the old format is
activated with the option "--old-opts" (removed "--opts-dict")
- Added tests for the job option parser.
- Moved "importOptions" from Gaudi.Configuration to
GaudiKernel.ProcessJobOptions. The function is still accessible from
Gaudi.Configuration (from Gaudi.Configuration import importOptions).
! 2007-10-29 - Marco Clemencic
- Moved the command line logic from the class Gaudi.Main.gaudimain to the
script gaudirun.py, using optparse instead of getopt.
- Added an option to format the dump of the computed properties as a Python
dictionary (formatted such that it can be evaluated in Python).
- Renamed some command line options.
- The different behaviors depending on the file name extension, are delegated
to Configuration.importOptions().
! 2007-10-29 - Pere Mato
- Adapted to new GaudiPython structure
==================== Gaudi v19r5 ======================================
! 2007-10-10 - Pere Mato
- Added GaudiRun alias to "gaudirun.py"
! 2007-10-02 - Pere Mato
Patch #1400: Support for configuration in Python pickle files
! 2007-09-28 - Pere Mato
- Renamed the meta module containing common configurables from ConfDb to
Configurables.
! 2007-09-24 - Hubert Degaudenzi
- changed back Configurables.py into Configurable.py (it was
a mistake)
! 2007-09-21 - Hubert Degaudenzi
- New package to contain the main part of the Gaudi application.
Moved the necessary parts from GaudiKernel.
import sys
import Gaudi.Configurables
sys.modules[__name__] = Gaudi.Configurables
# File: Gaudi/CommonGaudiConfigurables.py
# Author: Pere Mato (per.mato@cern.ch)
"""
This module would scan all known Gaudi configurable modules for
'Configurable' classes and fill __all__ such that it can be imported
by any module requiring it.
"""
from GaudiKernel.Configurable import Configurable
from GaudiKernel.ConfigurableMeta import ConfigurableMeta
__all__ = []
packages = ['GaudiSvc', 'GaudiAlg', 'GaudiAud', 'GaudiPoolDb', 'RootHistCnv',
'GaudiUtils' ]
#--Loop open all listed packages and populate __all__ with the names and
# the local scope with the Configurable classes
for package in packages :
try:
mod = __import__( '%s.%sConf'%(package,package), globals(), locals(), ['%sConf'%package] )
for nam in dir(mod) :
cls = getattr(mod, nam)
if type(cls) is ConfigurableMeta and issubclass(cls, Configurable) :
globals()[nam] = cls
__all__.append(nam)
except ImportError:
# ignore the configurables from missing packages.
pass
#--Fix some of the name idiosyncrasies in Gaudi
def addConfigurableAs(ori, new) :
gbl = globals()
if ori in gbl: # do the aliasing only if the original is available
gbl[new] = gbl[ori]
__all__.append(new)
gbl[new].DefaultedName = new
addConfigurableAs('EvtDataSvc','EventDataSvc')
addConfigurableAs('DetDataSvc','DetectorDataSvc')
addConfigurableAs('HistogramSvc','HistogramDataSvc')
addConfigurableAs('HbookCnv__PersSvc','HbookHistSvc')
addConfigurableAs('RootHistCnv__PersSvc','RootHistSvc')
addConfigurableAs('EvtPersistencySvc','EventPersistencySvc')
addConfigurableAs('DetPersistencySvc','DetectorPersistencySvc')
addConfigurableAs('HistogramPersistencySvc','HistogramPersistencySvc')
addConfigurableAs('Gaudi__MultiFileCatalog','FileCatalog')
addConfigurableAs('Gaudi__IODataManager','IODataManager')
# File: Gaudi/python/Gaudi/Configuration.py
# Author: Pere Mato (per.mato@cern.ch)
from GaudiKernel.Constants import *
from GaudiKernel.Configurable import *
from GaudiKernel.ConfigurableDb import loadConfigurableDb, cfgDb
from GaudiKernel.ConfigurableDb import getConfigurable as confDbGetConfigurable
from CommonGaudiConfigurables import *
from GaudiKernel.ProcessJobOptions import importOptions, importUnits
from GaudiKernel.ProcessJobOptions import InstallRootLoggingHandler as _InstallRootLoggingHandler
import logging
log = logging.getLogger(__name__)
# Ensure that a root logging handler is always present.
_InstallRootLoggingHandler()
allConfigurables = Configurable.allConfigurables
def _fillConfDict():
nFiles = loadConfigurableDb()
log = logging.getLogger( 'PropertyProxy' )
log.debug( "Read module info for %d configurables from %d genConfDb files",
len(cfgDb), nFiles )
if len(cfgDb.duplicates()) > 0:
log.warning( "Found %d duplicates among the %d genConfDb files :",
len(cfgDb.duplicates()), nFiles )
log.warning( "--------------------------------------------------" )
log.warning( " -%s: %s - %s",
"<component name>", "<module>", "[ <duplicates> ]" )
log.warning( "--------------------------------------------------" )
dups = cfgDb.duplicates()
for cfgName in dups.keys():
log.warning( " -%s: %s - %s",
cfgName,
cfgDb[cfgName]['module'],
str([ d['module'] for d in dups[cfgName]]) )
pass
del dups
log.warning( "Fix your cmt/requirements file !!" )
pass
else:
log.debug( "No duplicates have been found: that's good !" )
pass
return
# fill the configurable dictionary at module load
_fillConfDict()
import os, sys
def importConfiguration(conf, local=locals()) :
local[conf] = confDbGetConfigurable(conf)
def configurationDict(all=False):
"""Return a dictionary representing the configuration.
The dictionary contains one entry per configurable which is a dictionary
with one entry per property.
The optional argument "all" is used to decide if to inluce only values
different from the default or all of them.
"""
from GaudiKernel.Proxy.Configurable import getNeededConfigurables
catalog = allConfigurables
keys = getNeededConfigurables() # use only interesting configurables
conf_dict = {}
if all:
for n in keys :
if n not in conf_dict:
conf_dict[n] = {}
for p, v in catalog[n].getDefaultProperties().items() :
conf_dict[n][p] = v
for n in keys :
if n not in conf_dict:
conf_dict[n] = {}
for p, v in catalog[n].getValuedProperties().items() :
conf_dict[n][p] = v
# purge empty configurables
keys = conf_dict.keys()
for n in keys:
if not conf_dict[n]:
del conf_dict[n]
return conf_dict
def getConfigurable(name, defaultType = None):
"""Helper function to get a configurable with the given name regardless
for the type.
If defaultType can be a class derived from configurable or a string. If not
specified, the tool name is used as type."""
if name in allConfigurables:
return allConfigurables[name]
else:
# if the configurable is not found, we need to instantiate it
if defaultType is None:
# try to use the name of the configurable as default type
defaultType = name
if type(defaultType) is str:
# we need to convert from string to actual class
if defaultType in globals():
# We the type is defined in the global namespace
defaultType = globals()[defaultType]
else:
# otherwise we try to get it from the Configurables database
defaultType = getattr(Configurables, defaultType)
return defaultType(name)
import sys, os
from time import time
from Gaudi import Configuration
import logging
log = logging.getLogger(__name__)
class gaudimain(object) :
def __init__(self) :
from Configurables import ApplicationMgr
appMgr = ApplicationMgr()
if "GAUDIAPPNAME" in os.environ:
appMgr.AppName = str(os.environ["GAUDIAPPNAME"])
if "GAUDIAPPVERSION" in os.environ:
appMgr.AppVersion = str(os.environ["GAUDIAPPVERSION"])
def generatePyOutput(self, all = False):
from pprint import pformat
conf_dict = Configuration.configurationDict(all)
return pformat(conf_dict)
def generateOptsOutput(self, all = False):
from pprint import pformat
conf_dict = Configuration.configurationDict(all)
out = []
names = conf_dict.keys()
names.sort()
for n in names:
props = conf_dict[n].keys()
props.sort()
for p in props:
out.append('%s.%s = %s;' % (n,p, repr(conf_dict[n][p])))
return "\n".join(out)
def _writepickle(self, filename) :
#--- Lets take the first file input file as the name of the pickle file
import pickle
output = open(filename, 'wb')
# Dump only the the configurables that make sense to dump (not User ones)
from GaudiKernel.Proxy.Configurable import getNeededConfigurables
to_dump = {}
for n in getNeededConfigurables():
to_dump[n] = Configuration.allConfigurables[n]
pickle.dump(to_dump, output, -1)
output.close()
def printconfig(self, old_format = False, all = False) :
msg = 'Dumping all configurables and properties'
if not all:
msg += ' (different from default)'
log.info(msg)
conf_dict = Configuration.configurationDict(all)
if old_format:
print self.generateOptsOutput(all)
else:
print self.generatePyOutput(all)
def writeconfig(self, filename, all = False):
write = { ".pkl" : lambda filename, all: self._writepickle(filename),
".py" : lambda filename, all: open(filename,"w").write(self.generatePyOutput(all) + "\n"),
".opts": lambda filename, all: open(filename,"w").write(self.generateOptsOutput(all) + "\n"),
}
from os.path import splitext
ext = splitext(filename)[1]
if ext in write:
write[ext](filename, all)
else:
log.error("Unknown file type '%s'. Must be any of %r.", ext, write.keys())
sys.exit(1)
def printsequence(self):
def printAlgo( algName, appMgr, prefix = ' ') :
print prefix + algName
alg = appMgr.algorithm( algName.split( "/" )[ -1 ] )
prop = alg.properties()
if prop.has_key( "Members" ) :
subs = prop[ "Members" ].value()
for i in subs : printAlgo( i.strip( '"' ), appMgr, prefix + " " )
elif prop.has_key( "DetectorList" ) :
subs = prop[ "DetectorList" ].value()
for i in subs : printAlgo( algName.split( "/" )[ -1 ] + i.strip( '"' ) + "Seq", appMgr, prefix + " ")
import GaudiPython
self.g = GaudiPython.AppMgr()
mp = self.g.properties()
print "\n ****************************** Algorithm Sequence **************************** \n"
for i in mp["TopAlg"].value(): printAlgo( i, self.g )
print "\n ****************************************************************************** \n"
## Instantiate and run the application.
# Depending on the number of CPUs (ncpus) specified, it start
def run(self, ncpus = None):
if not ncpus:
# Standard sequential mode
result = self.runSerial()
else:
# Doing the checking here because it is not done in runParallel
ncpus = int(ncpus)
assert(ncpus > 0)
# Parallel mode
result = self.runParallel(ncpus)
return result
def runSerial(self) :
#--- Instantiate the ApplicationMgr------------------------------
import GaudiPython
log.debug('-'*80)
log.debug('%s: running in serial mode', __name__)
log.debug('-'*80)
sysStart = time()
self.g = GaudiPython.AppMgr()
success = self.g.run(self.g.EvtMax).isSuccess()
success = self.g.exit().isSuccess() and success
if not success:
return 1
sysTime = time()-sysStart
log.debug('-'*80)
log.debug('%s: serial system finished, time taken: %5.4fs', __name__, sysTime)
log.debug('-'*80)
return 0
def runParallel(self, ncpus) :
from Gaudi.Configuration import Configurable
import GaudiPython.Parallel as gpp
c = Configurable.allConfigurables
log.info('-'*80)
log.info('%s: running in parallel mode', __name__)
log.info('-'*80)
sysStart = time()
sc = gpp.setupSystem(ncpus, c)
if not sc:
return 1
sysTime = time()-sysStart
log.info('-'*80)
log.info('%s: parallel system finished, time taken: %5.4fs', __name__, sysTime)
log.info('-'*80)
return 0
import os, sys
__configurables_module_fullname__ = __name__ + '.Configurables'
__ignore_missing_configurables__ = False
## Small class that allows to access all the configurables as attributes of the
# instance.
# Used as module to allow code like
# @code
# from Gaudi.Configuration import Configurables
# Configurables.MyConf()
# @endcode
class _ConfigurablesModule(object):
## Initializes the instance
def __init__(self):
## If set to true, does not raise an AttributeError if the configurable is not found.
self.ignoreMissingConfigurables = False
def __getattr__(self, name):
# trigger the load of the configurables database
from Gaudi.Configuration import confDbGetConfigurable, cfgDb
# return value
retval = None
# handle the special cases (needed for modules): __all__, __path__
if name == "__all__":
retval = cfgDb.keys()
elif name == "__path__":
retval == None
elif name in cfgDb.keys(): # ignore private names