Skip to content
Snippets Groups Projects

Redesign of job options management

Merged Marco Clemencic requested to merge clemenci/Gaudi:joboptssvc-redesign into master
  1. Jul 23, 2020
  2. Jul 22, 2020
  3. Jul 07, 2020
    • Marco Clemencic's avatar
      Merge branch 'master' of https://gitlab.cern.ch/gaudi/Gaudi into joboptssvc-redesign · cb1bdede
      Marco Clemencic authored
      * 'master' of https://gitlab.cern.ch/gaudi/Gaudi: (47 commits)
        Update change log
        Add retry option to the ci-job tests
        Update instructionsetLevel for VectorClass 2.01.02
        Update LCG version to 97a
        Bump version to v33r2
        Added a test conditions service, a stalling conditions alg job option, and the debug info
        Adapt to replacement of libPyROOT
        Workaround for ROOT-10770
        Workaround for ROOT-10769
        Link GaudiExamples to ROOTHist for ROOT::RHist (ROOT >= 6.20)
        Explicitly include TObjString.h
        use LCG_97python3 in CI tests
        Fix deprecation warning from py37
        Fix propagation of StopIteration exception
        Update HiveDataBroker to use C++20 ranges if available.
        Fixes for c++20
        Fix thread-safety problem in GaudiHandle that was observed to cause a rare crash in Athena.
        Add ubsan suppressions for listcomponents and genconf
        Add some more leak sanitizer suppressions
        Tweak undefined behaviour sanitizer suppression file for newer gcc versions
        ...
      
      Conflicts:
      	GaudiAlg/src/lib/GaudiCommon.icpp
      	GaudiCoreSvc/src/IncidentSvc/DataOnDemandSvc.cpp
      	GaudiExamples/tests/qmtest/refs/Aida2Root.ref
      	GaudiKernel/GaudiKernel/Property.h
      cb1bdede
    • Marco Clemencic's avatar
      Fix tests · 91367507
      Marco Clemencic authored
      91367507
    • Marco Clemencic's avatar
      Optimize case insensitive fallback · 6ef8abd8
      Marco Clemencic authored
      6ef8abd8
    • Marco Clemencic's avatar
      a8a9861b
  4. May 20, 2020
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · 113f5ca1
      Marco Clemencic authored
      * master: (35 commits)
        SequenceSemantics: Allow explicilty setting payload semantics, introduces __eq__ and __ne__ for _ListHelper
        clean up headers in ThreadPoolSvc
        Fix bugs in ThreadPoolSvc:
        compactify if-statement condition and remove std::move in AlgTask.h
        add state update to semi-serial mode of execution
        switching to isFailure() in AlgTask.h
        remove unnecessary #include
        Fix compatibility with gsl 3
        Fixed formatting
        GaudiConfig2: Add toStringProperty() method, for compatiblity with GaudiHandles
        Remove now unused ATLAS_GAUDI_V21 and HAVE_GAUDI_PLUGINSVC definitions
        Fixed formatting
        GaudiConfig2 semantics: Add update() method to _DictHelper. Main purpose is to set is_diry to true. Required to assign an empty dict to a property
        GaudiConfig2: Add getFullJobOptName() method, required by GaudiHandles (and useful in other cases)
        add INFO message about task scheduling settings, and clean up messages in AlgTask
        rename AlgoExecutionTask to AlgTask
        remove move semantics from AvalancheScheduler::signoff, which was added for symmetry with other facilities but is actually an overkill
        fix ref and option files to reflect renaming of properties in Algorithm and AvalancheSchedulerSvc
        Fixed formatting
        extend the category of tasks that can be scheduled preemptively to 'CPU-blocking' from just 'I/O-bound' (terminological change only)
        ...
      113f5ca1
  5. Apr 30, 2020
  6. Apr 29, 2020
    • Marco Clemencic's avatar
      Merge remote-tracking branch 'upstream/master' into joboptssvc-redesign · b0390140
      Marco Clemencic authored
      * upstream/master:
        Resolve template dependencies in SerializeSTL
        fix deprecated copy constructor warning in StreamBuffer::ContainedLink
        remove unused capture of `this` in MultiTransformerFilter
        GaudiProject: preserve permissions when installing cmake folder
        get_host_binary_tag: do not silently pass if command fails
        get_host_binary_tag: fix regexp matching end of string
        AlgExecStateSvc: require minimum state size to be 1
        make gitlab use cpp syntax highlighting for icpp files
        IncidentSvc: improve error reporting
        suggestions
        fix range in vector_of_const_ and make a test for it
        this works better
        deref instead of operator* to enable gaudi ranges in mergingtransformers
        remove mention of deprecated tbb::recursive_mutex
        Update change log
        Update Gaudi and LCG versions
        Fix minor doc issue in Dockerfile
        Make formatting check more stable and fast
        Removed FindXercesC now that new versions of Xerces are integrated with cmake
        do not run public headers build test on generated headers
      b0390140
  7. Mar 13, 2020
  8. Mar 12, 2020
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · 950dda26
      Marco Clemencic authored
      * master:
        Fix special case in GaudiAlg/Tuple.h
        Drop the WARN_UNUSED macro
        Fix some ignored StatusCodes with proper handling
        Hide "unchecked" StatusCodes
        Declare StausCode as [[nodiscard]] (C++17)
      
      Conflicts:
      	GaudiCommonSvc/src/PersistencySvc/EvtCollectionStream.cpp
      	GaudiCoreSvc/src/JobOptionsSvc/JobOptionsSvc.cpp
      	GaudiCoreSvc/src/JobOptionsSvc/PythonConfig.h
      	GaudiExamples/src/IO/ReadAlg.cpp
      	GaudiKernel/src/Lib/Application.cpp
      	GaudiKernel/src/Util/genconf.cpp
      	GaudiKernel/tests/src/test_PropertyHolder.cpp
      950dda26
  9. Mar 04, 2020
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · 6701af0b
      Marco Clemencic authored
      * master: (22 commits)
        Fix propagation of Context and RootInTES from GaudiSequencer to members
        Add test to expose non-propagation of GaudiSequencer RootInTES
        fixup some type something about DataObjectHandle payload
        Make it compatible with older versions of rangesv3
        Fixed formatting
        Get rid of some warnings
        Applied Marco's suggestions
        Apply suggestion to GaudiKernel/src/Lib/ChronoEntity.cpp
        Avoid explicit call to `EnableAutoLoading` for ROOT > 6.18
        Fixed formatting
        Avoid using project(Gaudi)
        A number of minor fixes to build on MacOS
        support DataObjectHandle<NamedRange<T>>
        Improve memory management in ParticlePropertySvc
        a few more scoped_lock
        AlgExecStateSvc: Remove a fatal message and minor cleanups
        Add unit test for Gaudi::Allocator::Arena and Gaudi::Arena::Monotonic.
        Modify EvtStoreSvc to assume+check that the store is empty in setRoot instead of calling clearStore.
        Migrate EvtStoreSvc to use Gaudi::Allocator::MonotonicArena<T> with an internally-managed memory resource.
        Add Gaudi::Arena::Monotonic<Alignment, UpstreamAllocator> memory resource and Gaudi::Allocator::MonotonicArena<T, Alignment> helper typedef.
        ...
      6701af0b
  10. Feb 25, 2020
  11. Feb 02, 2020
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · aeaf5ecb
      Marco Clemencic authored
      * master: (25 commits)
        prefer std::scoped_lock
        Define a module __spec__ for Configurables
        clang requires -ffp-contract=fast with -mfma to actually enable FMA instructions
        add some missing `filter_evtcontext`
        allow legacy algorithms to work with LHCb condition handles
        genconf: Use 'PublicToolHandle', 'PrivateToolHandle' and 'ServiceHandle' instead of GaudiHandleBase (same for handle-arrays)
        Change argument type to std::string_view to avoid dynamic allocation.
        Add options to EvtStoreSvc to restrict registering addresses
        deprecate use of AnyDataHandle
        Move PyBuffer_FromReadWriteMemory helper into GaudiPython::Helper struct.
        Add test for Cling warnings when using GaudiPython::Helper struct.
        Fixed formatting
        more const std::string& -> std::string_view, std::string conversions
        more const std::string& -> std::string_view, std::string conversions
        more const std::string& -> std::string_view, std::string conversions
        more const std::string& -> std::string_view, std::string conversions
        more const std::string& -> std::string_view or std:string+std::move conversions
        even more const std::string& -> std::string_view conversions
        Fixed formatting
        more const std::string& -> std::string_view conversions
        ...
      aeaf5ecb
  12. Dec 23, 2019
  13. Dec 22, 2019
    • Marco Clemencic's avatar
      58afb1cb
    • Marco Clemencic's avatar
      Merge remote-tracking branch 'upstream/master' into joboptssvc-redesign · 529d3a28
      Marco Clemencic authored
      * upstream/master: (38 commits)
        genconf: Fixing the handling of NAN
        Updated version and change log
        Fixed GaudiPluginService copyright and license
        fix quoting in genconf
        Add copyright and license
        Add Doxygen comments to Histrogram counters
        Add detection of strongly connected components in the data flow precedence rules
        Ignore unicode decode errors in test stdout
        Remove unused (obsolete) option file
        Add experimental support for histograms as counters
        Enable sse 4.2 by default for clang 6.0 and newer as well
        Better debug message for stall on conditions
        add data flow ambiguity finder (enabled by default): a DO declared to be produced by multiple algorithms terminates the job at initialization
        Introduce new configuration Python backend
        Forgot special case in !1009
        Transformer.h - Adjust clang warning supression pragmas
        fix issue #105
        Use latest CMake from lcg contrib
        Add a gcc8 build until we drop support for it
        Move Python3 builds to LCG 96b and gcc9
        ...
      529d3a28
  14. Dec 06, 2019
  15. Dec 05, 2019
  16. Dec 04, 2019
  17. Nov 21, 2019
  18. Nov 20, 2019
  19. Nov 19, 2019
  20. Nov 17, 2019
  21. Nov 14, 2019
  22. Nov 12, 2019
  23. Nov 08, 2019
  24. Nov 07, 2019
    • Marco Clemencic's avatar
      add more b/w compatibility checks · 255623b6
      Marco Clemencic authored
      255623b6
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · 3fd0805f
      Marco Clemencic authored
      * master: (86 commits)
        Use StatusCode extension to return filter passed or failed from functional algorithms
        Standalone Fixes, master branch (2019.10.09.)
        Fixes for "defects" detected by Coverity
        Use xenv 1.0.0 if not available from the system
        ATLAS now has const correct ToolHandles
        Add missing release notes
        Update change log
        Updated VTune to 2019
        Update version to v32r2 and LCG to 96b
        Backward compatible interface of IAlgorithm for GaudiPython
        Make sure we get consistent version of Python from LCG
        GaudiKernel instructionsetLevel : Add additional AVX512 levels
        Various ToolHandleArray bug fixes
        FunctionalDetails.h - Better workaround for range v3 view->views namespace change
        Fix ccache caching for build:gcc8:opt:python3
        Improve test of Functional input/output types in Python
        Make produce_consume test stable across version of Python
        Better handling of unicodes in GaudiTesting
        GaudiMP: allow non KeyedContainer objects in FSR (ignored)
        non-reentrant interfaces for GaudiPython
        ...
      3fd0805f
  25. May 31, 2019
  26. May 30, 2019
  27. May 27, 2019
  28. May 26, 2019
    • Marco Clemencic's avatar
      Merge remote-tracking branch 'upstream/master' into joboptssvc-redesign · 65259680
      Marco Clemencic authored
      * upstream/master:
        GaudiKernel : Define LIKELY macros for clang
        GaudiKernel - Check LIKELY macros before defining them.
        Fix IOpaqueAddress lifetime in EvtStoreSvc
      65259680
    • Marco Clemencic's avatar
      1f16cdfc
    • Marco Clemencic's avatar
      Merge tag 'v32r0' into joboptssvc-redesign · 7a529914
      Marco Clemencic authored
      * tag 'v32r0': (144 commits)
        Fix propagation of RootInTES from parents to children
        Do not use tempraries in Gaudi::Utils::AlgContext
        Update change log
        Update clang format to 8
        Factor out TES path related code out of GaudiCommon
        small C++17 adaptations
        HiveDataBoker: Improve error message when unknown requested input
        Example of custom application steering class from Python
        Drop Python based initialization in gaudirun.py (serial)
        Drop support for GaudiPython initialization in gaudirun.py
        Remove support for custom main loop in gaudirun.py
        Refactor --printsequence logic
        Modify gaudirun to use Gaudi::Application
        Add possibility of overriding Gaudi::Application::run with a callable
        Modified AsyncEventProcessor test to use Gaudi.exe
        Modified GaudiMain.cpp to use Gaudi::Application
        Add a dedicatd class for application steering
        Add example implementation of IAsyncEventProcessor
        Add optional extension of EventProcessor for asynchronous processing
        Fix timing unit in Brunel scheduler scenario
        ...
      7a529914
  29. Mar 06, 2019
    • Marco Clemencic's avatar
      Distinguish between setProperty and setPropertyRepr · 78388ba7
      Marco Clemencic authored
      - moved setProperty(name, value) from Propertyholder to IProperty
      - renamed setProperty(name, repr) to setPropertyRepr(name, repr)
      
      this is needed to be able to tell is a string is meant to be a value
      or the represenataion of a value
      78388ba7
  30. Mar 05, 2019
  31. Feb 16, 2019
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · 67440acf
      Marco Clemencic authored
      * master:
        Update version of CMake used in CI jobs to 3.13.4
        Update change log and release notes
        Use gcc 8 (instead of gcc 7) and drop gcc 6.2 in CI jobs
        Minor clean up of .gitignore
        Print a configuration time warning if clang-format or yapf are not found
        Update Gaudi version to v31r0 and LCG to 95
        Update .gitlab-ci.yml
        Made the logic in GaudiProjectConfig.cmake compatible with the GAUDI_ATLAS=TRUE mode.
        Make gaudi_add_compile_test also work for succeeding tests
        update chronoentity (see #60)
        introduced CPUCrunchSvc
        Applied the suggested clang formatting changes.
        Fix lookup/use of nosetests for LCG 95
        Defined the StatusCode constant values explicitly.
      67440acf
  32. Feb 02, 2019
    • Marco Clemencic's avatar
      Merge branch 'master' into joboptssvc-redesign · c7ba90b4
      Marco Clemencic authored
      * master:
        Restore mutex to protect GaudiCommon::m_countersOwn
        Fixed formatting
        Workaround for https://bugs.llvm.org/show_bug.cgi?id=35641
        remove pointless comment after last "#endif"
        Run YAPF on Python files without .py suffix (scripts)
        Minor usability improvement to check-formatting
        Run re-formatting in parallel
        Simplify check-formatting CI
        Upgrade to clang-format-7
        Add .icc to the C++ sources list
        Opt for a more compact formatting
        Use yapf 0.24.0 to reformat Python files
        Indent preprocessor directives
        Update format style to clang 6.0
        Force use of clang-format 6.0
      c7ba90b4
    • Marco Clemencic's avatar
      Merge commit 'f1750312' into joboptssvc-redesign · e1b0cc9c
      Marco Clemencic authored
      * commit 'f1750312':
        Make Counters work with Gaudi::Algorithm
        transformer now filters out eventcontexts
        Added a boolean parameter to turn on verbose view algorithm status
        Fixed formatting
        make DataObjectHandle::{put,getOrCreate} const
        Added lock_guards to make the filling of histograms thread safe
        add unit test for EventIDRange intersections
        Fix EventIDBase::max() to properly handle mixtures of Run/LB and TimeStamps
        fix parameter-ordering problem in the tuple-based constructor of EventIDBase
        fix EventIDRange.isInRange
        EventIDBase/Range: Allow ranges limited by both timestamps and Run/LB boundaries
        add some switch defaults + one useless statement
      e1b0cc9c
  33. Jan 23, 2019
  34. Jan 22, 2019
Loading