Steps to reduce DaVinci memory usage when compiling functor libraries by varying optimisation flags.

At the moment with functor libraries that reach up to e.g. 7000-8000 functors, high memory usage such as up to 12 GB is observed while compiling functors. This can be in part attributed to the high optimisation level used e.g. -O3 and flags like -pipe which disallow writing of temporary files for intermediate output storage.

DPA's needs are probably more centered around being able to run smoothly in the environments available e.g. grid with strict resource constraints in some places, while RTA's are centered around highest possible throughput (i.e. optimise as much as possible without worrying about the configure-time resources used) in a smaller farm of powerful nodes.

It seemed interesting to see what compile time flags could be used to vary the memory use and compile times, i.e. can we recover from this large memory use at the cost of some speed?

with -O0 -pipe only the functor library took 288 seconds to compile, max pss: 10056214
with -O0 --param ggc-min-expand=10 -gsplit-dwarf it took 433 seconds but max pss: 6272024
with -O1 -fno-inline --param ggc-min-expand=10 -gsplit-dwarf the functor library took 565 seconds to compile, max RSS: 6804140
with -O2 --param ggc-min-expand=10 -gsplit-dwarf the functor library took 778 seconds to compile, max pss: 6990475

A mix of PSS and RSS is used. You can think of the PSS as being just a little bit lower than the RSS numbers but still roughly representative of the usage in these tests.

I used prmon to record memory use of the whole application. the functor_jitter script replaces -O3 flag with the above tested flags. The compile time quoted above is the same time recorded in the application log.

We see that -pipe introduces a really big jump in memory, which makes sense, so we take that out and see a 2x time increase but more acceptable memory use, though no optimisation. With -O1 -fno-inline there is another jump in memory and time.

I think it would be a good idea to allow swapping of -O3 and -pipe to be replaced with flags like the above in FunctorFactory controllable from environment variables, as with the other configurables. Then someone can actually play around with this or even come up with something to mitigate some of the recent issues running on the grid.

Helpful reading: https://wiki.debian.org/ReduceBuildMemoryOverhead

functor_jitter for reference:

#!/usr/bin/env python
# Auto-generated script to create a jitter for the FunctorFactory
import os
import subprocess as sp
import sys
from multiprocessing import Pool

header = os.environ['FUNCTORFACTORY_PREPROCESSED_HEADER']

header = os.path.join(os.path.dirname(header), 'Functors/JIT_includes.h')

if len(sys.argv) != 4:
    raise Exception(
        'expect 4 arguments! e.g. functor_jitter {N_jobs} {source_directory} {output_lib_name}'
    )

n_jobs = None if sys.argv[1] == '-1' else int(sys.argv[1])
source_dir = sys.argv[2]
lib_name = sys.argv[3]
files = [fn for fn in os.listdir(source_dir) if fn.endswith('.cpp')]
os.chdir(source_dir)

# debug info is only needed for debugging or the throughput tests. Those jobs
# should set this env var otherwise if not set we explicitly force the debug
# level to zero to reduce memory and compilation time overhead of JIT by a
# factor of >2
extra_args = os.environ.get('THOR_JIT_EXTRA_ARGS', '-g0')

cmd = f'''
#!/bin/sh
export PATH=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9/bin:/cvmfs/lhcb.cern.ch/lib/lcg/releases/binutils/2.40-acaab/x86_64-el9/bin
export LD_LIBRARY_PATH=/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9/lib64
/cvmfs/lhcb.cern.ch/lib/lcg/releases/gcc/13.1.0-b3d18/x86_64-el9/bin/g++'''

my_pool = Pool(n_jobs)
return_codes = []

for file in files:
    compile_cmd_pch = cmd + ' -H -std=c++20     -DGAUDI_V20_COMPAT -DUSE_DD4HEP -DUSE_TORCH -DDD4HEP_USE_TBB -DBOOST_SPIRIT_USE_PHOENIX_V3 -DDD4HEP_USE_GEANT4_UNITS=1 -DDD4HEP_USE_XERCESC -DFMT_SHARED -DJSON_USE_IMPLICIT_CONVERSIONS=1 -DJSON_DIAGNOSTICS=0 -DBOOST_FILESYSTEM_NO_LIB -DBOOST_FILESYSTEM_DYN_LINK -DBOOST_THREAD_NO_LIB -DBOOST_THREAD_DYN_LINK -DBOOST_REGEX_NO_LIB -DBOOST_REGEX_DYN_LINK -DBOOST_SYSTEM_NO_LIB -DBOOST_SYSTEM_DYN_LINK -DBOOST_CONTAINER_NO_LIB -DBOOST_CONTAINER_DYN_LINK     -march=x86-64-v2  -fmessage-length=0 -pipe -fdiagnostics-color -pedantic -Wall -Wextra -Werror=return-type -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -Wsuggest-override -ffile-prefix-map=/scratch/s2111661/stack-el9-pch=.     -O3 -DNDEBUG -fPIC     -I/scratch/s2111661/stack-el9-pch/Rec/Phys/FunctorCore/include -I/scratch/s2111661/stack-el9-pch/Rec/Phys/DaVinciInterfaces/include -I/scratch/s2111661/stack-el9-pch/Rec/Phys/DaVinciTypes/include -I/scratch/s2111661/stack-el9-pch/Rec/Phys/DaVinciKernel/include -I/scratch/s2111661/stack-el9-pch/Rec/Phys/DaVinciMCKernel/include -I/scratch/s2111661/stack-el9-pch/Rec/Pr/PrKernel/include -I/scratch/s2111661/stack-el9-pch/Rec/Phys/SelKernel/include -I/scratch/s2111661/stack-el9-pch/Rec/Phys/SelTools/include -I/scratch/s2111661/stack-el9-pch/Rec/Tr/TrackKernel/include -I/scratch/s2111661/stack-el9-pch/Rec/Tr/TrackFitEvent/include     -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/Boost/1.82.0-fbfc9/x86_64-el9-gcc13-opt/include -isystem /scratch/s2111661/stack-el9-pch/Detector/InstallArea/x86_64_v2-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/DD4hep/01.27.02-a9b45/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/tbb/2021.10.0-2a247/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/ROOT/6.30.04-dd2db/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/XercesC/3.2.4-9e637/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/fmt/10.0.0-f6609/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/rangev3/0.11.0-79ab4/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/Vc/1.4.4-9e9f2/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/yamlcpp/0.6.3-d05b2/x86_64-el9-gcc13-opt/lib/cmake/yaml-cpp/../../../include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/yamlcpp/0.6.3-d05b2/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/jsonmcpp/3.10.5-f26c3/x86_64-el9-gcc13-opt/include -isystem /scratch/s2111661/stack-el9-pch/Gaudi/InstallArea/x86_64_v2-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/cppgsl/3.1.0-7a1b6/x86_64-el9-gcc13-opt/include -isystem /scratch/s2111661/stack-el9-pch/LHCb/InstallArea/x86_64_v2-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/AIDA/3.2.1-3fe9f/x86_64-el9-gcc13-opt/src/cpp -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/Python/3.9.12-9a1bc/x86_64-el9-gcc13-opt/include/python3.9 -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/clhep/2.4.7.1-b7a7d/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/eigen/3.4.0-6ce89/x86_64-el9-gcc13-opt/include/eigen3 -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/GSL/2.7-30ba4/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/vdt/0.4.4-260e4/x86_64-el9-gcc13-opt/include -isystem /cvmfs/lhcb.cern.ch/lib/lcg/releases/HepMC/2.06.11-d5a39/x86_64-el9-gcc13-opt/include     -include {2} {1} -c {0}'.format(file, extra_args, header)    
    # compile_cmd_pch = compile_cmd_pch.replace('-O3', '-O0 --param ggc-min-expand=10 -gsplit-dwarf')
    compile_cmd_pch = compile_cmd_pch.replace('-O3', '-O1 -fno-inline --param ggc-min-expand=10 -gsplit-dwarf')
    # compile_cmd_pch = compile_cmd_pch.replace('-O3', '-O1 --param ggc-min-expand=10 -gsplit-dwarf')
    # compile_cmd_pch = compile_cmd_pch.replace('-O3', '-O2 --param ggc-min-expand=10 -gsplit-dwarf')

    compile_cmd_pch = compile_cmd_pch.replace('-pipe', '')
    print(compile_cmd_pch)

    res = my_pool.apply_async(sp.check_call, (compile_cmd_pch, ), {'shell': True, })

    return_codes.append(res)

my_pool.close()
my_pool.join()

if not all([r.successful() for r in return_codes]):
    print('Nonzero exit codes in compilation jobs')
    exit(1)

# we know all our libs will be on the LD_LIBRARY_PATH so just point the linker there
my_env = os.environ.copy()
my_env['LIBRARY_PATH'] = my_env['LD_LIBRARY_PATH']

# include the CXX_FLAGS here again. This is for example needed when linking
# using clang as those flags contain the --gcc-tolchain= flag poiting clang to
# the gcc installation.
link_cmd = cmd + ' -march=x86-64-v2  -fmessage-length=0 -pipe -fdiagnostics-color -Wall -Wextra -Werror=return-type -Wwrite-strings -Wpointer-arith -Woverloaded-virtual -Wnon-virtual-dtor -Wsuggest-override -ffile-prefix-map=/scratch/s2111661/stack-el9-pch=. -O3 -DNDEBUG -fPIC -shared {1} -lFunctorCoreLib -lTrackEvent -lPhysEvent -lMCEvent -lRecEvent -lHltEvent -lTMVA -o {0} *.o'.format(lib_name, extra_args)

exit(sp.call(link_cmd, shell=True, env=my_env))

Rec/Phys/FunctorCore/CMakeLists.txt

###############################################################################
# (c) Copyright 2000-2024 CERN for the benefit of the LHCb Collaboration      #
#                                                                             #
# This software is distributed under the terms of the GNU General Public      #
# Licence version 3 (GPL Version 3), copied verbatim in the file "COPYING".   #
#                                                                             #
# In applying this licence, CERN does not waive the privileges and immunities #
# granted to it by virtue of its status as an Intergovernmental Organization  #
# or submit itself to any jurisdiction.                                       #
###############################################################################
#[=======================================================================[.rst:
Phys/FunctorCore
----------------
#]=======================================================================]

if(USE_DD4HEP)
  gaudi_add_library(FunctorCoreLib
    SOURCES
        src/Cache.cpp
        src/Core.cpp
        src/FunctorDesc.cpp
    LINK
        PUBLIC
            Boost::headers
            Detector::DetectorLib
            Gaudi::GaudiKernel
            Gaudi::GaudiPluginService
            LHCb::DAQEventLib
            LHCb::LbDD4hepLib
            LHCb::EventBase
            LHCb::LHCbKernel
            LHCb::LHCbMathLib
            LHCb::LoKiMCLib
            LHCb::PartPropLib
            LHCb::RecEvent
            LHCb::MCEvent
            LHCb::MCInterfaces
            LHCb::HltEvent
            LHCb::TrackEvent
            Rec::DaVinciInterfacesLib
            Rec::DaVinciKernelLib
            Rec::DaVinciMCKernelLib
            Rec::PrKernel
            Rec::SelKernelLib
            Rec::SelToolsLib
            Rec::TrackKernel
            ROOT::RIO
            ROOT::Tree
        PRIVATE
            Gaudi::GaudiAlgLib
	    )
else()
  gaudi_add_library(FunctorCoreLib
    SOURCES
        src/Cache.cpp
        src/Core.cpp
        src/FunctorDesc.cpp
    LINK
        PUBLIC
            Boost::headers
            Detector::DetectorLib
            Gaudi::GaudiKernel
            Gaudi::GaudiPluginService
            LHCb::DAQEventLib
            LHCb::EventBase
            LHCb::LHCbKernel
            LHCb::LHCbMathLib
            LHCb::LoKiMCLib
            LHCb::PartPropLib
            LHCb::RecEvent
            LHCb::MCEvent
            LHCb::MCInterfaces
            LHCb::HltEvent
            LHCb::TrackEvent
            Rec::DaVinciInterfacesLib
            Rec::DaVinciKernelLib
            Rec::DaVinciMCKernelLib
            Rec::PrKernel
            Rec::SelKernelLib
            Rec::SelToolsLib
            Rec::TrackKernel
            ROOT::RIO
            ROOT::Tree
        PRIVATE
            Gaudi::GaudiAlgLib
	    )	  
endif()
	  
gaudi_add_module(FunctorCore
    SOURCES
        src/Components/ExampleAlg.cpp
        src/Components/Factory.cpp
    LINK
        Boost::headers
        FunctorCoreLib
        Gaudi::GaudiAlgLib
        Gaudi::GaudiKernel
)

gaudi_add_executable(TestFunctors
    SOURCES
        tests/src/TestFunctors.cpp
    LINK
        Boost::unit_test_framework
        FunctorCoreLib
        LHCb::PhysEvent
        LHCb::TrackEvent
        LHCb::LHCbMathLib
        Rec::PrKernel
        Rec::SelToolsLib
    TEST
)

gaudi_add_executable(InstantiateFunctors
   SOURCES tests/src/InstantiateFunctors.cpp
   LINK
       Boost::unit_test_framework
       FunctorCoreLib
       LHCb::LoKiMCLib
       LHCb::PhysEvent
       LHCb::TrackEvent
       LHCb::LHCbMathLib
       LHCb::MCEvent
   TEST
)

# This target only exists to try and have a reliable way to figure out when to
# rebuild the preprocessed header.
# We do not use gaudi_add_executable as we don't want it installed.
add_executable(jit_includes_test src/functor_jit_dummy/test_includes.cpp)
target_link_libraries(jit_includes_test
    PRIVATE
        FunctorCoreLib
        LHCb::LinkerEvent
        LHCb::PhysEvent
        LHCb::TrackEvent
        LHCb::MCEvent
        LHCb::LHCbMathLib
)

gaudi_install(PYTHON)

gaudi_add_tests(QMTest)
gaudi_add_tests(pytest ${CMAKE_CURRENT_SOURCE_DIR}/python)


string(TOUPPER ${CMAKE_BUILD_TYPE} CMAKE_BUILD_TYPE_UPPER)

set(preprocessed_header_name "preprocessed_functorfactory_header.ii")

# When building the FunctorCache or when using a monobuild there is no install
# directory, thus we need to support running from the build directory and from
# the InstallArea. Executables like the below functor_jitter script and
# libraries are easy because the `CMAKE_CURRENT_BINARY_DIR` and the
# InstallArea are automatically in the `PATH` and `LD_LIBRARY_PATH` env
# variables. But to find the preprocessed header we need to play a small trick:
# The lhcb_env command sets an env variable for the build environment and the
# installed project, so that's what we do first and point to the InstallArea.
# Then we issue the command again but pass the PRIVATE flag which will only set
# the variable for the build directory thus overwriting the previously set env
# var for the build directory only.
lhcb_env(SET
  FUNCTORFACTORY_PREPROCESSED_HEADER
  "${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_INCLUDEDIR}/${preprocessed_header_name}"
)
lhcb_env(PRIVATE SET
  FUNCTORFACTORY_PREPROCESSED_HEADER
  "${CMAKE_CURRENT_BINARY_DIR}/${preprocessed_header_name}"
)
lhcb_env(PRIVATE SET
  FUNCTORFACTORY_STD_HEADER
  "${CMAKE_CURRENT_BINARY_DIR}/Functors/JIT_includes.h"
)

# # generate temporary file because I don't want to waste more time tyring to
# figure out how to freaking handle stupid whitespace in generator expressions
# and lists
#
# Includes are split into user (-I) and system (-isystem) according to the usual
# way our cmake splits them (only includes from the current project are -I).
# FIXME Have upstream projects use -I, see https://gitlab.cern.ch/lhcb/LHCb/-/issues/191
# FIXME In a super-project build, includes from other projects will be isystem
#       (unlike for any other compilation unit).
# FIXME for some reason, dd4hep has /usr/include in its include directories. because we
#       include these directories with -isystem, we trigger https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70129.
#       Thus, explicitly exclude /usr/include
file(GENERATE
    OUTPUT "tmp_preprocessor.sh"
    CONTENT "# auto generated
exec ${CMAKE_CXX_COMPILER} -x c++ -std=c++${GAUDI_CXX_STANDARD} \
-D$<JOIN:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:TestFunctors,COMPILE_DEFINITIONS>>, -D> \
${CMAKE_CXX_FLAGS} \
${CMAKE_CXX_FLAGS_${CMAKE_BUILD_TYPE_UPPER}} \
-I$<JOIN:$<FILTER:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,INCLUDE_DIRECTORIES>>,INCLUDE,^${PROJECT_SOURCE_DIR}>, -I> \
-isystem $<JOIN:$<FILTER:$<FILTER:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,INCLUDE_DIRECTORIES>>,EXCLUDE,/usr/include>,EXCLUDE,^${PROJECT_SOURCE_DIR}>, -isystem > \
-E ${CMAKE_CURRENT_SOURCE_DIR}/include/Functors/JIT_includes.h \
-o ${preprocessed_header_name}"
)


# generate the preprocessed header which depends on jit_includes_test
add_custom_command(OUTPUT ${preprocessed_header_name}
                   COMMAND sh tmp_preprocessor.sh
                   DEPENDS ${generated_header_name} "tmp_preprocessor.sh" jit_includes_test
                  )

install(FILES "${CMAKE_CURRENT_BINARY_DIR}/${preprocessed_header_name}" TYPE INCLUDE)


file(GENERATE
    OUTPUT "tmp_preprocessor_pch.sh"
    CONTENT "# auto generated
exec ${CMAKE_CXX_COMPILER} -x c++-header -std=c++${GAUDI_CXX_STANDARD} \
-D$<JOIN:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,COMPILE_DEFINITIONS>>, -D> \
${CMAKE_CXX_FLAGS} \
${CMAKE_CXX_FLAGS_${CMAKE_BUILD_TYPE_UPPER}} -fPIC \
-I$<JOIN:$<FILTER:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,INCLUDE_DIRECTORIES>>,INCLUDE,^${PROJECT_SOURCE_DIR}>, -I> \
-isystem $<JOIN:$<FILTER:$<FILTER:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,INCLUDE_DIRECTORIES>>,EXCLUDE,/usr/include>,EXCLUDE,^${PROJECT_SOURCE_DIR}>, -isystem > \
 ${CMAKE_CURRENT_SOURCE_DIR}/include/Functors/JIT_includes.h \
-o Functors/JIT_includes.h.gch")


add_custom_command(OUTPUT Functors/JIT_includes.h.gch
                  COMMAND sh tmp_preprocessor_pch.sh
                  DEPENDS ${generated_header_name} "tmp_preprocessor_pch.sh" jit_includes_test
                 )

install(FILES "${CMAKE_CURRENT_BINARY_DIR}/Functors/JIT_includes.h.gch"
    DESTINATION ${CMAKE_INSTALL_INCLUDEDIR}/Functors
)


# avoid "warning: style of line directive is a GCC extension" because we
# include a preprocessed header. Are there better solutions? We could first
# precompile the preprocessed header in initalize() and then use that pch...
# something for later
string(REPLACE " -pedantic" "" cxx_flags_without_pedantic ${CMAKE_CXX_FLAGS})

# the below logic assumes that `CMAKE_CXX_COMPILER` points to a compiler
# wrapper. These wrappers are created by the cmake logic defined in
# lcg-toolchains
file(READ "${CMAKE_CXX_COMPILER}" CMAKE_CXX_COMPILER_CONTENT)
string(REPLACE " \"$@\"" "" CMAKE_CXX_COMPILER_CONTENT ${CMAKE_CXX_COMPILER_CONTENT})
string(STRIP ${CMAKE_CXX_COMPILER_CONTENT} CMAKE_CXX_COMPILER_CONTENT)

# Specify the libraries a JIT compiled functor library will link against. The
# list here is defined based upon the includes present in
# `FunctorCore/include/JIT_includes.h`
set(JIT_LINK_LIBS "-lFunctorCoreLib -lTrackEvent -lPhysEvent -lMCEvent -lRecEvent -lHltEvent -lTMVA")

file(GENERATE
    OUTPUT "functor_jitter_tmp"
    CONTENT "#!/usr/bin/env python
# Auto-generated script to create a jitter for the FunctorFactory
import os
import subprocess as sp
import sys
from multiprocessing import Pool

header = os.environ['FUNCTORFACTORY_PREPROCESSED_HEADER']

header = os.path.join(os.path.dirname(header), 'Functors/JIT_includes.h')

if len(sys.argv) != 4:
    raise Exception(
        'expect 4 arguments! e.g. functor_jitter {N_jobs} {source_directory} {output_lib_name}'
    )

n_jobs = None if sys.argv[1] == '-1' else int(sys.argv[1])
source_dir = sys.argv[2]
lib_name = sys.argv[3]
files = [fn for fn in os.listdir(source_dir) if fn.endswith('.cpp')]
os.chdir(source_dir)

# debug info is only needed for debugging or the throughput tests. Those jobs
# should set this env var otherwise if not set we explicitly force the debug
# level to zero to reduce memory and compilation time overhead of JIT by a
# factor of >2
extra_args = os.environ.get('THOR_JIT_EXTRA_ARGS', '-g0')

cmd = '''
${CMAKE_CXX_COMPILER_CONTENT}'''

my_pool = Pool(n_jobs)
return_codes = []

for file in files:
    compile_cmd = cmd + ' -std=c++${GAUDI_CXX_STANDARD} \
    -D$<JOIN:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,COMPILE_DEFINITIONS>>, -D> \
    ${cxx_flags_without_pedantic} \
    ${CMAKE_CXX_FLAGS_${CMAKE_BUILD_TYPE_UPPER}} -fPIC \
    {1} -c {0}'.format(file, extra_args)


    compile_cmd_pch = cmd + ' -H -std=c++${GAUDI_CXX_STANDARD} \
    -D$<JOIN:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,COMPILE_DEFINITIONS>>, -D> \
    ${CMAKE_CXX_FLAGS} \
    ${CMAKE_CXX_FLAGS_${CMAKE_BUILD_TYPE_UPPER}} -fPIC \
    -I$<JOIN:$<FILTER:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,INCLUDE_DIRECTORIES>>,INCLUDE,^${PROJECT_SOURCE_DIR}>, -I> \
    -isystem $<JOIN:$<FILTER:$<FILTER:$<REMOVE_DUPLICATES:$<TARGET_PROPERTY:FunctorCoreLib,INCLUDE_DIRECTORIES>>,EXCLUDE,/usr/include>,EXCLUDE,^${PROJECT_SOURCE_DIR}>, -isystem > \
    -include {2} {1} -c {0}'.format(file, extra_args, header)    

    print(compile_cmd_pch)

    res = my_pool.apply_async(sp.check_call, (compile_cmd_pch, ), {'shell': True,})

    return_codes.append(res)

my_pool.close()
my_pool.join()

if not all([r.successful() for r in return_codes]):
    print('Nonzero exit codes in compilation jobs')
    exit(1)

# we know all our libs will be on the LD_LIBRARY_PATH so just point the linker there
my_env = os.environ.copy()
my_env['LIBRARY_PATH'] = my_env['LD_LIBRARY_PATH']

# include the CXX_FLAGS here again. This is for example needed when linking
# using clang as those flags contain the --gcc-tolchain= flag poiting clang to
# the gcc installation.
link_cmd = cmd + ' ${cxx_flags_without_pedantic} ${CMAKE_CXX_FLAGS_${CMAKE_BUILD_TYPE_UPPER}} \
-fPIC -shared {1} ${JIT_LINK_LIBS} -o {0} *.o'.format(lib_name, extra_args)

exit(sp.call(link_cmd, shell=True, env=my_env))
")

# we don't yet have cmake 3.20 so file(GENERATE) doesn't accept permissions yet
# thus we add a proxy command that copies the generated file and makes it
# executable
add_custom_command(OUTPUT "functor_jitter" DEPENDS "functor_jitter_tmp"
  COMMAND cp "functor_jitter_tmp" "functor_jitter"
  COMMAND chmod a+x "functor_jitter"
)


install(PROGRAMS "${CMAKE_CURRENT_BINARY_DIR}/functor_jitter" TYPE BIN)

add_custom_target(FunctorCoreJit ALL DEPENDS ${preprocessed_header_name} "Functors/JIT_includes.h.gch" "functor_jitter")
# this is only here to handle dependencies of a FunctorCache outside Rec e.g.
# in Moore TODO this is technically a hack since `FunctorCoreJit` is only a
# runtime dependency but at the moment, there is no appropriate target for
# runtime dependencies to which `FunctorCoreJit` should be added.
add_dependencies(FunctorCore FunctorCoreJit)

cc @cburr

relates to #175 (closed)

Edited May 17, 2024 by Ryunosuke O'Neil