Security Scan Instability
This is a continuation of the series of issues seen from Fermilab security scans cause crashes to xdaq Applications (other issues were #4484 (closed) and #4492).
Below is what I was able to capture at the time it happened in case it helps to find the instability's cause.
'''Here is the xdaq output at time of crash:''' {{{ 14 Feb 2018 19:00:31.956 [140271505876736] INFO gov.fnal.ftbf-daq-08.p:2000.hyperdaq::Application.lid(3) <> - Exiting process with code 10
14 Feb 2018 19:00:50.418 [140477751670528] INFO gov.fnal.ftbf-daq-08.p:2003.hyperdaq::Application.lid(3) <> - Exiting process with code 10 14 Feb 2018 19:00:52.005 [140477751670528] ERROR gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - Caught exception: xgi::exception::Exception 'Missing ClassName element' raised at uploadApplication(/usr/local/src/xdaq/baseline13/trunk/daq/hyperdaq/src/common/Application.cc:1900) 14 Feb 2018 19:01:20.857 [140477751670528] WARN gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - No callback method found for incoming request [view/TWiki/WebHome] in hyperdaq::Application 14 Feb 2018 19:01:20.874 [140477751670528] WARN gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - No callback method found for incoming request [view/TWiki/WebHome] in hyperdaq::Application 14 Feb 2018 19:01:20.893 [140477751670528] WARN gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - No callback method found for incoming request [view/TWiki/WebHome] in hyperdaq::Application 14 Feb 2018 19:01:21.265 [140477751670528] WARN gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - No callback method found for incoming request [mrtg.cgi] in hyperdaq::Application 14 Feb 2018 19:01:21.272 [140477751670528] WARN gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - No callback method found for incoming request [mrtg.cgi] in hyperdaq::Application 14 Feb 2018 19:01:21.280 [140477751670528] WARN gov.fnal.ftbf-daq-08.p:2003.executive::Application.lid(0).xdaq.XgiDispatcher <> - No callback method found for incoming request [mrtg.cgi] in hyperdaq::Application }}}
'''Here is the gdb 'where' dump:''' {{{
0 0x00000038bf67a1e8 in _int_malloc () from /lib64/libc.so.6 1 0x00000038bf67aaac in malloc () from /lib64/libc.so.6 2 0x00007f9386b8e888 in operator new(unsigned long) () at ../../.././libstdc++-v3/libsupc++/new_op.cc:49 3 0x00007f93890099da in std::_Rb_tree<std::basic_string<char, std::char_traits, std::allocator >, std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > >, std::_Select1st<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > >, std::less<std::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > > >::_M_copy(std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > > const*, std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > >) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libxdaq.so 4 0x00007f93890099ad in std::_Rb_tree<std::basic_string<char, std::char_traits, std::allocator >, std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > >, std::_Select1st<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > >, std::less<std::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > > >::_M_copy(std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > > const, std::_Rb_tree_node<std::pair<std::basic_string<char, std::char_traits, std::allocator > const, std::basic_string<char, std::char_traits, std::allocator > > >) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libxdaq.so 5 0x00007f9389009f02 in std::vector<xcept::ExceptionInformation, std::allocatorxcept::ExceptionInformation >::_M_insert_aux(__gnu_cxx::__normal_iterator<xcept::ExceptionInformation, std::vector<xcept::ExceptionInformation, std::allocatorxcept::ExceptionInformation > >, xcept::ExceptionInformation const&) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libxdaq.so 6 0x00007f9387c1f722 in xcept::Exception::Exception(std::basic_string<char, std::char_traits, std::allocator > const&, std::basic_string<char, std::char_traits, std::allocator > const&, std::basic_string<char, std::char_traits, std::allocator > const&, int, std::basic_string<char, std::char_traits, std::allocator > const&) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libxcept.so 7 0x00007f9388fc5ea0 in xdaq::ApplicationDescriptorFactory::getZone(std::basic_string<char, std::char_traits, std::allocator > const&) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libxdaq.so 8 0x00007f9388fd072c in xdaq::ApplicationContextImpl::getZone(std::basic_string<char, std::char_traits, std::allocator > const&) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libxdaq.so 9 0x00007f9383317fae in hyperdaq::Application::viewGroup(xgi::Input*, xgi::Output*) () from //home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libhyperdaq.so 10 0x00007f9383331269 in xgi::Methodhyperdaq::Application::invoke(xgi::Input*, xgi::Output*) () from //home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libhyperdaq.so 11 0x00007f9383e69dd1 in executive::XgiDispatcher::processIncomingMessage(xgi::Input*, xgi::Output*) () from //home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libexecutive.so 12 0x00007f93839bc410 in pt::http::ReceiverLoop::onRequest(std::basic_string<char, std::char_traits, std::allocator > const&, std::basic_string<char, std::char_traits, std::allocator > const&) () from //home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libpthttp.so 13 0x00007f93839c244d in pt::http::ReceiverLoop::process(toolbox::task::WorkLoop*) () from //home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libpthttp.so 14 0x00007f93888f4739 in toolbox::task::WaitingWorkLoop::process() () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libtoolbox.so 15 0x00007f93888f0cca in toolbox::task::WorkLoop::run() () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libtoolbox.so 16 0x00007f93888f1107 in toolbox::task::thread_func(void*) () from /home/otsdaq/otsdaq/products/xdaq/v13_4_0b/slf6.x86_64/lib/libtoolbox.so 17 0x00000038bfa07aa1 in start_thread () from /lib64/libpthread.so.0 18 0x00000038bf6e8bcd in clone () from /lib64/libc.so.6 }}}