Use singularity directly instead of its Python API
This MR drops the usage of spython
. Instead, I use singularity
directly through an asynchronous subprocess. This way, I'm able to get stdout and stderr. Using spython
we don't get stderr (I was using contextlib.redirect_stderr
hack to pass stderr to build_logs, see !65 (merged), but that did not solve the problem of reading from stderr what lb_wrapcmd
writes to), and what's worse we get the deadlock when the size of stderr exceeds the buffer size (typically 64 kB). The buffer limit for StreamReader
wrappers used by async subprocess
is also 64 kB, but the limit
argument allows for increasing that (and it raises asyncio.exceptions.LimitOverrunError
instead of deadlock). I set it to 128 kB, which seems to be fine for Geant4
output (which was a trigger for this MR), but we may fine-tune it in the future.
Since I was touching on collecting the logs in this MR, I took the opportunity to drop sending the logs directly to OpenSearch
(we agreed to go for a fluent-bit
service) and change the format of the logs so that each log line is sent to file as JSON
with some metadata added.
If we go for this MR, !98 (closed) and !89 (closed) can be closed.