Skip to content

Use singularity directly instead of its Python API

Maciej Pawel Szymanski requested to merge maszyman-async-sub-dev into master

This MR drops the usage of spython. Instead, I use singularity directly through an asynchronous subprocess. This way, I'm able to get stdout and stderr. Using spython we don't get stderr (I was using contextlib.redirect_stderr hack to pass stderr to build_logs, see !65 (merged), but that did not solve the problem of reading from stderr what lb_wrapcmd writes to), and what's worse we get the deadlock when the size of stderr exceeds the buffer size (typically 64 kB). The buffer limit for StreamReader wrappers used by async subprocess is also 64 kB, but the limit argument allows for increasing that (and it raises asyncio.exceptions.LimitOverrunError instead of deadlock). I set it to 128 kB, which seems to be fine for Geant4 output (which was a trigger for this MR), but we may fine-tune it in the future.

Since I was touching on collecting the logs in this MR, I took the opportunity to drop sending the logs directly to OpenSearch (we agreed to go for a fluent-bit service) and change the format of the logs so that each log line is sent to file as JSON with some metadata added.

If we go for this MR, !98 (closed) and !89 (closed) can be closed.

Edited by Maciej Pawel Szymanski

Merge request reports