Draft: Run Luigi task directly from Python instead of subprocess
Merge request reports
Activity
assigned to @clemenci
8 8 # granted to it by virtue of its status as an Intergovernmental Organization # 9 9 # or submit itself to any jurisdiction. # 10 10 ############################################################################### 11 from subprocess import run 11 import luigi 12 from lb.nightly.scheduler.workflow import Slot - Comment on lines +11 to +12
I would move these two lines into the
start_slotfunction with a check to nicely report thatstart_slotcan be used only ifschedulerandluigiare available (instead of anImportError).In this way, I believe, we should be able to delay invoke the
start_slotfunction even if we do not havescheduleravailable. changed this line in version 3 of the diff
added 1 commit
- 97d771ef - move imports inside task function and prevent ImportError if unavailable
added 1 commit
- db86f25d - send logs from luigi call of start_slot task to the file
added 17 commits
-
c3315dd0...42c383fa - 11 commits from branch
master - 35dd5770 - Run Luigi task directly from Python instead of subprocess
- 82126e09 - add lb-nightly-scheduler to ci job
- 304f1527 - move imports inside task function and prevent ImportError if unavailable
- b8b867fc - send logs from luigi call of start_slot task to the file
- 0f705ebd - fix f-string construction
- c36e80b0 - Reduce log level in luigi call from DEBUG to INFO
Toggle commit list-
c3315dd0...42c383fa - 11 commits from branch
Unfortunately, we can't really use this approach because of
AssertionError: daemonic processes are not allowed to have children. AFAIU, theceleryworker is running as a daemon, andluigi.build()usesmultiprocessing(when number of workers is>1). I'm not sure if workarounds for this are worth the effort and we may just stick with runningluigiinsubprocess.mentioned in issue #14