Skip to content

NXCALS-1765 gobblin cleanup

#Merge request template @acc-logging-team - please have a look for approval.

Please take an especially careful look on my ansible changes. I really wanted to work within the given structure, but you never know...

features

  • every etl instance, prior to launching, now will check if it has directories to clean (param clean.directories)
  • if so, it will recursively remove all the empty directories starting from the given roots (potentially removing even the root itself)

minor

  • utility class DirectoryIterator, which discovers all the subdirectories starting from a given root, skipping if the content triggers the predicate (responding to additions and removals)
  • utility class HdfsCleaner, which removes all the directories given by an iterator, measures the removal time and size

Closes NXCALS-1765

Edited by Kamil Krzysztof Krynicki

Merge request reports

Loading