Skip to content

NXCALS-7851 Limit number of IO operations on HDFS on closing app

Rafal Mucha requested to merge NXCALS-7851-limit-number-of-io-operations-with-hdfs-on-close into develop

Reasoning - during closing HDFS ETL, all opened files from cache are flushed and closed. These operations are executed on default thread fork join pool, which takes as limit number of processor/parallelism. Secondly - our ETLs are restarted every five days, what cause extra load, when all are restarted in the same time, flushing and closing files.

@acc-logging-team - please have a look for approval.

Please describe the merge request here.

Closes NXCALS-7851

Edited by Rafal Mucha

Merge request reports