Skip to content

NXCALS-2029 split into hours

Kamil Krzysztof Krynicki requested to merge feature-NXCALS-2029-bucket-hdfs-into-hours into develop

#Merge request template @acc-logging-team - please have a look for approval.

features:

  • data is now bucketed by in staging
  • compaction jobs can handle bucketed and unbucketed data
  • compactor will not realize we did the old 'switcheroo'
  • new passing more data via CompactionJobData and limiting the assumptions SparkCompactor makes

minor

  • significant speedup on compaction job creation (x10)
  • migrated out of joda-time on gobblin

Closes NXCALS-2029

Edited by Kamil Krzysztof Krynicki

Merge request reports