Skip to content

Resolve NXCALS-6139 "Implement adaptive partitioning algorithm in compactor"

Jakub Wozniak requested to merge NXCALS-6139-Implement-adaptive-partitioning-algorithm-in-compactor into develop

Merge request template

COMPATIBILITY NOTE

Please make sure that this change is BACKWARDS COMPATIBLE and a new server will work correctly with clients using old APIs versions.

It may be that data sent in the new format will break the old clients.

In case of any potential incompatibility we have to inform our clients and schedule release & deployment with them.

@acc-logging-team - please have a look for approval.

Uses Adaptive Partitioning in Compactor. This effectively replaces Compaction jobs by AdaptiveCompaction jobs. Compactor from now on will create files that are split by time partitions & entity buckets. The number of splits depends on the size of the staging day directory. The time partition & entity bucket are stored in the output (in data) file name so they can be later on used to prune the files that do not contain the query data (please see the test class).

Lots of renames, but those are automatic and can be ignored.

Most important classes/files to be reviewed:

  • compaction/src/main/java/cern/nxcals/compaction/processor/AdaptiveCompactProcessor.java
  • compaction/src/test/java/cern/nxcals/compaction/processor/AdaptiveCompactProcessorTest.java
  • compaction/src/main/java/cern/nxcals/compaction/service/Compactor.java
  • service/src/main/java/cern/nxcals/service/internal/compaction/AdaptiveCompactionJobCreator.java
  • service/src/main/java/cern/nxcals/service/internal/CompactionService.java

Renames but more significant as done on DB:

  • service-db/src/main/resources/sql/latest/triggers/NXCALS-5854-create-partition-resource-infos-changelog-trigger.xml
  • service-db/src/main/resources/sql/versions/NXCALS-6383/NXCALS-6383-change-partition-infos-into-partition-hists.xml

Closes NXCALS-6139

Edited by Jakub Wozniak

Merge request reports