Skip to content

Support multiple dataframe in preprocessing

Anthony Correia requested to merge anthonyc/processing_with_multi_dataframes into main

Here are the changes implemented by this MR

  • The preprocessing was cleaned up (I forgot to remove a few things in the previous branch)
  • A new subdirs parameter in the preprocessing configuration, that can be used to have several dataframes as input
  • A preprocessing step that loops over the input dataframes.
  • A preprocessing step that uses groupby to loop over events, which is faster than filtering the events at every iteration
  • A new "sorted-wise edges" that links the hits by sorting the hits by increasing z
  • An updated "plane-wise edges" that links the hits that belong to adjacent planes within a particle
  • A new training that is being slowly set up (track-weighting)

I tried to implement some multi-processing in the preprocessing and but seemed to be actually slower so I gave up on the idea.

Edited by Fotis Giasemis

Merge request reports