Package              : Rich/RichMirrorAlignmentGanga
Package Coordinators : Matt Coombes, Anatoly Solomin, Paras Naik
Purpose              : RICH mirror alignment program

2016-06-07 Rich/RichMirrorAlignmentGanga v13r4p1

Adding to the git master branch, starting from lb-dev --nightly=lhcb-head Panoptes v6r0. Responding to https://its.cern.ch/jira/browse/RICH-4 Changed default string pointing to a settings file for the Online alignment to a dummy string Rich/RichMirrorAlignmentGanga should now be used for Offline alignments only Use Rich/RichMirrorAlignmentOnline for Online alignments

2016-06-07 - Paras Naik
- Changed default string pointing to a settings file for the Online alignment to a dummy string
- Rich/RichMirrorAlignmentGanga should now be used for Offline alignments only

2015-08-05 Rich/RichMirrorAlignmentGanga v13r4

Intended for PANOPTES v5r6. Fixed a problem with the ordering of Brunel job submissions in RichMirrorAlignment.py. This is critical for re-starting the alignment if it was killed for any reason mid-alignment. The ROOT version of the hadd command now depends directly on the Brunel version used for reconstruction.

2015-08-05 - Paras Naik
- Fixed a problem with the ordering of Brunel job submissions in RichMirrorAlignment.py.
  This is critical for re-starting the alignment if it was killed for any reason mid-alignment.
- The ROOT version of the hadd command now depends directly on the Brunel version used for reconstruction.
- Please use 'phiBinFactor', 'groupSlices' and 'groupBins' which have never been used have now been removed.

2015-07-10 Rich/RichMirrorAlignmentGanga v13r3

Support for new setting of RichMirrCombinFit (warningFactor) Updated Brunel Options for Brunel v47r9.

2015-07-10 - Paras Naik
- Updated Brunel Options for Brunel v47r9.
- Added support for new setting of RichMirrCombinFit called warningFactor.

2015-07-09 Rich/RichMirrorAlignmentGanga v13r2

Added phiBinFactor to the settings. This enables you to merge phi bins.

2015-07-09 - Paras Naik
- Added phiBinFactor to the settings. This enables you to merge phi bins.

2015-06-23 Rich/RichMirrorAlignmentGanga v13r1

Important update, with Several Upgrades and Bug Fixes Can now read multiple eventSelector/ioHelper input files. Re-enabled ability to have magnifCoeffMode == 0 operation. Enabled tentative Ganga 6.1 support, but this is untested since Ganga 6.1 is not out yet. Now checks that you didn't accidentally run RichMirrCombinFit before merging histos. Added scripts/XMLcompare.py. Takes any two XML files and compares all Y and Z mirror tilts. Added files/2015OfflineDBsets, a repository for .db slices that we should be including in our 2015 Offline Alignments

2015-06-23 - Paras Naik
- Important update, with Several Upgrades and Bug Fixes
- Can now read multiple eventSelector/ioHelper input files.
- Re-enabled ability to have magnifCoeffMode == 0 operation.
- Enabled tentative Ganga 6.1 support, but this is untested
  since Ganga 6.1 is not out yet.
- Now checks that you didn't accidentally run RichMirrCombinFit
  before merging histos.

2015-06-14 - Paras Naik
- Added scripts/XMLcompare.py. Takes any two XML files and compares
  all Y and Z mirror tilts. RichMirrAlign already produces
  similar printouts, but this script allows you to survey
  the differences with greater precision.
  Use python XMLcompare.py -h for help
- Added files/2015OfflineDBsets, a repository for .db slices
  that we should be including in our 2015 Offline Alignments

2015-06-14 Rich/RichMirrorAlignmentGanga v13r0

This version was expected to be released in PANOPTES v5r6. Added support for the option 'stopTolerance' in the settings file. RichMirrAlign now will say the alignment has converged if updated compensations to all mirror tilts from the last iteration are less than stopTolerance*mrad. Re-enabled the ability to skip the first Reco , Merge, or CombinFit of any iteration (instead of just for iteration 0). Added 2015 Brunel Options to be used with Brunel v47r5 or earlier. Added hadd merger, which keeps trying to merge the histos until they merge successfully.

2015-06-13 - Paras Naik
- Added support for the option 'stopTolerance' in the settings file.
  RichMirrAlign now will say the alignment has converged if updated
  compensations to all mirror tilts from the last iteration are
  less than stopTolerance*mrad.
- Moved j.do_auto_resubmit=True to before 'print j' so that you
  can see the status of do_auto_resubmit when it prints the job.
- Added a directory under files called HPD_xml_zeros. Here there
  are dummy XML files (all zeros)). We no longer do the HPD alignment,
  but we still will need to provide these dummy XMLs.

2015-06-12 - Paras Naik
- Re-enabled the ability to skip the first Reco , Merge, or CombinFit
  of any iteration (instead of just for iteration 0).
- stop_or_continue decision = STOP_NAN now occurs not only on nan
  but also -inf and inf
- updated 2015 settings example to have minAverageBinPop be 2.0
  instead of 6.0 for 2015 runnings over calibration data

2015-06-11 - Paras Naik
- Added 2015 Brunel Options to be used with Brunel v47r5 or earlier.
  Updated settings file for 2015 example.
  (Code from Claire Prouve)
- From now on, please specify the brunel and LHCb version in the
  settings file. If you leave it blank it will use the newest version.

2015-06-09 - Paras Naik
- Added hadd merger, which keeps trying to merge the histos until they merge
  successfully. (Code from Claire Prouve)
- Changes to the Online part of RichMirrorAlignmentGanga, to obtain the
  Analyzer working directory

2015-06-09 Rich/RichMirrorAlignmentGanga v12r5

This version expected to be released in PANOPTES v5r5. Stopped producing useless rubbish magnification factor files. Introducing RichMirrorAlignment.py, a python script that can run BOTH the Online and Offline mirror alignments. This is our current and future driver script. RichMirrorAlignmentGanga.py is still supported, but support may decrease in the future. Allow the possibility of using a specific mirror alignment .xml file as the zeroth iteration XML file.

2015-06-09 - Anatoly Solomin
 - in job/RichMirrorAlignment.py stopped producing useless rubbish
   magnification factor files whose origin was due to a bug.
 - Formatted numbers in magnification factor files for easier
   visual examination.

2015-05-18 - Paras Naik
 - Introducing job/RichMirrorAlignment.py, a python script that can
   run BOTH the Online and Offline mirror alignments.
   Please note that the Online thread of the script is not yet fully
   operational. However it is believed after several tests that the
   Offline thread in the script has equivalent functionality to
   job/RichMirrorAlignmentGanga.py .

2015-04-23 - Paras Naik
 - Allow the possibility of using a specific mirror alignment .xml
   file as the zeroth iteration XML file. Thus we can take the final
   .xml file from a previous alignment and use it as the starting
   point for the next one.

2015-04-23 Rich/RichMirrorAlignmentGanga v12r3

This version expected to be released in PANOPTES v5r4. Now we submit grid jobs in parallel, up to a factor of 9 increase in speed. Updated test and example settings files.

2015-04-23 - Paras Naik
 - Changed the python driver script to submit jobs in parallel
   instead of in series (new feature of Ganga 6).
   To achieve maximal benefit, the user MUST explicitly specify
   'NumWorkerThreads = 9' in their ~/.gangarc file after BOTH of the
   commented '#NumWorkerThreads' lines already in the gangarc file.
 - Provide 5 minute updates while the queues are still submitting jobs
 - Provide 15 minute updates while waiting for 80% of subjobs
   to come back from all jobs.
 - Also updated test and example settings files.

2015-04-15 Rich/RichMirrorAlignmentGanga v12r2

This version had been expected to be released in PANOPTES v5r4. No change from previous version, except proper documentation.

2015-04-15 Rich/RichMirrorAlignmentGanga v12r1

Created and added different Brunel options for Rich1 and Rich2 for 2010 DST, 2011 RAW, and 2012 RAW data. We now only reconstruct photons from the Rich detector we care about in the current alignment, not both Rich detectors. Rich1 RAW now reconstructs only > 20 GeV tracks, Rich2 RAW reconstructs only >40 GeV tracks.

2015-04-15 - Paras Naik - Created and added different Brunel options for Rich1 and Rich2 for 2010 DST, 2011 RAW, and 2012 RAW data. - We now only reconstruct photons from the Rich detector we care about in the current alignment, not both Rich detectors. - Rich1 RAW now reconstructs only > 20 GeV tracks, Rich2 RAW reconstructs only >40 GeV tracks. - Updated Ganga driver script to support these changes.

2015-04-14 Rich/RichMirrorAlignmentGanga v12r0

This version was intended for PANOPTES v5r4. Updated the Offline Ganga driver script to work with RichMirrAlign v18r1 and RichMirrCombinFit v14r0. Updated Brunel options to turn off reconstruction of track/photons etc. not needed for the alignment.

2015-04-14 - Paras Naik - Many improvements and comments added to the script. - Idea is now to *possibly* use this script directly in Online alignment, using backEnd == 'online' flags. - Currently 'online' sections do nothing, non-experts: Do not use 'online'. - Maintains backwards compatibility of Offline alignment 2015-04-11 - Paras Naik - Updated the Offline Ganga driver script. Now works with RichMirrAlign v18r1 and RichMirrCombinFit v14r0. Many improvements to python structure of the code. New running instructions detailed on the mirror alignment TWiki. - Commented out earlyData.py line in 2010 data Brunel options. This is no longer needed. - Created settings directory for new *settings*.py file for v12r0 Ganga driver script - Moved old *settings*.py files to "prior" directory under "settings" directory. 2015-04-10 - Chris Jones - Only run the Forward tracking in the RAW options and apply the same momentum cut as in the RICH upfront during the track finding, to avoid the slow track fit fitting tracks we subsequently reject. 2015-04-09 - Chris Jones - Update options to turn off reconstruction of track/photons etc. not needed for the alignment (i.e. non-long tracks)...

2014-04-27 Rich/RichMirrorAlignmentGanga v11r2

This version will be released in PANOPTES v5r2.

2012-07-19 Rich/RichMirrorAlignmentGanga v11r0

This version will be released in PANOPTES v4r2.

2012-11-29 - Matt Coombes - Changes for compatability with new release of RichMirrCombinFit 2012-07-19 - Matt Coombes - Large changes to add in HPD alignment. - Set flags to HPD align. Download and copy HPD db same as Mirror case - Added option alignHPDs to setup script - Only currently support Rich1 HPD alignment. - Set overwrite flag to True in smart merger

2011-07-27 Rich/RichMirrorAlignmentGanga v10r3

This version will be released in PANOPTES v4r1.

2012-03-28 - Matt Coombes - Added Fixed Mirror Magnification coefficients to package in files/MagCoeffs - Replaced setup script to latest - Added Fix to options file to run with latest Brunel in UnPack = ['Reconstruction'] not true.

2011-07-27 Rich/RichMirrorAlignmentGanga v10r2

This version released in PANOPTES v3r4.

2011-07-27 - Matt Coombes - Fixed problem with adding DB slices. - Added ExtOpts which plot histograms only for deltaTheta window specified

2011-06-08 Rich/RichMirrorAlignmentGanga v10r1

This version released in PANOPTES v3r4. Migrated to lxml instead of pyxml. Updated for the 2011 running. Migrated to optimized list of RICH2 mirror combinations. Numerical part migrated from Job application GaudiPython to Executable. Usability improvements. New naming conventions.


2011-11-15  - Matt Coombes
- Updated RichAlignmentGanga.py for new Ganga.

2011-06-14  - Matt Coombes
- Bug found from update 2011-06-08 which prevented RICH1 alignment working.
  Fixed to make script read 'Rich'+RichDetectorStr+'MirrAlign.conf' not
  'Rich2MirrAlign.conf'. Found and fixed in 4 places.
- Fixed bug in adding Database slices

2011-06-08 - Anatoly Solomin
- Incremented version number v10r1.
- Numerical part migrated from Job application GaudiPython to
  Executable. Thus re-enabled the framework to run with the
  checked out local versions of the numeric packages
  (RichMirrCombinFit and RichMirrAlign).
  Accordingly, the wrappers:
  Rich1MirrCombinFitWrapper.py,
  Rich2MirrCombinFitWrapper.py,
  Rich1MirrAlignWrapper.py and
  Rich2MirrAlignWrapper.py are totally rewritten.

2011-05-31 - Anatoly Solomin
- Incremented version number v9r1.
- Fixed prettifying of the XML files, which was spoiled when the
  python-xml API in the framework was migrated from pyxml to lxml.
- Streamlined job, file, etc. naming conventions for better
  usability. Modified programmatic name formation accordingly
- Improved code readability.

2011-05-27 - Matt Coombes
- Incremented version number v8r7.
- For testing reasons added lines to make adding DB slices. This
  is very useful when running alignment tests over lot of different
  databases, (MDMS, HPD images, etc.).
- Changed 2011 data files name.

2011-05-26 - Anatoly Solomin
- Following Matt's tested recipe, eliminated usage of stored
  password. One "lhcb-proxy-init"  before starting ganga is enough.

2011-05-26 - Matt Coombes
- Added files for 2011 running.
- Included new 2011 selected event lists for RICH1 and RICH2.
- Added new, optimized list of RICH2 mirror combinations into the
  "files" folder.

2011-05-25 - Matt Coombes
- Increment version v8r6.
- Added options file to run over 2011 RAW data.
- 2011 data selection makes use of new, optimized list of RICH2
  mirror pairs.
- 2011 data options include momentum cut for RICH1.
- Added "thisCase" variable name to job name designed to help with
  running several simultaneous alignment jobs.

2011-05-20 - Anatoly Solomin
- Increment version v8r5.
- Corrected these release notes: fixed the header of the
  previous release, to address the fact that in PANOPTES
  v3r0, v3r1 and v3r2 these packages were temporarily
  excluded, because xqilla was not yet available in the
  officially released LHCb environment.
- Finally, migrated to lxml, which became officially available,
  from the obsolete pyxml (NB! SetupProject Ganga pytools).
- In all "Popen" calls replaced tcsh with bash: was previously
  discovered and partly replaced by Matt, due to tcsh word limit.

2011-03-30 Rich/RichMirrorAlignmentGanga v8r4

This version released in PANOPTES v3r3. Adapted to the "Completely new approach"; prepared code for migration from pyxml to lxml. Several improvements, including 2011 data running adaptation

2011-03-30 - Matt Coombes
- Added files for 2011 data running.
- Popen now uses bash shell due to tcsh word limit.

2011-02-09 - Matt Coombes, Anatoly Solomin
- Removed: dependency on user asolomin.
- LSF mode altered and may not be fully supported.

2010-12-08 - Anatoly Solomin
- Fixed: xml files are now pretty formatted.

2010-12-05 - Anatoly Solomin
- Increment version v8r2.
- Deleting subjobs except first and last upon completion made safer.

2010-11-24 - Anatoly Solomin
- Increment version v8r1.
- Enabled script to be run off a Scientific Linux box locally,
  using solely AFS.

2010-11-18 - Anatoly Solomin
- Increment version v8r0.
- Migrated from pyxml to lxml.

2010-11-09 - Anatoly Solomin
- Increment version v7r1.
- Adapted to the "Completely new approach".

2010-06-03 Rich/RichMirrorAlignmentGanga v6r3

This version released in PANOPTES v2r13. Improvements of robustness on the Grid.

2010-06-04 - Anatoly Solomin
- Augmented force_status('failed') with force=True.

2010-06-03 - Anatoly Solomin
- Increment version v6r3.
- Added 'killed' status in failing a subjob if it is in
  too-long-interruption of 'running' status.

2010-05-26 Rich/RichMirrorAlignmentGanga v6r2

This version released in PANOPTES v2r12. On-fly evaluation of conversion coefficients; new parameter control; Dirac backend added; magnification coefficients variably calibrated.

2010-05-26 - Anatoly Solomin
- Increment version v6r2.
- Made magnification coefficients calibration tilts
  variate-able instead of being always 1 mrad.

2010-05-24 - Anatoly Solomin
- Adapted for delta theta window width parameter.

2010-05-20 - Anatoly Solomin
- Adapted for the pre-misaligned case.

2010-05-18 - Anatoly Solomin
- Made it possible to re-run selected cases of a chosen iteration
  by hand.

2010-05-14 - Anatoly Solomin
- Increment version to v5r7.
- Added option to run with MC-determined offsets at the stage of
  fitting the angular distributions of photons.
- Improved and compacted naming conventions.

2010-05-04 - Anatoly Solomin
- Increment version to v5r6.
- Added option to run with pre-misaligned conditions.

2010-05-02 - Anatoly Solomin
- Migrated to DiracSplitter for Dirac backend.
- Improved sirvival part of the waiting mechanism.
- Added automatc re-run of an iteration in case at least one of the
  9 jobs of the iteration was killed.
- Improved removal of redundant intermediate subjobs' outputs.

2010-04-29 - Anatoly Solomin
- Hardcoded Brunel's version.
- Removed redundant AppConfig/Brunel lookup.
- Fixed inputsandbox contents for Brunel.

2010-04-28 - Anatoly Solomin
- Increment version to v5r5.
- Waiting mechanism made aware of Dirac backend differences.
- Added CPUTime for Dirac backend.

2010-04-26 - Anatoly Solomin
- Increment version to v5r4.
- Rewritten waiting mechanism.
- Added handling of the stuck-in-submitted case.
- Added possibility to start from any iteration number.

2010-04-15 - Anatoly Solomin
- Increment version to v5r2.
- New options in the ...-Settings files. Passed to the
  Rich2MirrCombinFit via program options.
- 2010-Collisions running enabled.
- Dirac and local backends added.
- All subjob materials are deleted except for the 0 and the last.
  Dramatic decrease of diskspace needed.
- Introduced waiting that allows for the AFS cache coherency
  latency for the output files.

2010-03-28 - Anatoly Solomin
- Increment version to v4r3.
- Conversion coefficients are re-evaluated on-fly for each iteration.
  Eight families of conversion coefficients: primary/secondary times
  Y/Z times +/-. Each mirror segment tilt around one axis results in
  final tilts around both axes: major around the corresponding, minor
  around the alternative one. That is 2D.
- Database patch usage included.
- Renamed the scripts for MC/Collisions and DST/MDF.
- Control parameters are now keyed, and typed as a key=value column
  in *.conf files.

2009-10-23 Rich/RichMirrorAlignmentGanga v3r3

This version released in PANOPTES v2r7. Skeleton script created, update of tags, renamed scripts.

2009-10-23 - Anatoly Solomin
- Increment version to v3r3.
- Skeleton script created.
- Default database tags updated.
- Renamed the scripts more decently:
  job/RichMirrorAlignmentGanga.py
  job/RichMirrorAlignmentGanga_settings.py

2009-07-21 Rich/RichMirrorAlignmentGanga v3r0

This version released in PANOPTES v2r5. Removed supporting files from CVS, Fest-compatible and reduced hardcodedness. TWiki manual.

2009-07-21 - Anatoly Solomin
- Increment version to v3r0.
- Essentially reduced hardcodedness.
- Manual published on new TWiki page:
  LHCbWeb > LHCbRICH > LHCbRichMirrorAlign.

2009-04-20 - Anatoly Solomin
- Update scripts and calibration files.

2009-03-04 - Anatoly Solomin
- Increment version: v1r4.
- Tune options/2008-Files_test.py.

2009-02-27 Rich/RichMirrorAlignmentGanga v1r3

This version is released in PANOPTES v2r2.

2009-02-27 - Anatoly Solomin
- Update release notes and requirements to v1r3.
- Optimized performance of monitoring on DST.

2009-02-25 - Anatoly Solomin
- Migrated all Brunel steering scripts in /options to 2008.

2008-11-01 Rich/RichMirrorAlignmentGanga v1r2

This version is the first official release.

2008-11-01 - Anatoly Solomin
- Migrated to Configurables.