Changed eos to cvmfs and included RAWtoALL

Changed eos to cvmfs for data-art Included RAWtoALL switches in Reco.py Renamed scripts to remove "_with_2020" part Removed the test for IDPVM postprocessing

@ncalace @swaban @npetters @asalzbur @goblirsc @lmijovic

This merge request affects 1 package:

• InnerDetector/InDetExample/InDetSLHC_Example

added 21.9 InnerDetector review-pending-level-1 labels

mentioned in commit 199fa8dd

merged

CI Result SUCCESS

	Athena
externals
cmake
make
required tests
optional tests

Full details available at NICOS MR-20772-2019-01-31-01-47
Athena: number of compilation errors 0, warnings 166
CI Jenkins server is switched to https://atlas-sit-ci.cern.ch. It is accessible world-wide (behind CERN SSO). In old links to Jenkins server aibuild080.cern.ch:8080 should be replaced with atlas-sit-ci.cern.ch For experts only: Jenkins output [CI-MERGE-REQUEST 33046]

Thanks for implementing the suggestions @dbiswas - however, it seems that these changes did not remove the old files, but just added new ones under a different name:

https://gitlab.cern.ch/atlas/athena/tree/21.9/InnerDetector/InDetExample/InDetSLHC_Example/test

This means that all of these (including the broken ones) will run nightly in ART. You need to also remove the old files explicitly with 'git rm'.

Since there were comments on the old MR, I think in such cases it is a good use case for a 'Work In Progress' merge request to allow comments.

@swaban in such cases where there are comments on changes, it should at least be checked with those who commented that they are now satisfied before merging.

It would also be good in such cases to link back to the other merge request (!20647 (merged)) so that others can see the history of comments, etc.

@nstyles I have created another merge request to remove those scripts - !20800 (merged)

@nstyles Now the duplicate scripts are removed. So, there are only 3 scripts for 25x100 and 3 scripts for 50x50 and 2 other scripts. However, from the email "ART jobs status report", I see that none of these tests are running for the last few days -

Nothing to report. No tests were registered till 2019-02-01 08:10:09 UTC.

However, when I first added my ART tests, they did run (but with failure) and I got an email like this -

ART jobs status report generated at 2019-01-29 08:10:16 UTC

Summary for InDetSLHC_Example [1]

Fail Success Active Branch

10 0 0 21.9/Athena/x86_64-slc6-gcc62-opt [1.1]

Can you please tell me why the tests are not running?

@ncalace @swaban @npetters @asalzbur @goblirsc @lmijovic @nstyles My 25x100 tests did run in last nightly and one of them produced this output - https://atlas-art-data.web.cern.ch/atlas-art-data/grid-output/21.9/Athena/x86_64-slc6-gcc62-opt/2019-02-01T2154/InDetSLHC_Example/test_InclinedDuals_single_muons_1GeV_25x100/dcube_muons_1GeV/

Please let me know if it looks reasonable or there is still some problem.

Hi @dbiswas,

I'm having a look at the results and I'm worried about some of the comparisons... for example:

• https://atlas-art-data.web.cern.ch/atlas-art-data/grid-output/21.9/Athena/x86_64-slc6-gcc62-opt/2019-02-03T2153/InDetSLHC_Example/test_InclinedDuals_single_muons_1GeV_25x100/dcube_muons_1GeV/plots/h1r_debd63b3-049c-4227-a0df-c049bb24ce8f.png

• https://atlas-art-data.web.cern.ch/atlas-art-data/grid-output/21.9/Athena/x86_64-slc6-gcc62-opt/2019-02-03T2153/InDetSLHC_Example/test_InclinedDuals_single_muons_1GeV_25x100/dcube_muons_1GeV/plots/h1r_13942dd8-bf19-496f-bdec-70e7e7bad194.png

• https://atlas-art-data.web.cern.ch/atlas-art-data/grid-output/21.9/Athena/x86_64-slc6-gcc62-opt/2019-02-03T2153/InDetSLHC_Example/test_InclinedDuals_single_muons_1GeV_25x100/dcube_digi_strip_muons_1GeV/plots/h1r_61d0b5b3-ecae-45fb-946a-8dc6f2bb7a9a.png

Can you remind me how you produced the reference files?

Thank you very much!

Cheers, Noemi

Hi @dbiswas,

Another issue is that these Reco_tf commands miss the line --steering doRAWtoALL - without this, it will try to do the standard HITtoRDO,RDOtoESD,ESDtoAOD steps, but won't find the correct pre/postExec/Includes for the steps.

Thanks,

Nick.

Hi @dbiswas,

as Noemi pointed out, some of the results doesn't seem to make sense. The cluster to track residuals and pulls are just off. Do you have logfiles from when you produced reference?

Thanks, Nora.

Hi @ncalace @npetters I produced the reference files in 20.20.10.8 using s3185 and r9977. I used the same EVNT files that you used for your 50x50 tests. However, I did that before the scripts were changed to include RAWtoALL switches. So, probably I need to update the references as well. If you have the script that you used recently to produce the references for 50x50, can you please share it with me?

Hi @nstyles I will include that line "--steering doRAWtoALL" in my scripts in the next MR.

Thanks,

Diptaparna

However, I did that before the scripts were changed to include RAWtoALL switches. So, probably I need to update the references as well.

It is not strictly necessary to use RAWtoALL for the 20.20 references, since this is only needed for an issue which is specific to 21.9 - it may make it easier to ensure that the configuration is comparable however.

Hello @dbiswas,

following Nick's comment, please check that the configuration used in 20.20.10.8 and in 21.9 is comparable.

Then, in order to produce all the needed histograms for the SimHit comparison, Pixel and Strip RDOs and IDPVM, I have moved in /afs/cern.ch/user/n/ncalace/public/ForDipta/20.20 the few tools I have updated in order to make the same plots provided in 21.9.

• You need to compile the 4 packages you find in the folder on top of 20.20.10.8 and run sim+digi+reco using the AMI tags you have already used (s3185 and r9977). Please, save the HITS, RDO and ESD files and the corresponding logs for 1, 10, 100 GeV muons.

As you know, you need to have the reference file for the simulation, for Pixel digitization, for Strip digitization and IDPVM.

• To create the reference file for the simulation, I have copied postInclude.SiHitAnalysis.py into the same folder. You can add it to the sim command to obtain directly the reference file.

• To create the reference file for the digi (both pixel and strip), I have copied RDOAnalysis_topOptions.py into the same folder. You need to edit it in order to run on the RDO file you have produced.

• To create the output for the IDPVM, I have copied as well the jo PhysValITk_jobOptions.py. This will allow you to obtain the same histogram we have added in 21.9. Again, you might need to edit it and run on the ESD file you have produced.

This is the recipe to follow to create reference files that are consistent with the output in 21.9. I'm pretty sure that @swaban can help you going through these instructions.

You can start working with 1 GeV single muons, and produce the dcube page for the three steps so that we can have a look and check that everything works fine.

Thank you very much!

Hello @dbiswas, Looking at your script, I have also noticed that you have dedicated xml config files for the dcube comparison. This shouldn't be the case since we want to compare the same set of plots as in the other tests.

Could you please update the all the tests using the default config files for the following variables?

dcubecfg_sim, dcubecfg_digi_pixel, dcubecfg_digi_strip, dcubecfg_rec

Thank you, Noemi

Hi @ncalace,

If I understand correctly, the simulation step should be identical for 50x50 and 25x100 tests. So, I think I can take the HITS and SiHitValid.root files that are being used as 50x50 reference and use them as the reference for 25x100 tests as well. Am I right? Or Do I need to separately produce the SiHitValid.root reference for 25x100 tests?

Thanks,

Diptaparna

Hello @dbiswas,

you are right. In fact, for the 25x100 tests, it makes sense to have only the digi+reco tests since the hit production is the same for both 25x100 and 50x50 simulation. At this point we can do two things:

• we could save a set of HITS from 21.9, produced in with a nightly we like, and use them as input to the reco job. If we do so, then we have to remember to update the reference in case something changes during sim.

• we find the way to run the job on the hits produced with the other test, which I guess is possible only if you merge the two tests...

Maybe @lmijovic or @nstyles have ideas on this and know how to run a test only after another is terminated?

Cheers, Noemi

Hello @ncalace @npetters ,

I have produced the 20.20 reference files for the 25x100 tests once again using 1 GeV single muon sample. The reference files along with the logs are in the following directories - /afs/cern.ch/user/d/dbiswas/work/public/ART/Reco2020_5Feb2019/20.20/RunArea/Simulate/Sgl_Muon_1GeV/ /afs/cern.ch/user/d/dbiswas/work/public/ART/Reco2020_5Feb2019/20.20/RunArea/Reconstr/SglMuon1GeV/ /afs/cern.ch/user/d/dbiswas/work/public/ART/Reco2020_5Feb2019/20.20/RunArea/Reconstr/RDO_Analysis_1GeV/

I have run the ART test using these reference yesterday night and here is the produced output - https://test-plots.web.cern.ch/test-plots/test_InclinedDuals_single_muons_1GeV_25x100/

Please let me know what you think about these.

Thanks,

Diptaparna