MC Job Options issueshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues2023-09-15T19:06:30+02:00https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/214Assigning same DSID twice2023-09-15T19:06:30+02:00Yiming AbulaitiAssigning same DSID twiceWhen I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
...When I try to generate new DSID, the same DSID assigned twice "524xxx/524547"
Bellow is the output from "./scripts/commit_new_dsid -m "Commit message" --dry-run ../100xxx/*"
Will move ../100xxx/100003 to 524xxx/524546.
Will move ../100xxx/100005 to 524xxx/524547.
Will move ../100xxx/100000 to 524xxx/524546.
Will move ../100xxx/100001 to 524xxx/524547.
Will move ../100xxx/100007 to 602xxx/602485.
Will move ../100xxx/100008 to 602xxx/602486.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/213Job Failed #314314072023-07-31T17:48:45+02:00Yanlin LiuJob Failed #31431407Job [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinJob [#31431407](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/31431407) failed for 2afb6c436741b53d42df6790447447250439f02d:
Dear expert, I'm not quite sure what's this error about? Any insights?
Thanks,
YanlinSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/212Allow running `mc??_valid` input files2023-07-31T09:40:55+02:00Spyros ArgyropoulosAllow running `mc??_valid` input filesCurrently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming conventi...Currently the CI cannot run on input files named like `mc15_valid.602232.Ph_ttj_MiNNLO_scale5_LHE.evgen.TXT.e8531/TXT.34045098._000002.tar.gz.1`
The `valid` replaces the COM energy so it is potentially a not well defined naming convention, since the COM energy is taken automatically from `ecmEnergy` from `log.generate` which is [directly printed from the transform](https://gitlab.cern.ch/atlas/athena/-/blob/main/Generators/EvgenJobTransforms/share/skel.GENtoEVGEN.py#L105) - from the command line arguments.
One solution is to ignore the `ecmEnergy` completely if we see that the input filed is named as `mc??_valid` however I think that's problematic because an LHE should correspond to a given COM energy and we should not allow it to be used indepndently of the `ecmEnergy`.
@katharin @mgignac @dhirschSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/211Job Failed #309375482023-07-11T15:39:23+02:00Yiming AbulaitiJob Failed #30937548Job [#30937548](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/30937548) failed for af0d387553a5529f8514e942f2756bd1e17166e1:
How can I pass this?
If I use full container name "mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_...Job [#30937548](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/30937548) failed for af0d387553a5529f8514e942f2756bd1e17166e1:
How can I pass this?
If I use full container name "mc15_13TeV.345054.PowhegPythia8EvtGen_NNPDF3_AZNLO_WpH125J_MINLO_lvbb_VpT.evgen.TXT.e
5706" then the CI job try to download 500K files.
If I use this "mc15_13TeV:TXT.10406367._041965.tar.gz.1, ...." in log.generate.short then some script add one extra "mc15_13TeV". Then the download failed because rucio can not download "mc15_13TeV:mc15_13TeV:TXT.10406367._041965.tar.gz.1".
If I use only "TXT.10406367._041965.tar.gz.1, ...." format as inputs then CI job failed saying "could not find scope". see the link above.
Any solution for this kind of thing?
I tested locally that "rucio download TXT.10406367._041965.tar.gz.1" works fine, but CI script complain about scope.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/208Speed up retrieval of free DSIDs2023-05-20T16:26:07+02:00Spyros ArgyropoulosSpeed up retrieval of free DSIDsSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/209commit_new_dsid: Symbolic link is not updated2023-05-20T16:26:07+02:00Yiming Abulaiticommit_new_dsid: Symbolic link is not updatedHi,
When I use "commit_new_dsid" to copy JO directory to GIT. Symbolic links are not updated with newly assigned DSIDS.
I used this command to move to directory to GIT area:
```
./scripts/commit_new_dsid -m "Commit message" -n ../100xx...Hi,
When I use "commit_new_dsid" to copy JO directory to GIT. Symbolic links are not updated with newly assigned DSIDS.
I used this command to move to directory to GIT area:
```
./scripts/commit_new_dsid -m "Commit message" -n ../100xxx/*
OK: ../100xxx/100000 is dir
OK: ../100xxx/100001 is dir
```
Then I checked the new directory(DSIDs).
```
[yabulait@lxplus750 mcjoboptions]$ ls -l 523xxx/523102/
lrwxrwxrwx. 0 yabulait zp 41 May 18 15:53 DirectPhotonFilter.py -> ../../100xxx/100000/DirectPhotonFilter.py
-rw-r--r--. 1 yabulait zp 163 May 18 15:53 log.generate.short
lrwxrwxrwx. 0 yabulait zp 66 May 18 15:53 MadGraphControl_MGPy8EG_DMS1_dijetgamma_pta.py -> ../../100xxx/100000/MadGraphControl_MGPy8EG_DMS1_dijetgamma_pta.py
```
Here you can see that the symbolic links are still "../../100xxx/100000/*", but it should be "../../523xxx/523101".
This happened few times recently. So I reported here.
Cheers,
AbletSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/207Job Failed #281910882023-05-11T07:28:09+02:00Yanlin LiuJob Failed #28191088Job [#28191088](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/28191088) failed for edb66c3a931427b434623f6c819b32e1bf786625:
Dear @sargyrop , @dhirsch and @mgignac,
Sorry to bother you. Could you help to advice on how to...Job [#28191088](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/28191088) failed for edb66c3a931427b434623f6c819b32e1bf786625:
Dear @sargyrop , @dhirsch and @mgignac,
Sorry to bother you. Could you help to advice on how to resolve this issue?
Thanks,
Yanlinhttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/205inputFilesPerJob check in logParser2023-03-14T16:19:16+01:00Spyros ArgyropoulosinputFilesPerJob check in logParserHi Spyros Argyropoulos
we would need an additional check added to JOs parser,
all JOs that are run with Gen_tf.py parameter --inputGeneratorFile should contain the setup of inputFilesPerJob, as done here:
https://gitlab.cern.ch/atlas-ph...Hi Spyros Argyropoulos
we would need an additional check added to JOs parser,
all JOs that are run with Gen_tf.py parameter --inputGeneratorFile should contain the setup of inputFilesPerJob, as done here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/601xxx/601755/mc.PhH7EG_PDF4LHC21_ggZH125_Zincl_gammagamma.py
but forgotten here:
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/601xxx/601757/mc.PhH7EG_PDF4LHC21_VBFH125_gammagamma.py
In fact transform sets in such situation default inputFilesPerPerJob=1, but ProdSys is not able to check it and does not want to start the job if this parameter is missing in the JOs.
Thanks,
ewelinA
AGENE-2179Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/206Investigate whether the comment from mcgensvc in the notify job can be made i...2023-03-09T12:39:28+01:00Spyros ArgyropoulosInvestigate whether the comment from mcgensvc in the notify job can be made in a way that needs resolvingSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/117Check number of files in gridpack2023-03-01T07:42:53+01:00Christian GutschowCheck number of files in gridpackThe number of files in a gridpack shouldn't exceed 80k, otherwise some grid sites will crash. This has happened a number of times recently, e.g. for the FxFx job where the gridpack contained several files per Feynman diagram. MadGraph co...The number of files in a gridpack shouldn't exceed 80k, otherwise some grid sites will crash. This has happened a number of times recently, e.g. for the FxFx job where the gridpack contained several files per Feynman diagram. MadGraph control cleans up logs and .o files in the latest release, but for older releases it would be good to have a dedicated pipeline step that throws an error if the number of files in the gridpack is larger than 80k. Probably something like `tar -ztvf *.tgz *.tar.gz` could work?S2.2020Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/200Failures on grid coming from treatment of runArgs.jobConfig not caught in CI2023-02-02T07:45:25+01:00Spyros ArgyropoulosFailures on grid coming from treatment of runArgs.jobConfig not caught in CIOriginal file: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/9938e84d065b3b479dd57e063889896daa7ff9e7/521xxx/521163/MadGraphControl_TRSM_HHH.py#L21
Original MR: !2251
Pipeline passed: https://gitlab.cern.ch/atlas-physic...Original file: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/9938e84d065b3b479dd57e063889896daa7ff9e7/521xxx/521163/MadGraphControl_TRSM_HHH.py#L21
Original MR: !2251
Pipeline passed: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/pipelines/5008939
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/9938e84d065b3b479dd57e063889896daa7ff9e7/521xxx/521163/MadGraphControl_TRSM_HHH.py#L21
This then failed on the grid: see ATLMCPROD-10348 example log: https://bigpanda.cern.ch//media/filebrowser/a8f96442-63a3-4574-aee8-53704fce19da/mc15_13TeV/tarball_PandaJob_5732336628_SiGNET/log.generate
Fix in: !2288
New file in: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/blob/master/521xxx/521163/MadGraphControl_TRSM_HHH.py
Offending line seems to be:
```
f_list = os.listdir(runArgs.jobConfig[0])
```
where `runArgs.jobConfig[0]` on the grid seems to evaluate to `521163` while on the CI it would evaluate to `../521163`. Not clear from which level `Gen_tf.py` runs on the grid.
@mborodin could you point me to the code that executes `Gen_tf.py` on the grid?Spyros ArgyropoulosSpyros Argyropoulos2023-02-05https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/203Test change to make sure errors on above grid job can be reproduced2023-02-01T14:37:13+01:00Spyros ArgyropoulosTest change to make sure errors on above grid job can be reproducedSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/202Ensure that `run_athena` calls `Gen_tf` the same way as it would run on the grid2023-02-01T14:37:04+01:00Spyros ArgyropoulosEnsure that `run_athena` calls `Gen_tf` the same way as it would run on the gridSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/201Understand how `runArgs.jobConfig` is evaluated on the grid2023-02-01T14:36:56+01:00Spyros ArgyropoulosUnderstand how `runArgs.jobConfig` is evaluated on the gridSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/199Branch CI pipline is not running!2023-01-04T18:07:30+01:00Yiming AbulaitiBranch CI pipline is not running!Hi @sargyrop
I pushed a new branch an hour ago.
dsid_yabulait_519065
But CI pipeline is not running! Is it disabled?
I used sparse checkout, and it is working much faster than before.
mkdir mcjoboptions
cd mcjoboptions
git ...Hi @sargyrop
I pushed a new branch an hour ago.
dsid_yabulait_519065
But CI pipeline is not running! Is it disabled?
I used sparse checkout, and it is working much faster than before.
mkdir mcjoboptions
cd mcjoboptions
git init
git remote add -f origin ssh://git@gitlab.cern.ch:7999/atlas-physics/pmg/mcjoboptions.git
git sparse-checkout init
git sparse-checkout set common scripts .gitignore
git pull origin masterSpyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/193Turn off branch pipelines2023-01-03T08:22:29+01:00Spyros ArgyropoulosTurn off branch pipelinesIf someone opens a MR when a branch pipeline is still running 2 concurrent pipelines are created.
One will fail since the one that finishes first will push to the branch and then the last CI job will try to push to a branch that is beh...If someone opens a MR when a branch pipeline is still running 2 concurrent pipelines are created.
One will fail since the one that finishes first will push to the branch and then the last CI job will try to push to a branch that is behind.
![Screenshot_2022-11-17_at_16.36.11](/uploads/afe31bf16cbf3496ccb1bf6f2703d7ac/Screenshot_2022-11-17_at_16.36.11.png)
We should turn off all branch pipelines.Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/197check_jo_consistency failed2022-12-19T16:20:55+01:00Yiming Abulaiticheck_jo_consistency failedHi,
When I commit a second time after fix something in Control file, the consistency check failed.
The consistency is checking some other files that is not a part of my commits (I am trying to register 518405-518446 range).
But job is fa...Hi,
When I commit a second time after fix something in Control file, the consistency check failed.
The consistency is checking some other files that is not a part of my commits (I am trying to register 518405-518446 range).
But job is failed due to some errors related to 421xxx/421100/..
So How can I fix this?
Cheers,
Ablet
Error part:
OK: No generator full name is found
Generators used: ['Py8', 'EG']
ERROR: file /builds/atlas-physics/pmg/mcjoboptions/scripts/../421xxx/421100/mc.Py8EG_A14NNPDF23LO_Ztautau.py contains includes pointing to MC15JobOptions
Failed Job
https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/26543765Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/196ERROR: /cvmfs/.../gridpack files are not readable to all users2022-12-13T09:37:44+01:00Yiming AbulaitiERROR: /cvmfs/.../gridpack files are not readable to all usersHi,
When we addding gripack files via symbolic links, the GIT CI reported the following errors.
At first it said mcgensvc and atlcvmfs can read the file. Then it complaining that not all users can read the file.
Isn't the permission to ...Hi,
When we addding gripack files via symbolic links, the GIT CI reported the following errors.
At first it said mcgensvc and atlcvmfs can read the file. Then it complaining that not all users can read the file.
Isn't the permission to mcgensvc and atlcvmfs enough? or should we make it readable to all ATLAS users?
ERRORs are taken from here: https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/26431913
EOS Console [root://eosuser.cern.ch] |/>
The access rights are set to: egroup:atlas-phys-exotics-tchannel-neutrinos:rxm!dq,u:atlcvmfs:rxm!dq,u:mcgensvc:rxm!dq,u:jneundor:rwx
OK: mcgensvc can read /eos/user/j/jneundor/HeavyNGeneration/gridpacks_regeneration
OK: atlcvmfs can read /eos/user/j/jneundor/HeavyNGeneration/gridpacks_regeneration
ERROR: file /eos/user/j/jneundor/HeavyNGeneration/gridpacks_regeneration/mc_13TeV.aMCPy8EG_NNPDF3NLO_HeavyN_ee_mN10TeV.GRID.tar.gz is not readable by all users.
The permissions are set to:Spyros ArgyropoulosSpyros Argyropouloshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/195Job Failed #262950132022-12-06T16:23:29+01:00Arpan GhosalJob Failed #26295013Job [#22933691](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22933691) failed for ea50f46043f0c61e3433e23f573da905e5273a06:
Hi @mcgensvc,
My pipeline fails because of no space left on device. Any suggestions on how I ...Job [#22933691](https://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/jobs/22933691) failed for ea50f46043f0c61e3433e23f573da905e5273a06:
Hi @mcgensvc,
My pipeline fails because of no space left on device. Any suggestions on how I might mitigate this issue?
Thankshttps://gitlab.cern.ch/atlas-physics/pmg/mcjoboptions/-/issues/194Notify changes assigns MR to wrong person2022-11-20T17:08:34+01:00Spyros ArgyropoulosNotify changes assigns MR to wrong personneed to assign to convenersneed to assign to convenersSpyros ArgyropoulosSpyros Argyropoulos