Productions for Charm: charm_cbaryon_to_phh_2024
A series of productions have been created from ef763c01 in !1784 (merged).
cc @msaur
Production Details for Pipeline 6203
Summary table
| Step | Production ID | Num Test LFNs | Run time | Estimated Output Size (MB) |
|---|---|---|---|---|
| data_LcpToPpKmPip_charm_Collision24_24c2_MagDown | 115789 | 1 | 0:28:18.010438 | 8.4 |
| data_LcpToPpKmKp_charm_Collision24_24c2_MagDown | 115790 | 1 | 0:27:26.048638 | 1.6 |
| data_LcpToPpKpPim_charm_Collision24_24c2_MagDown | 115791 | 1 | 0:29:30.298201 | 3.1 |
| data_XicpToPpPimPip_charm_Collision24_24c2_MagDown | 115792 | 1 | 0:29:18.369303 | 4.1 |
| data_LcpToPpPimPip_charm_Collision24_24c2_MagDown | 115793 | 1 | 0:31:33.146724 | 6.3 |
| data_XicpToPpKmKp_charm_Collision24_24c2_MagDown | 115794 | 1 | 0:43:11.417199 | 1.2 |
| data_LcpToPhiPp_charm_Collision24_24c2_MagDown | 115795 | 1 | 0:31:51.413876 | 0.9 |
| data_LcpToPpKmKp_fromb_charm_Collision24_24c2_MagDown | 115796 | 1 | 0:31:32.976074 | 0.3 |
| data_LcpToPpKpPim_fromb_charm_Collision24_24c2_MagDown | 115797 | 1 | 0:20:05.233457 | 0.5 |
| data_LcpToPpPimPip_fromb_charm_Collision24_24c2_MagDown | 115798 | 1 | 0:43:50.676327 | 0.7 |
| data_LcpToPhiPp_fromb_charm_Collision24_24c2_MagDown | 115799 | 1 | 0:33:31.405797 | 0.4 |
| data_XicpToPpKmKp_fromb_charm_Collision24_24c2_MagDown | 115800 | 1 | 0:25:35.463800 | 0.4 |
| data_XicpToPpPimPip_fromb_charm_Collision24_24c2_MagDown | 115801 | 1 | 0:41:45.465118 | 0.6 |
| data_LcpToPpKmPip_charm_Collision24_24c2_MagUp | 115802 | 1 | 0:31:10.806084 | 7.8 |
| data_LcpToPpKmKp_charm_Collision24_24c2_MagUp | 115803 | 1 | 1:53:12.153637 | 1.3 |
| data_LcpToPpKpPim_charm_Collision24_24c2_MagUp | 115804 | 1 | 0:43:12.693767 | 2.7 |
| data_XicpToPpPimPip_charm_Collision24_24c2_MagUp | 115805 | 1 | 0:26:47.247913 | 4.0 |
| data_LcpToPpPimPip_charm_Collision24_24c2_MagUp | 115806 | 1 | 2:09:56.134374 | 6.4 |
| data_XicpToPpKmKp_charm_Collision24_24c2_MagUp | 115807 | 1 | 0:26:46.537295 | 1.1 |
| data_LcpToPhiPp_charm_Collision24_24c2_MagUp | 115808 | 1 | 0:29:33.447216 | 1.0 |
| data_LcpToPpKmKp_fromb_charm_Collision24_24c2_MagUp | 115809 | 1 | 0:38:34.739195 | 0.3 |
| data_LcpToPpKpPim_fromb_charm_Collision24_24c2_MagUp | 115810 | 1 | 0:28:46.763578 | 0.4 |
| data_LcpToPpPimPip_fromb_charm_Collision24_24c2_MagUp | 115811 | 1 | 0:34:31.177254 | 0.8 |
| data_XicpToPpKmKp_fromb_charm_Collision24_24c2_MagUp | 115813 | 1 | 0:28:03.023020 | 0.3 |
| data_XicpToPpPimPip_fromb_charm_Collision24_24c2_MagUp | 115814 | 1 | 0:25:54.020422 | 0.5 |
| data_LcpToPpKmPip_charm_Collision24_24c3_MagUp | 115815 | 1 | 0:30:09.929956 | 6.9 |
| data_LcpToPpKmKp_charm_Collision24_24c3_MagUp | 115816 | 1 | 0:20:50.468729 | 1.5 |
| data_LcpToPpKpPim_charm_Collision24_24c3_MagUp | 115817 | 1 | 0:42:00.771406 | 2.9 |
| data_XicpToPpPimPip_charm_Collision24_24c3_MagUp | 115818 | 1 | 0:33:40.889057 | 4.1 |
| data_LcpToPpPimPip_charm_Collision24_24c3_MagUp | 115819 | 1 | 0:42:53.123198 | 5.9 |
| data_XicpToPpKmKp_charm_Collision24_24c3_MagUp | 115820 | 1 | 0:42:49.509617 | 1.2 |
| data_LcpToPhiPp_charm_Collision24_24c3_MagUp | 115821 | 1 | 0:32:45.612488 | 0.8 |
| data_LcpToPpKmKp_fromb_charm_Collision24_24c3_MagUp | 115822 | 1 | 0:40:02.367661 | 0.4 |
| data_LcpToPpKpPim_fromb_charm_Collision24_24c3_MagUp | 115823 | 1 | 0:34:19.339196 | 0.4 |
| data_LcpToPpPimPip_fromb_charm_Collision24_24c3_MagUp | 115824 | 1 | 0:45:37.523191 | 0.7 |
| data_LcpToPhiPp_fromb_charm_Collision24_24c3_MagUp | 115825 | 1 | 0:25:52.712233 | 0.4 |
| data_XicpToPpKmKp_fromb_charm_Collision24_24c3_MagUp | 115826 | 1 | 0:29:52.697546 | 0.3 |
| data_XicpToPpPimPip_fromb_charm_Collision24_24c3_MagUp | 115827 | 1 | 0:40:16.333610 | 0.4 |
| data_LcpToPhiPp_fromb_charm_Collision24_24c2_MagUp | 115828 | 1 | 0:20:54.376654 | 0.4 |
Metadata
{"ci_ids": [115789, 115790, 115791, 115792, 115793, 115794, 115795, 115796, 115797, 115798, 115799, 115800, 115801, 115802, 115803, 115804, 115805, 115806, 115807, 115808, 115809, 115810, 115811, 115813, 115814, 115815, 115816, 115817, 115818, 115819, 115820, 115821, 115822, 115823, 115824, 115825, 115826, 115827, 115828], "request_ids": [140800, 140801, 140802, 140803, 140804, 140805, 140806, 140807, 140808, 140809, 140810, 140811, 140812, 140813, 140814, 140815, 140816, 140817, 140818, 140819, 140820, 140821, 140822, 140823, 140824, 140825, 140826, 140827, 140789, 140790, 140791, 140792, 140793, 140794, 140795, 140796, 140797, 140798, 140799]}Activity
added Charm ProductionNew labels
assigned to @msaur
added Productionactive label and removed ProductionNew label
@cburr Sorry for troubles, I misunderstood that it is preferred to split jobs into smaller parts (one tuple per line per data-taking period) to prevent memory overload. I guess in that case also #840 (closed) should be killed as it is based on same logic.
I guess it should be more like set of line per tuple and data-taking period?
No worries, you weren't to know.
The memory stuff was a managable workaround for the first few weeks of data taking but we're now the load of actually reading the all data is significant. We also have the workaround in lhcb/LHCb#369 and the next DaVinci release will improve things dramatically.
For now, reducing the number of times you read the same dataset is the most important thing. In the next few weeks we'll have some better options for making things easier.
And yeah, #840 (closed) is also going to be stopped as it's trying to read 62,000 TB of data (I'm using TB to try and express just how absurd it is...).
Thanks for all the details. I am aware of push to fix memory issues with functors (large set of improvements by Gerhard scheduled to be released soon) but not so up to date with offline operational issues.
So, before making more even more mess, let me check with you directly. If the point is to reduce number of how many times we are accessing a specific data set (with accessing 62 PB I think I won a honorable mention of being the worst offender at next LHCb week report :-D ), then the best would be to have just one DaVinci job doing all the tupling and putting everything into one ROOT file. However, that looks rather problematic given expected output size and it would combine too many things together.
Probably doable in case of this AP (!1784 (merged)) but #840 (closed) (!1581 (merged)) is trying to do a central tupling of about half of BnoC lines with different groups looking into those samples. Would be doable to group code at !1581 (merged) into 5 jobs (based on similarity of lines), so in total it would be 5 * 3 (sprucing versions and magnet polarity) = 15 jobs; or still better to have just one big job and then deal with splitting of resulting ROOT file downstream?
-
Believe it or not this wouldn’t be the worst case, though the others started during c1 so I’d already entered discussions before it got out of hand.
Over the next few weeks @roneil is going to be working on some new options for helping with this, like the ability to split one file with many trees into many files with fewer trees, to "prescale" the input data for early studies and to be able to apply post processing to reduce the size of the tuples.
Obviously this isn’t ready yet but it should be soon. I don’t know how flexible you are with waiting a few weeks before having the “right” solution available?
Hello @cburr, going back to this, I can wait with a planned massive production for a few weeks (I would like to have those tuples before the end of year both for starting a proper analysis and tuning selection for 2025) but would still like to run a smaller AP over 3-4 charm lines and 2-3 BnoC lines. That way the AP could be modified to have all the lines in one DaVinci jobs and also we could have some data to start looking on. Would that be fine?
added Productiondone label and removed Productionactive label
mentioned in merge request !1837 (merged)