Jorge Camarero Vera (c4ca96e2) at 20 Jul 11:36
Configuring repos for cs8
Ah, ok I see, the changes are in the xrootd-ssi-protobuf-interface
repository
I am not sure that this will work as file id is uint64, but string is required.
We can't really use Metadata
out of box as fid
is uint64 but dCache needs string
there (UUID).
I think the change makes sense. Of course we need to update dCache driver to match the changes and re-do the whole testing. Thus I would like to see this merged after first public release.
Hi @mdavis,
The jumbo cta.eos.Notification
is not optimal, but I see the rational behind. By having operation per request type makes it easy to use. Thus I think it an acceptable compromise. The promotion of archive_file_id
and storage_class
from extended attributes to a first class citizens keeps the logic clean and obvious. Thanks for that.
I see, you have introduced additional create
operation, which makes the forntend EOS friendly. There should ne no problem with it. Moreover, this change will allow us to fully generate archiveReportURL
on the storage server side (dCache or EOS).
So, overall, I think the change makes sense. Of course we need to update dCache driver as well to match the changes and do the testing. Thanks.
After investigating the full software stack of CASTOR repack it was understood that temporal collocation of data is achieved by repacking tapes in order. It became apparent however that the order in which tapes were repacked could be improved.
The CASTOR repack tools order tapes by the last time they were written to. This logic works well for the very first repack of a set of tapes but gets worse and worse as tapes are repacked again and again as the time when the previous repack took place is used. CASTOR uses the last time a tape was written to due to implementation constraints.
CTA should use the creation times of the files written to a tape when ordering that tape in the repack queue. The creation time of a file should not change no matter how many times a file is repacked from one tape to another. Of course the first time a tape is repacked will still be the best as each file was created and written to the tape one after another. Over time tapes will contain files from different tapes that were not repacked during exactly the same time period.
We need to provide and support CTA CI workflows for external users as this is the common platform to debug T1 issuers and external developers requirements.
This public CI workflow is limited to the following use cases:
postgres
catalogue backend on a local filesystem backed objectstore* with CTA release built in our gitlab instanceIndeed this constrained environment allows a developer to work on his laptop with no external dependency.
A few questions needs to be answered initially:
cta-lib
requires oracle-instantclient19.3-basic
not provided by any public repo (proprietary package)@jleduc started a companion document to document the various issues do not hesitate to complement it. Indeed many more questions are emerging and a gitlab issue (and comments) is not structured enough to get the full picture.
We are experiencing problems with IBM tapes where the problem is located in the VOL1 region.
Because cta-taped performs a check of the VOL1 file to see if the good tape was mounted, we can not easily repack them and we have to use low-level tools.
Please implement a way where in this particular case VOL1 check can be optionally skipped so that normal repack can continue.
Thanks. Vlado
The current CI workflow is complex and not very flexible.
The Kubernetes environment is setup with cumbersome commands and not in a "standard" way.
The current CI does not allow to run parallels pipelines with different software environments.
By example it would be good to be able to test the same code with (most urgent first):
This is a prerequisite before adding new type of tests to the CI, so avoid doing them twice.
This task should be done after #43
The list includes:
In EOSCTA docs we have the guidelines to CTA Coding Conventions. However them do not cover ALL code styles. Also it would good to have a tool that applies the formatting automatically + a common config file in CTA repo so all developers use the same style. We can have several configs - one for each favorite IDE.
It would decrease the number of merge conflicts when rebasing dev branches against master.
Internet suggests the following formatters:
If you know another tools, please add and describe them in the comments.
For https://gitlab.cern.ch/cta/operations/-/issues/560#note_5419932 it would be useful if we could keep track of how many scheduler and xrootd threads the frontend is using and expose that so we can monitor it (log it periodically, for example).
During Recall/Migration tape sessions, a lot of errors can occurr. The current error handling logic of the tape sessions is not ideal, as mostly it just revolves around stopping the tape session in progress, when there are better possible alternatives for handling some errors. For example https://gitlab.cern.ch/cta/CTA/-/issues/1096#note_5142172, when there is a mismatch between the checksum of the file and the calculated checksum in the session, the session is stopped in progress, instead of just reporting the error to the user and continuing the session. We should list the errors we can just report from those that warrant a canceled tape session (if any).
Please update this ticket description with any errors you have come across and relevant links.
From https://gitlab.cern.ch/cta/CTA/-/issues/1096
For both theses cases the best option is to report the error to the user and continue the mount.
From https://gitlab.cern.ch/cta/CTA/-/issues/1076
From https://gitlab.cern.ch/cta/CTA/-/issues/1054
From https://gitlab.cern.ch/cta/operations/-/issues/665
The ACTIVITIES_WEIGHTS table is no longer in use and should be removed in the next schema upgrade.