Move subdirectories from CTA dev repo into new repos

changed the description

I will look at the CTA/doc portion of this and post a candidate solution for how to best preserve and remove the relevant git history from the main repo.

I have tried a bit and found the following:

Proposed solution

There exists a tool called git-filter-repo https://github.com/newren/git-filter-repo, which is not a standard git tool but seems to do exactly what we want.

Recipe for the docs-only repo:

cd <some place that is not the location of your local CTA dev repo>
git clone ssh://git@gitlab.cern.ch:7999/cta/CTA.git # Clone master of CTA repo
cd CTA/
git-filter-repo --subdirectory-filter doc/
git remote add origin <url_of_new_repo>
git push --set-upstream origin master

diff -r shows no difference between the doc/ directory of the original CTA repo and the docs-only repo.

I have created an example repo to verify that the history is preserved the way we want it to be: https://gitlab.cern.ch/rbachman/cta-docs-filter-repo

Recipe for the doc-less repo: (also excludes the LTO_RAO directory due to push file size limit)

cd <some place that is not the location of your local CTA dev repo>
git clone ssh://git@gitlab.cern.ch:7999/cta/CTA.git # Clone master of CTA repo
cd CTA/
git-filter-repo --invert-paths --path doc/ --path LTO_RAO/ --path migration/ --path EOSCTATAPE/
git remote add origin <url_of_new_repo>  # Or use the old one when satisfied, then push with --force
git push --set-upstream origin master

Example repo: https://gitlab.cern.ch/rbachman/cta-nodocs-filter-repo

Tags can be migrated to a new repo by adding the --tags flag to the push. I'm not sure how this will impact existing releases, still looking into this. Also note that force-pushing these changes to the real CTA repo does involve rewriting history, so we should discuss if we really want this and warn collaborators.

Recipe for the repo with all the paths mentioned above removed:

cd <some place that is not the location of your local CTA dev repo>
git clone ssh://git@gitlab.cern.ch:7999/cta/CTA.git # Clone master of CTA repo
cd CTA/
git-filter-repo --invert-paths --path doc/ --path LTO_RAO --path migration/ --path EOSCTATAPE/
git remote add origin <url_of_new_repo>  # Or use the old one when satisfied, then push with --force
git push --tags --set-upstream origin master

Example repo: https://gitlab.cern.ch/rbachman/cta-filter-all

Update:

I managed to get the repo-import to work (import from URL works), but it turns out that this does in fact not clone Gitlab resources such as releases. Nonetheless, here is what the imported CTA repo looks like after a git-filter-repo and a force push: https://gitlab.cern.ch/rbachman/CTA

Not imported:

Issues (looks like these can be handled individually by the Move Issue feature https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#move-an-issue , or in bulk by a Gitlab admin)
Merge requests
Releases
Project settings (new project settings seem to be default ones)
- General settings
- Branch protection rules
- CI runners

Merging branches across the repos

If possible, we would want to avoid having to migrate all branches to the cloned project in one go. Branches are copied during the git project clone, but are not updated by the filter-repo tool. Trying to merge such a branch straight into the repo is very bad https://gitlab.cern.ch/rbachman/CTA/-/merge_requests/1 .

However, running the same git-filter-repo command that was used on master on your feature branch seems to resolve this issue https://gitlab.cern.ch/rbachman/CTA/-/merge_requests/2 .

Recipe for developers migrating from one project to the other (works in simple case):

# Commit all remaining local changes you want to include, then:
git push  # push to original repo just in case
git remote add newrepo <git_url_of_new_project>
git checkout -b "<my_branch_name>_filter-repo> 
git-filter-repo --force --invert-paths --path doc/ --path LTO_RAO/ --path migration/ --path EOSCTATAPE/
git push newrepo
# Then create merge request and merge
# Close original branch on the new project

Tested scenario:

graph TD
orig[Original CTA] -- Gitlab project import --> import[Imported CTA];
import -- git-filter-repo --> filter[Filtered imported CTA];

orig -- check out branch --> branch[Orig. CTA branch];
branch -- commit --> mod[Modified branch];
mod -- git-filter-repo --> modfilt[Filtered modified branch];
modfilt -- push new project --> pushmodfilt[New branch on new CTA];
pushmodfilt -- merge request on master --> filter;

Summary: It looks like git-filter-repo is the way to go.

Alternative 1

(would not recommend if the above is good enough):

There exists an approach using git format-patch and git-am to turn the repo history into a set of patches, which can be applied to a fresh repo. The technique is described here: https://savorywatt.com/2015/01/25/move-files-and-folders-between-git-repos-using-patches/

When trying this however it turns out that it is quite cumbersome: Every time a patch includes changes outside of the target directory (docs/) it fails and one has to modify it manually. For the docs repo 423 patches are produced, and a good amount of the first 50 I tried need at least some basic manual intervention. The intervention count would then probably also have to be doubled, since the reverse changes must be done to rewrite the doc-less history repo.

Alternative 2:

It looks like a good amount of people have used the built-in git-filter-branch tool in the last years for similar use cases https://stackoverflow.com/questions/10067848/remove-folder-and-its-contents-from-git-githubs-history . The tools manpage however contains the following and recommendation to use git-filter-repo:

WARNING
       git filter-branch has a plethora of pitfalls that can produce non-obvious manglings of the intended history rewrite (and can leave you with little time to
       investigate such problems since it has such abysmal performance). These safety and performance issues cannot be backward compatibly fixed and as such, its
       use is not recommended. Please use an alternative history filtering tool such as git filter-repo[1]. If you still need to use git filter-branch, please
       carefully read the section called “SAFETY” (and the section called “PERFORMANCE”) to learn about the land mines of filter-branch, and then vigilantly avoid
       as many of the hazards listed there as reasonably possible.

Instructions for testing a merge into the test-repo

Please test the following instructions with your development branch.

You will need an install of git >= 2.22.0 and to install the git-filter-repo tool (not included in git).

CC7 machines can get a modern version of git through the LCG releases https://lcginfo.cern.ch/pkg/git/. if your machine has the /cvmfs/sft.cern.ch repo enabled you can easily use the existing git install as described here https://lcgdocs.web.cern.ch/lcgdocs/lcgreleases/introduction/ .

If your distribution does not have a package for it, git-filter-repo can be installed using pip https://pypi.org/project/git-filter-repo/ .

These commands will rewrite history in the newly created branch.

# Commit all remaining local changes you want to include, then:
git checkout <my_branch_name>
git push  # push to original repo just in case
git remote add newrepo https://gitlab.cern.ch/rbachman/CTA.git
git checkout -b <my_branch_name>_filter-repo>
git-filter-repo --force --invert-paths --path doc/ --path LTO_RAO/ --path migration/ --path EOSCTATAPE/
git push -u newrepo <my_branch_name>_filter-repo>
# Then create merge request and merge

$ python3 -m pip install --user git-filter-repo
Collecting git-filter-repo
  Downloading https://files.pythonhosted.org/packages/3f/20/18a86303ad957da565d153043d1c8de58d185a49ece69632f46edb7207e5/git_filter_repo-2.34.0-py2.py3-none-any.whl (99kB)
    100% |████████████████████████████████| 102kB 2.1MB/s
Installing collected packages: git-filter-repo
Successfully installed git-filter-repo-2.34.0
$ git-filter-repo --invert-paths --path doc/ --path LTO_RAO --path migration/ --path EOSCTATAPE/ --path python/
Error: need git >= 2.22.0
$ git --version
git version 1.8.3.1
$ sudo yum update git
{...}
No packages marked for update

CentOS Linux release 7.9.2009 (Core)

I'm being told by other CC7 users that it is normal to update one's git install :)

Note: Expiry date for old ST egroups (it-dep-st-*) has been set to 2022-09-07. See KB0008134.

I believe we are still using this e-group to grant Developer access to GitLab. When we switch to the new repo, we should replace it with the new group it-dep-sd.

I have used git subtree to pull cta-dcache module out of main tree (!186 (merged)). The procedure was like this:

create a branch that will contain only commits to a particular directory
create a new bare repo
pull newly created branch as master into that repo
clone from that repo
add remote and push

(from my history file)

git subtree split --prefix cta-dcache -b cta-dcache-only
cd ../
mkdir cta-dcache-frontend
git init --bare
git push ../cta-dcache-frontend cta-dcache-only:master
cd ..
git clone cta-dcache-frontend cta-dcache-frontend-repo
cd cta-dcache-frontend-repo
git remote rename origin old-origin
git remote add origin git@gitlab.desy.de:dcache/cta-dcache-frontend.git
git push -u origin master

Then submodule was added to the tree:

cd CTA
git checkout -b dcache-frontend-submodule origin/master
git submodule add https://gitlab.desy.de/dcache/cta-dcache-frontend.git cta-dcache

This works for the purpose of splitting out the subdirectories into separate repositories.

However, we actually want to rewrite history, as this is needed to clean up the emails in CTA/EOSCTATAPE which are mentioned at the top of the issue. With the submodule approach one still has the commits with the cta-dcache directory in the history.

I have followed the procedure with my branch https://gitlab.cern.ch/cta/CTA/-/tree/cta-1204 and done the following merge request: https://gitlab.cern.ch/rbachman/CTA/-/merge_requests/3

I did not find any difficulties. In addition, after applying the command git-filter-repo to branch cta-1204_filter-repo, I tried to rebase it on top of newrepo/master. All went smoothly, which means that rebuild hashes of cta-1204_filter-repo match the rebuilt hashes of newrepo/master:

(env) afonso@ctadevjoaoafonso:~/workspace/CTA_proto$ git rebase newrepo/master
Current branch cta-1204_filter-repo is up to date.

We decided at the meeting today that I will create the proper new CTA repo and proceed as far as I can with the setup. The new repo will be created from scratch, without exporting/importing, since there is no need to preserve the old branches. I will then alert everyone on MM once I get to the point of doing something that warrants putting the old repo into read-only mode (such as moving over the old issues).

I've created the new project: https://gitlab.cern.ch/cta/cta-new . Once the move is complete we rename it and change the URL to match the present CTA one. The settings should now be copied over with only some minor changes (some were in an undefined state in the original project, which Gitlab does not allow me to replicate, and I've kept the 100MB file size limit around this time to avoid future problems related to this).

I've also submitted the request for the bulk issue-move, just asking it is possible for now: https://cern.service-now.com/service-portal?id=ticket&table=u_request_fulfillment&n=RQF2080595

@mdavis and @rbachman: CTA/python contains the code for eosfstgcd that must be distributed and versioned along with CTA: this is the eos garbage collector that runs on the EOSCTA disk servers.

Cannot go to cta/operations.

CTA/migration contents can now be found in a repo of their own: https://gitlab.cern.ch/cta/castor-eoscta-migration. Please note that removing it breaks CI for existing tags in the new repo. As such I recommend getting the code for already existing tags from the original CTA repo, which we will rename and archive.

The CTA/doc directory now has its own repo at https://gitlab.cern.ch/cta/cta-doc. Please add future presentation slides there.

CTA/LTO_RAO has been merged into the existing similarly named project at https://gitlab.cern.ch/cta/tape-RAO . I have compressed the offending 100MB+ log artifact such that it should no longer be a problem.

added Building and Packaging workflowin review z_deprecated: Public Release labels

We did the switch today, all that remains is to re-tag remaining issues and clean up any other collateral damage. There is a community post on how to migrate branches https://cta-community.web.cern.ch/t/cta-gitlab-project-and-repository-migration/154, but we've found that the instructions in fact can go wrong. The cause for this is not yet fully understood, but perhaps something along the lines of "cherry-picking" commits across repos is a valid alternative https://stackoverflow.com/questions/5120038/is-it-possible-to-cherry-pick-a-commit-from-another-git-repository .

closed

mentioned in issue #110 (closed)

Move subdirectories from CTA dev repo into new repos

Designs

Child items ...

Activity

Proposed solution

Update:

Merging branches across the repos

Alternative 1

Alternative 2:

Instructions for testing a merge into the test-repo

Move subdirectories from CTA dev repo into new repos

Is blocked by

Activity

Proposed solution

Update:

Merging branches across the repos

Alternative 1

Alternative 2:

Instructions for testing a merge into the test-repo