Upgrade clang-format and use Ruff for Python linting and formatting

This MR updates clang-format to version 16 and switch to Ruff for Python linting and formatting (see #42 (closed) and gaudi/Gaudi!1591 (merged)).

I didn't include (yet) a full reformatting as it touches almost all files (because of a bug fix in the formatting logic) and we have too many MRs that would end up with conflicts.

Another thing that should be taken into account is that pre-commit relies on connections to remote Git repositories for the bootstrap. At the moment it tries to connect to github, so it fails on machines that cannot access github (like some of the LHCb private build machines), see gaudi/Gaudi#247. One option would be add mirrors on gitlab of the repositories we use (like those in https://gitlab.cern.ch/lhcb-core/mirrors).

This is a partial follow up on https://mattermost.web.cern.ch/lhcb/pl/umdze4wazjdq58pnr9dse9ezqa

Fixes #42 (closed)

added RTA label

mentioned in merge request !4373 (merged)

mentioned in merge request !4377 (merged)

mentioned in merge request !4378 (merged)

mentioned in merge request Detector!466 (merged)

@clemenci Maybe I am miss-reading your description above. Are you saying with the changes here it will no longer be possible to run clang format by hand ?

added 1033 commits

• 0f45e59a...e0dd6c85 - 1032 commits from branch master
• b56e116d - Use pre-commit for formatting, flake8 and additional checks

Compare with previous version

marked this merge request as ready

added 1 commit

• 2f2fd379 - Use pre-commit for formatting, flake8 and additional checks

Compare with previous version

changed the description

added 1 commit

• b958fd61 - Fix Python 3 compatibility (and formatting)

Compare with previous version

mentioned in merge request Lbcom!712 (merged)

mentioned in merge request !4472 (merged)

What's the next step here ? This is now more than one year old. Can we have it merged this winter ?

It would indeed be nice to update clang-format and switch from YAPF to Black (or the equivalent ruff as in Gaudi).

I need a few minutes (30min max) to update this MR to the current state in Gaudi, then it's a matter of agreeing on when to apply the formatting change.

resolved all threads

added 688 commits

• b958fd61...5aee24da - 686 commits from branch master
• 3216979e - use pre-commit for formatting and additional checks
• b49f4488 - Fix Python 3 compatibility (and formatting)

Compare with previous version

changed title from Use pre-commit for formatting, flake8 and additional checks to Use pre-commit for formatting and additional checks

changed title from Use pre-commit for formatting and additional checks to Upgrade clang-format and use Ruff for Python linting and formatting

changed the description

added 4 commits

• b49f4488...c66f36fd - 2 commits from branch master
• 750bfe18 - Update clang-format and switch to Ruff for Python linting and formatting
• 4c961400 - Fix Python 3 compatibility (and formatting)

Compare with previous version

added 2 commits

• 6cf3caf4 - Update clang-format and switch to Ruff for Python linting and formatting
• 49d6674d - Fix Python 3 compatibility (and formatting)

Compare with previous version

added 2 commits

• f8599d09 - Update clang-format and switch to Ruff for Python linting and formatting
• 2f0a93ce - Fix Python 3 compatibility (and formatting)

Compare with previous version

approved this merge request

resolved all threads

merged

mentioned in commit e8e10fd2

@sponce @clemenci Can you remind me, now this is merged how do I manually run the formatting on a file, before committing it ? I vaguely remember there is some git command to do this ? Also, is thre something I now need to run in my local checkout to enable the pre-commit stuff with git ?

The precommit is on by default. So reindentation will be applied on commit.

If you want to reindent manually, just say pre-commit run <fil1> <file2>... or even pre-commit run --allfiles

I've never tried it with LHCb obviously but will soon, but at least with Gaudi its never worked for me having the files automatically formatted correctly on commit. I have always had to wait for the pipeline in gitlab to fail and then use the git am instructions there. If the formatting is suppose to happen automatically on commit locally, then what am I missing that means it is not working for me ?

So on a quick test it seems to work for me now with LHCb. Need to figure out why it isn't with Gaudi...

Hi @jonrob, I implemented things for DaVinci a good 2 years ago and things have been running just fine :). You should be fine as well.

I'm talking about Gaudi not DaVinci. Gaudi has also been using it for a while but its never worked for me for some reason.

I suspect my issue was I never knew I needed to run pre-commit install. Having now done this it seems to work as advertised.

I think once this is rolled out to more LHCb projects its something that should be better advertised, and properly automated in lb-stack.

pre-commit install is automatically invoked by lb-project-init. I do not know about lb-stack.

Thats fine, but lb-project-init is only really run once, when a project is first cloned, so if the pre-commit came along after that it wouldn't have helped, which I think was my case for Gaudi.

lb-stack will run it for new setups, but again only when first created.

b.t.w. I see this is now merged for LHCB and Lbcom - Is the plan to roll it out to the remaining LHCb projects as well soon ?

Yes, in steps.

For the moment we have the updated pre-commit-config, but we should run a global reformat ASAP.

I've already prepared the global reformat and written the instructions for people to rebase their MRs once I push it. I just wanted to double check that with @clemenci tomorrow morning and push it then.

Then I plan to let a couple of days go through to see how LHCb and Lbcom behave before we go for Rec and Moore, which are usually more busy

mentioned in merge request Rec!4224 (merged)

mentioned in merge request Moore!4234 (merged)