[CI] Fix pipeline in main due to incorrect rules of update-compile-commands

Description

Fixes this error message in main:

'update-compile-commands' job needs 'check-compile-commands' job, but 'check-compile-commands' does not exist in the pipeline. This might be because of the only, except, or rules keywords. To need a job that sometimes does not exist in the pipeline, use needs:optional.

Also quickly added retry: 1 to the integration tests as these very rarely fail and we don't want to fail the pipeline because of it.

Checklist

• Documentation reflects the changes made.
• Merge Request title is clear, concise, and suitable as a changelog entry. See our contributing docs

References

Closes #1263 (closed)