Add support for absolute submodule URLs

What does this MR do?

This adds support for absolute URLs in submodules when cloned by the runner helper.

• Closes Support absolute URLs to submodules (#3374 - closed),
• Closes Install ssh client in helper container (either ... (#6605 - closed) (by removing the need for SSH in helper images),
• Closes "cannot run SSH" when fetching changes in docke... (#1113 - closed) (by removing the need for SSH in helper images),
• Closes How to use absolot Urls by submodules? (#28215 - closed)
• Closes Relative URLs for submodules aren't handled cor... (gitlab#21975 - closed) (by enabling use of absolute URLs for forking workflows)
• Related Automatically add an ssh key that can access al... (gitlab#21270)

In addition to absolute HTTPS URLs, it adds support for the most commonly used SSH/Git rewrites and this works based on some manual QA, see below.

This passes the -c url.<base-https-url-with-token>.insteadOf=<base-url-to-replace> flag to the submodule update command to rewrite the URL for specifically that command, without polluting any local git config. insteadOf is a common approach for workarounds for this, already mentioned in a few of the issues above, though those usually suggest writing it to the config (which would be a bad idea for shell executors and for subsequent scripting in the job itself IMO).

❗ By default this will rewrite only absolute HTTPS URLs. For Git/SSH rewrites, this relies on a GIT_SUBMODULE_FORCE_HTTPS flag:

In my experience, rewriting Git/SSH URLs to HTTPS can trigger a lot of 429 responses with recursive submodules, which the git client currently does not handle gracefully, and might break existing workflows. To avoid this, I added a GIT_SUBMODULE_FORCE_HTTPS flag which if true will also rewrite git@ and ssh://git@ URLs (the most commonly used ones), with additional -c flags.

⚠ This has some limitations right now, e.g. some edge cases with weird URL formats (https://stackoverflow.com/questions/31801271/what-are-the-supported-git-url-formats). I don't see those used on GitLab/GitHub at all, really. so I think covering the well-known URLs might be a good start for this MR and iterate on it later.

Example pipeline (manual QA):

Repo with private submodule defined with relative, HTTPS, Git, and SSH URLs:

https://gitlab.com/nejc/submodule-absolute-url/blob/test-urls/.gitmodules

Results (see pipeline https://gitlab.com/nejc/submodule-absolute-url/-/pipelines/762280350):

1. ❌ On shared runner, only relative URL works, fatal for all others:
   https://gitlab.com/nejc/submodule-absolute-url/-/jobs/3686229406 I ran out of shared CI minutes. But you know the status quo.

2. ❌ On this branch, with insteadOf enabled for https, without GIT_SUBMODULE_FORCE_HTTPS, relative and absolute HTTPS URLs work (Git, SSH fail):
   https://gitlab.com/nejc/submodule-absolute-url/-/jobs/3686229405

3. ✔ On this branch, with insteadOf and GIT_SUBMODULE_FORCE_HTTPS, all work (relative, HTTPS, Git, SSH):
   https://gitlab.com/nejc/submodule-absolute-url/-/jobs/3686229404

Why was this MR needed?

See linked issues. Requiring relative URLs for submodules is a big pain for forking workflows, especially when either a fork or source project are in a subgroup and relative URLs start breaking.

What's the best way to test this MR?

1. Download a binary from this branch or checkout and make runner-and-helper-bin-host
2. Register the runner against your project with the tag submodule-absolute-url
3. Add a private submodule 4 times using different URL formats, as shown in https://gitlab.com/nejc/submodule-absolute-url/-/blob/test-urls/.gitmodules:
   • relative
   • absolute HTTPS
   • absolute Git
   • absolute SSH
4. Reuse the configuration from https://gitlab.com/nejc/submodule-absolute-url/-/blob/test-urls/.gitlab-ci.yml to test different scenarios.

What are the relevant issue numbers?

#3374 (closed), #6605 (closed), #1113 (closed), #28215 (closed), gitlab#21270.

🛠 with ❤ at Siemens