Include predefined templates in .gitlab-ci.yml files

Problem to solve

Following discussions on https://gitlab.com/gitlab-org/gitlab-ee/issues/6167 and https://gitlab.com/gitlab-org/gitlab-ee/issues/5613, this issue tightens up the two by defining the final requirement: include external vendored files.

Further details

Security Products have a release and distribution by nature, because the tools are not shipping as part of GitLab (https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md). They are provided as docker images to be downloaded for each job run. This makes versioning particularly complex and tedious. The best and easiest way to deal with the job definitions is to provide them as vendored template, shipped with each version of GitLab. GitLab-CI already supports external files includes (https://docs.gitlab.com/ee/ci/yaml/#include). Nonetheless, the syntax does not allow includes of files present in the local instance.

Proposal

Create a new sub-folder in https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates for security jobs templates (like SAST, DAST, etc.).

Include templates for security features snippets to that folder: templates should be self-contained so they can be used as independent .gitlab-ci.yml files.

This sub-folder content will be nicely shown for .gitlab-ci.yml templates when adding a new .gitlab-ci.yml via GitLab, thus making it also possible to discover.

Support a new syntax in .gitlab-ci.yml to include predefined templates:

Track the usage (see below).

include:
  - template: SAST

This will load SAST.gitlab-ci.yml file, with the same rules defined in https://docs.gitlab.com/ee/ci/yaml/#include.

These are additional benefits:

• This will work for any template file included in GitLab (not just security)
• We can create job definition templates
• We can include them in a .gitlab-ci.yml file
• Linting the file is not reporting any error
• The included file content is run as part of the pipeline

What does success look like, and how can we measure that?

We can measure the usage by tracking how many times the template: keyword is parsed.