Skip to content

Introduce an explicit concept of `secrets:` in GitLab CI

Problem to solve

TL;DR The usage of variables: is confusing today and we want to add a proper secrets handling.

We had awesome discussion with @grzesiek and @krasio about Vault integration and Secrets.

This extends the proposal from !31831 (closed) and #28321 (closed).

Proposal

secrets:

We decided that we see a big value in having explicit secrets:

  1. The secrets: are different than variables:
  2. The secrets: clearly indicate a purpose: We want a secret to be used!
  3. The variables: can be used by variable expansion and are by definition "known"
  4. The secrets: can be only used during the execution of the CI job by the Runner
  5. We have many credential providers that can be used by secrets:
  6. The CI job if it wants to use any secret needs to request it explicitly, only requested are exposed
  7. Secrets create a clear split between variables: and secrets: on their purpose
  8. Secrets access should be auditable by know: who, when and for what
  9. All secrets are masked, always

Vault integration

  1. Vault is one of the secret providers
  2. We want Vault to be easy to be use in default case, but allow flexibility for our customers that have different approach
  3. We want Vault auth in general to be configured via UI, but not for the first iteration!
  4. Today, we consider Vault to be configured using CI Variables: VAULT_URL, VAULT_AUTH_.... This allows us to have a single vault per job, but many different vaults used by different jobs, as these VAULT_* variables can be defined for a specific environment scope or in context of a specific job
  5. We offer a simple opinionated syntax, we use by default kv2 engine
  6. We convert a simple syntax (a string) into a complex hash having all details explicitly stated. This is how we process secrets internally, but we expose an easy to start usage of Vault.
  7. We allow users to fine-configure all details if the simple syntax is not enough for them

Vault is configured by adding to CI Variables (can be via CI/CD Settings, or part of YAML):

  • VAULT_URL: this is required
  • VAULT_AUTH_* (username, password, role): this is optional
secrets:
  # production/s3 is a `path:`, where `secret-key` is a fieldname
  # @path/to/engine indicates a path used by `kv-v2` which we consider to be a default engine
  VAULT_SIMPLE_SECRET:
    vault: production/s3/secret-key
    file: true/false # optional, with default: false
                     # false: VAULT_SIMPLE_SECRET would hold a masked value
                     # true: VAULT_SIMPLE_SECRET would hold a path to ephemeral file holding a secret

  VAULT_SIMPLE2_SECRET:
    vault: production/s3/secret-key@path/to/engine

  VAULT_COMPLEX_SECRET:
    vault:
      engine:
        name: kv-v2
        path: path/to/engine
      path: production/s3
      field: secret-key

Rethinking the purpose of CI/CD Variables (mid-term future?)

  1. CI/CD Variables are multipurpose and confusing as they are a "known values", like MY_URL and "secrets", like AWS_S3_SECRET_KEY
  2. All variables are always exposed to each CI job which has a security consequences, as it increases a chance of exposing secret when it does not need it
  3. GitLab can be another secret provider (aka. internal), similar to other secret providers that we implement
  4. GitLab does offer an internal Secret Provider
  5. One of the ideas behind internal secret provider is to mark some Variables as of type Secret
  6. Another idea is to split the current variables into being Variables and Secrets
  7. Secret as described earlier needs to be written explicitly in CI job as required to be used

We would

secrets:
  MY_INTERNAL_SECRET:
    gitlab:
    file: true/false # optional, with default: false
                     # false: MY_INTERNAL_SECRET would hold a masked value
                     # true: MY_INTERNAL_SECRET would hold a path to ephemeral file holding a secret

Other credentials storage (future)

Any other credential storage takes a path of vault: following the same design pattern, like:

secrets:
  MY_CREDHUB_SECRET:
    credhub: path/to/secret

Intended users

Links / references

Edited by Krasimir Angelov