Secrets in automation and CI/CD: one source, short-lived credentials, no values in Git

In an environment that has grown for a few years without a plan for secrets, credentials are everywhere: in pipeline variables across a dozen projects, in encrypted files in repositories with the key in somebody’s home directory, in Kubernetes Secrets created by hand, in configuration files on servers and in administrators’ notes. Rotating one database password means finding every place it is used, and nobody knows where all of them are.

Order does not require an expensive licence, only a few decisions and three mechanisms: an operator that carries secrets from the manager into the cluster, OIDC tokens for pipelines, and a scanner that makes sure nothing finds its way back into the repository.

Rules, whatever the tool

1. One source of truth. Every secret has one place where it is stored and changed. Every other place fetches it from there.
2. Nothing in plain text in the repository. The repository contains a reference to a secret, not its value.
3. Minimal permissions per consumer. The pipeline that builds an image has no access to the production database password. An application reads only its own secrets.
4. Rotation without code changes. Changing a password is a change in one place and, at most, a restart of the consumer.
5. You know who read it. At least for high-value secrets.

Where to keep secrets

Option	Read auditing	Rotation	Who sees the value	When it is enough
CI variables (masked, protected)	none	by hand in every project	project maintainers	secrets used only by the pipeline, as a start
SOPS or Sealed Secrets in the repository	none	re-encrypt and commit	anyone holding the key	GitOps without a separate server
Secrets manager (e.g. OpenBao)	yes	one change at the source	consumers covered by a policy	many consumers, auditing required

Open-source managers include OpenBao, developed under the Linux Foundation after HashiCorp changed the Vault licence in 2023 and compatible with the Vault API. Whatever the tool, check whether directory (LDAP) or SSO sign-in is available in the free edition; some products keep it in paid plans. People’s passwords belong in a separate password manager: they have different owners and a different lifecycle from automation secrets.

Kubernetes: an ExternalSecret instead of a hand-made Secret

External Secrets Operator reads secrets from the manager and creates ordinary Secret objects from them. The repository contains only a definition saying “fetch this and that”, which can safely be versioned and reviewed in merge requests. The connection to the manager is described by one object for the whole cluster:

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: openbao
spec:
  provider:
    vault:
      server: https://bao.corp.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: external-secrets
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets

The application gets its own ExternalSecret in its namespace:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: openbao
  target:
    name: orders-db
    creationPolicy: Owner
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: apps/orders/db
        property: password

refreshInterval sets how often the operator checks the value in the manager, so a rotation at the source reaches the Secret within an hour at most; an application that reads it as environment variables still needs a restart. creationPolicy: Owner makes the ExternalSecret the owner of the Secret, so the Secret disappears together with it. The external-secrets.io/v1 API version is the only one served from operator release 0.17; manifests using v1beta1 have to be updated before upgrading the operator.

GitLab CI: an OIDC token instead of a long-lived token

Instead of keeping a secrets-manager token in project variables, the job asks GitLab for a signed JWT (id_tokens), valid only while the job runs and stating which project and branch it comes from. The manager verifies the signature and issues its own token for a few minutes. The manager-side configuration, done once by an administrator:

vault write auth/jwt/config \
  oidc_discovery_url="https://gitlab.example.com" \
  bound_issuer="https://gitlab.example.com"

vault write auth/jwt/role/ci-orders-deploy - <<'EOF'
{
  "role_type": "jwt",
  "user_claim": "project_path",
  "bound_audiences": ["https://bao.corp.example.com"],
  "bound_claims": {
    "project_path": "apps/orders-api",
    "ref_protected": "true"
  },
  "policies": ["orders-deploy-read"],
  "token_explicit_max_ttl": 300
}
EOF

The role accepts only a token from the apps/orders-api project running on a protected branch or tag. A merge request from a working branch will not get the secret, even if someone changes .gitlab-ci.yml in it. The pipeline job:

deploy-config:
  stage: deploy
  image: $SECRETS_CLI_IMAGE   # image with the vault client, version pinned in the template
  id_tokens:
    BAO_ID_TOKEN:
      aud: https://bao.corp.example.com
  variables:
    VAULT_ADDR: https://bao.corp.example.com
  script:
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=ci-orders-deploy jwt="$BAO_ID_TOKEN")"
    - export DB_PASSWORD="$(vault kv get -field=password secret/apps/orders/db)"
    - ./scripts/migrate.sh
  rules:
    - if: $CI_COMMIT_REF_PROTECTED == "true"

The secrets: keyword with built-in Vault integration is in GitLab’s paid plans; in the free edition, id_tokens with a sign-in in the script, as above, achieves the same.

Ansible: no_log and diff

The playbook takes the secret from a variable supplied by AWX (a credential with a secrets-manager lookup plugin) or from a lookup. Tasks that use it must not print it:

- name: Register the host in the monitoring API
  ansible.builtin.uri:
    url: https://monitoring.corp.example.com/api/hosts
    method: POST
    headers:
      Authorization: "Bearer {{ monitoring_api_token }}"
    body_format: json
    body:
      name: "{{ inventory_hostname }}"
    status_code: [200, 201]
  no_log: true

- name: Configuration file with the database password
  ansible.builtin.template:
    src: app.conf.j2
    dest: /etc/app/app.conf
    owner: root
    group: app
    mode: "0640"
  diff: false
  no_log: true

no_log: true hides the task’s parameters and result, even when run with -vvv. diff: false prevents the file content from being printed when run with --diff. no_log does not, however, protect a variable captured with register: if it later goes into debug, the value appears in the job log.

Inventory and scanning: gitleaks

Migration starts with a list. Gitleaks scans the whole repository history, not only the current files:

gitleaks git --redact --report-format json --report-path orders-api-leaks.json ~/src/orders-api

Since version 8.19, history is scanned with gitleaks git and a directory without history with gitleaks dir; the older detect and protect commands are being phased out. --redact hides the values in the output so that the report does not become a leak itself. In the pipeline, GitLab has a ready-made secret detection template based on Gitleaks, available in all plans:

include:
  - template: Jobs/Secret-Detection.gitlab-ci.yml

A one-off history scan in the pipeline is enabled with the SECRET_DETECTION_HISTORIC_SCAN: "true" variable. Every secret found in history is treated as exposed and replaced. Removing it from the current version of a file removes it neither from history nor from copies of the repository on laptops.

Check your own environment

Scan every cloned repository:

for repo in ~/src/*/; do
  echo "== $repo"
  gitleaks git --redact "$repo"
done
# Good: a no-leaks message for every repository.
# Bad: a finding count above zero; each finding is a secret to rotate.

Secrets without an owner, i.e. created by hand or by a script, sorted oldest first:

kubectl get secrets -A -o json | jq -r '.items[]
  | select(.metadata.ownerReferences == null)
  | select(.type != "helm.sh/release.v1" and .type != "kubernetes.io/service-account-token")
  | "\(.metadata.creationTimestamp)\t\(.metadata.namespace)/\(.metadata.name)"' | sort
# Anything older than, say, a year is a candidate for migration to an ExternalSecret and rotation.
# TLS certificates from cert-manager also show up here and are fine.

The synchronisation state of ExternalSecrets:

kubectl get externalsecrets -A
# Good: READY True and STATUS SecretSynced.
# Bad: SecretSyncedError, meaning the application runs on a stale value or will not start after a restart.

etcd encryption and permissions on Secrets:

sudo k3s secrets-encrypt status
kubectl auth can-i list secrets -n orders --as=system:serviceaccount:ci:default
# Good: Encryption Status: Enabled, and the answer no for accounts that should not read secrets.