Secrets Management

Centralized guidance for handling sensitive configuration (API keys, tokens, credentials) across deployment and observability.

Principles

• Least privilege: grant only required scopes (read metrics, push images, etc.).
• No secrets committed: use GitHub Environments/Secrets or a cloud secrets manager; reference paths instead of embedding values.
• Rotation friendly: design workflows so new secret versions propagate with a simple redeploy.
• Separation: keep operational credentials (deployment) distinct from data provider/broker keys.

Sources of Secrets

• GitHub Environments: environment-specific secrets (e.g., GRAFANA_API_KEY, BROKER_API_KEY, DATA_PROVIDER_KEY).
• Cloud secrets manager (optional): long-lived external credentials injected at runtime.
• Local dev .env (excluded from VCS): developer-only keys for experimentation.

Common Secret Categories

• Observability: Grafana API key, Loki/Tempo endpoints.
• Deployment: container registry write access (if not using built-in GITHUB_TOKEN), OIDC role assumptions.
• Strategy Execution: broker API credentials, data feed keys.
• Security Controls: kill-switch tokens, encryption keys (future).

Usage Patterns

• Workflows: reference secrets via GitHub Actions secrets syntax (see .github/workflows/ files). Avoid echoing secret values.
• Services: read secrets from environment variables at startup (e.g., os.environ['BROKER_API_KEY']). Fail fast if missing.
• Logging: never log full secret values; consider hashing for correlation only when necessary.

Standard secret names and env vars

• Grafana Cloud (observability):

  • GRAFANA_OTLP_ENDPOINT — OTLP gateway URL for metrics/traces.
  • GRAFANA_API_KEY — service token with write scopes (metrics:write, logs:write, traces:write).
  • GRAFANA_LOKI_ENDPOINT — Loki push endpoint (if sending logs directly; otherwise use Collector/Promtail).

• Cross-service environment variable:

  • DEPLOYMENT_ENVIRONMENT — set to dev or prod; added as resource attribute for telemetry and as a label for metrics.

Notes:

• Store production values under the GitHub Environment “prod”. For local dev, use a .env file (git-ignored) or shell environment.
• If a single token must be reused initially, document that and plan to split tokens per environment later.

Rotation Process (Example)

1. Add new secret version in GitHub Environment or secrets manager.
2. Update environment variable mapping if name changed (prefer stable names to avoid code churn).
3. Trigger deployment workflow for prod.
4. Verify service health and that telemetry still reports (Grafana dashboards reachable).
5. Remove/disable old secret version.

Auditing & Hygiene

• Periodically query which secrets are defined per environment and prune unused ones.
• Ensure least-privilege roles for OIDC; do not store static cloud keys if OIDC suffices.
• Document newly introduced secrets here (category + purpose) rather than values.

Future Enhancements

• Add secret scanning automation (GitHub Advanced Security or equivalent).
• Introduce sealed secrets / hardware-backed KMS integration.
• Add provenance attestation tying deployed artifact digest to secret version metadata.

References

• Deployment: docs/devops/deployment.md
• Environments: docs/devops/environments.md
• Observability: docs/devops/observability.md