Security & Secrets
How tenant secrets are stored and how the platform's trust boundaries are enforced, as of November 2026 (security review + hardening rollout 2026-10-27 → 2026-11-12).
SecretCrypto — the keyring
backend/app/helpers/SecretCrypto.js is AES-256-GCM for every secret we must replay (OAuth tokens, API keys, signing secrets). Blobs are { v, iv, tag, ct }; v names the key version, so keys rotate without breaking stored data. Most blobs are bound to their owner with AAD (e.g. company:<id>|<provider>), so a blob copied onto another record won't decrypt. Never use helpers/CryptHelper.js (hardcoded key, reused IV).
| Env var | Meaning |
|---|---|
SECRET_ENCRYPTION_KEYS | JSON keyring {"v2":"<32 random bytes, base64>", …} |
SECRET_ENCRYPTION_KEY_CURRENT | Version new blobs are encrypted with (optional when only one key exists) |
SECRET_ENCRYPTION_KEY | Legacy passphrase, read as version v1 |
Missing keys throw at boot in production; locally a fixed dev fallback is used.
Parity. The backend, the schedular and ms-communication (app/helpers/SecretCrypto.js, a copy with the same format) must hold the same keys — the schedular sends from tenant mailboxes and runs every CRM job, and ms-communication decrypts WhatsApp tokens. ms-ai has its own keyring for its Postgres secrets (src/core/crypto/secret-crypto.ts, same format) and never shares the backend's. In flux, SECRET_ENCRYPTION_KEY lives in each service's secret.sops.yaml.
Key fingerprints. On first use each pod records an HMAC fingerprint per key version in credential_keys. A pod whose key differs from the recorded one refuses to read or write vault secrets (instead of corrupting them) and logs one diagnostic line (b3d551eb4) with fingerprint prefixes only.
Rotating a key: add the new version to SECRET_ENCRYPTION_KEYS in backend and schedular (and ms-communication), point SECRET_ENCRYPTION_KEY_CURRENT at it, restart, then run Admin → Integrations Maintenance → Re-encrypt stored secrets (POST /admin/maintenance/reencrypt-secrets, async job, dry-run first). Drop the old key only once a dry run shows 0 stale and 0 failed. ms-ai's equivalent is the Encrypt ms-ai secrets card (backend loops ms-ai POST /credentials/encrypt-at-rest until remaining is 0): it encrypts Shopify/WooCommerce credentials (integrations.credentials) and Twilio/Telnyx keys (elabs_phone_number.meta) and re-encrypts ones on an older key.
Credential vault
backend/app/services/CredentialVault.js is the only way in or out of the credentials collection (CredentialModel): one document per owner (company_id or agency_id) + provider, unique on the three; data is a SecretCrypto blob (select: false, removed from toJSON), plus a UI-safe hint and a rev counter.
get(owner, provider)returns plaintext for calling the provider — never return or log it.status()is the UI-safe form (connected,hint,updated_at).set()merges a patch (null removes a field) under a Redis lock per record plus compare-and-set onrev. An unreadable record is only replaced on a fresh connect (reset: true); any other write refuses, so a pod with the wrong key can't overwrite secrets it can't read.- An unreadable vault record never locks a tenant out of the dashboard (
813768c42):presentSecretslogs and returnsnull. - Changes by people and every delete are audited (
dataredacted); job token refreshes are not.
Providers in the vault: GoHighLevel (go_high_level for a location, ghl_self_account, agency ghl_agency), HubSpot, Salesforce, Outlook, Jobber, Calendly, Google Calendar, Cal.com, Simpro, Tapi, ManyReach, PracSuite, Semble, ServiceM8, tenant mailbox (mailbox), login codes (ghl_login, platform_login) and platform_webhook. WhatsApp tokens stay on the agent (whatsapp_agents.*.connection.*_enc) but are SecretCrypto blobs and never sent to the browser.
The rollback-safe dual-write window (vault + old fields) is closed: 856f5b145 removed the old secret fields and the dual-write code, and flux 1a2f8c40 set CREDENTIAL_VAULT_DUAL_WRITE: 'false' in the backend, schedular and ms-ai configmaps (no code reads it any more). The vault migration cards were removed from the admin panel (a5bdabda).
Hashed secrets
Secrets that callers present to us are stored as a SHA-256 hash (app/lib/secretHash.js) and looked up by it. Zapier and Lead API keys exist only as a hash on the company (zapier.secret_hash, lead_api.secret_hash). Login codes are kept in the vault (so the owner can see theirs) and the vault keeps the lookup hash on the company (go_high_level.secret_hash, platform_login.secret_hash) current on every write. API keys are shown once at generation; afterwards the dashboard only gets a hint (app/lib/presentSecrets.js). The GHL login code is returned only to the owner's menu link, never to team members.
Private files & signed links
app/lib/privateFiles.js lists upload folders that are never public: knowledge-base documents, certification proof, chat attachments, WhatsApp media, issued partner certificates and product-update videos. They are stored with a private S3 ACL and no url, and served only by GET /files/:id/get to a caller it can identify or one holding a signed link. File deletes are limited to files the caller's company/agency owns (23ab5038c).
Signed links (app/lib/fileLink.js): …/files/<fileId>/get?exp=<unixSeconds>&sig=<hex>, sig = HMAC-SHA256(secret, "file:<fileId>.<exp>"), valid 365 days, revocable by rotating the secret. Secret: FILE_LINK_SECRET (falls back to RECORDING_LINK_SECRET); no secret → nothing is signed and every signature is rejected (fail closed). ms-communication has a sign-only copy (app/helpers/fileLink.js, same test vectors) for links it puts in transcripts and emails. In PROD FILE_LINK_SECRET is in the backend and ms-communication secrets.
Related maintenance cards (Admin → Integrations Maintenance): Sign chat attachment links, Chat file link protection, Make private files private in storage.
Super-admin restrictions
- IP allowlist. A
user_type: 'admin'user is refused (403 Admin access is not allowed from this IP) on sign-in (password, Google, Apple, code) and on every authenticated request unless the client IP is in theadmin_allowed_ipscollection (app/lib/adminIpAllowlist.js, called frommiddlewares/passport.js). It fails closed on a DB error. There is no admin screen for the list — rows are added directly in Mongo. user_typeis immutable (f1f945416): set once at creation, dropped from every update. Admins can't be created through the app, andadmin_accessis only writable by the admin team controller.
Sessions & the frontend
SESSION_INVALID.middlewares/passport.jstags its 401s withcode: 'SESSION_INVALID'; auth DB errors answer a generic 503 instead. The dashboard (frontend/src/utils/authInterceptors.js) logs out only on aSESSION_INVALID401 from the backend, ms-ai, ms-sessions or ms-communication, and sends the bearer token and platform header only to those origins. Any other 401 (a third-party API, a provider) no longer ends the session.- Sanitised HTML. Text from outside — lead/visitor replies in the notifications drawer, Nox output, scraped citation sources — goes through
utils/sanitizeHtml.js(its own DOMPurify instance; drops style, id, name, form fields and media) beforev-html. Visitor HTML never runs in the dashboard: the chat-list preview parses withDOMParser(utils/stripHtml.js) and live replay uses a script-free sandboxed iframe. Admin email logs render in a sandboxed frame. - Secrets never reach other users through user endpoints (
84325ba0f), and Nox screen sharing never sends secrets.
Leaked-secret clean-up (2026-10)
After the October 2026 incident, maintenance cards revoked and rotated the leaked set: Revoke Slack tokens and Rotate agent webhook secrets remain on Integrations Maintenance (outbound/SMS agents of the leaked companies get a new webhook_secret; the tenant's caller must be updated). Slack credentials are no longer hardcoded (c583ea511). Flux rotated PROD PARTNER_API_KEY, DEV secrets and fixed PROD secret mismatches (4e03d3c4, 9cfff5b2), and moved provider keys from configmaps into secret.sops.yaml.
Error reporting & personal data
Sentry (BUGSINK_DSN) runs with sendDefaultPii: false; app/lib/sentryScrub.js drops request bodies, client IP and event.user (a4a132393). See Observability.
Related
- Cross-service communication — inter-service keys
- Deployment — where each secret lives in flux
- Admin overview