Deployment & GitOps
Deployment is GitOps: the desired cluster state lives in the
flux-deployments repo, and FluxCD continuously reconciles the cluster to
match it. Engineers don't kubectl apply to prod — they merge manifest changes,
and Flux (plus image automation) does the rest.
flux-deployments layout
Apps/
DEV/ PROD/ CONFIG-MANAGER/
<service>/ deployment.yaml, service.yaml, configmap.yaml,
secret.sops.yaml, ingress.yaml, kustomization.yaml
Infrastructure/
cert-manager/ Traefik/ Monitoring/ ImageAutoUpdator/ cloudflare/
clusters/
my-cluster/flux-system/ (gotk-components.yaml, gotk-sync.yaml, kustomization.yaml)
- Apps/ — one folder per service per environment (
DEV→kk-ns-dev,PROD→kk-ns-prod), each a Kustomize bundle. Deployed services: backend, frontend (PROD alsofrontend-beta), ms-ai, ms-communication, ms-sessions, ms-voice-engine, widget, knock-agents, schedular, video-worker (2026-11-13), PROD-only livekit (2026-11-05), plus DEV per-service Redis (redis-backend,redis-communication,redis-sessions).CONFIG-MANAGER/holds shared config. - Infrastructure/ — cluster-wide components (below).
- clusters/my-cluster/ — Flux bootstrap: the
GitRepositorysource and rootKustomizationthat pull in Apps + Infrastructure.
Environments
A single cluster (my-cluster) with DEV (kk-ns-dev) and PROD
(kk-ns-prod) namespaces. Container images are published to Azure Container
Registry (knockknockacr.azurecr.io), e.g. kk-backend-repo-prod:1.0.510.
(Note: some service buildspec.yml files also target AWS ECR/EKS — the platform
spans AWS and Azure; the Flux image automation tracks ACR tags.)
Infrastructure components
| Component | Role |
|---|---|
| Traefik | ingress controller (LoadBalancer, 2 replicas, HTTP→HTTPS, metrics) |
| cert-manager | TLS via Let's Encrypt (staging + prod ClusterIssuers, wildcard) |
| Monitoring | Prometheus + Grafana + AlertManager (Prom-stack), Loki (logs), Grafana dashboards |
| ImageAutoUpdator | Flux ImageUpdateAutomation per service — auto-bumps image tags on new pushes |
| cloudflare | DNS / WAF |
The deploy flow
1. merge code → CI (buildspec) builds image → push to ACR/ECR with new tag
2. Flux ImageAutoUpdator polls the registry (~1 min) → detects the new tag
3. it rewrites the image: tag in the target deployment YAML (Kustomize image
policy marker), commits & pushes to flux-deployments
4. Flux GitRepository syncs → Kustomization reconciles → rolling update
(maxSurge 1, maxUnavailable 0)
So shipping is usually just "merge and let CI + Flux carry it" — no manual cluster ops. See CI/CD for the build half.
Secrets management
Secrets are SOPS-encrypted (Age key) and committed as secret.sops.yaml files;
Flux decrypts them in-cluster during reconciliation. Never commit plaintext
secrets — encrypt with the project SOPS config (.sops.yaml).
In the 2026-10-27 security pass every provider key that had been in a configmap
(APIFY_TOKEN, APOLLO_API_KEY, BACKEND_API_KEY, GO_HIGH_LEVEL_V2_SSO_KEY,
JOBBER_CLIENT_SECRET, ORIGAMI_API_KEY, PARTNER_API_KEY, RB2B_API_KEY,
SALESFORCE_CLIENT_SECRET, SNITCHER_API_KEY, UNIPILE_*, RECORDING_LINK_SECRET, …) moved
into secret.sops.yaml — check the secret, not the configmap, before assuming a key is
missing. Values that must be identical across services:
| Value | Must match in |
|---|---|
SECRET_ENCRYPTION_KEY (and SECRET_ENCRYPTION_KEYS / _CURRENT if used) | backend, schedular, ms-communication (ms-ai has its own) — see Security & Secrets |
FILE_LINK_SECRET | backend, schedular, ms-communication. The schedular signs links too (ChatTranscriptEmailJob, CRM sync notes); without the key it falls back to RECORDING_LINK_SECRET, and links it signs then fail on the API pods. PROD schedular has no FILE_LINK_SECRET as of 2026-11-16. |
VOICE_ENGINE_RECORDING_API_KEY | backend, ms-communication, ms-voice-engine |
Notable per-service deployment facts
- backend runs API replicas plus a separate scheduler pod (
SERVICE_CRON=true, 1 replica). CronMonitor takes a per-minute Redis claim per job firing (3d508d45d), so scaling the schedular no longer double-runs crons. - ms-sessions runs an HTTP deployment plus an autoscaling recording-worker deployment (HPA on queue depth).
- ms-ai migrations run as an init step with
npm run migration:run:prod(switched frommigration:runon 2026-09-02 so the compiled, production-mode migration runner is used — the dev runner needs ts-node and the source tree). - ms-voice-engine is a StatefulSet (not a Deployment): PROD 2 replicas,
HPA 2–6, PDB
minAvailable: 1,terminationGracePeriodSeconds: 900, port 7860, IngressRoutevoice-engine.knock-knockapp.com, imagekk-ms-voice-engine-repo-prod(Flux image policykk-ms-voice-engine-prd).DRAIN_TIMEOUT_S(600) must outlast the longest call or an unattended Flux roll cuts live calls. ItsREDIS_PASSWORDlives only in its own sops secret; without it/readystays 503. Currently dormant (ms-aiVOICE_S2S_MODE=direct). - schedular jobs fire on the wall clock with a fixed per-job offset (2026-10-26,
c566fd28f): each interval job getshash(class name) % intervalseconds, so 5/10/15/60-minute jobs no longer all start in the same second (13 jobs at 04:02 UTC on 2026-10-24 took Cosmos from 27% to 99% CPU). A job runs when its wall-clock slot changes, so dropped node-cron ticks no longer drift it, and an outage longer than an interval yields one run, not a burst. CronMonitor's per-minute claim slot is centred on the job's own offset. - DEV schedular memory limit raised 400Mi → 768Mi (request 256Mi → 384Mi) on 2026-09-03 after OOMKills on startup — the scheduler constructs every job in its constructor, so its boot footprint is the whole job set.
video-worker
Apps/{PROD,DEV}/video-worker/ (2026-11-13, 1615bd7f): video-worker-deployment,
1 replica, image kk-backend-video-repo-{prod,dev} built from backend
Dockerfile.video (Debian + Chromium + ffmpeg; image policies kk-backend-video-prd /
-dev). Same code and envFrom (backend-cm, backend-secret) as the backend, with
SERVICE_VIDEO='true' and SERVICE_CRON='': it only consumes the BullMQ
product-video queue (release videos for product updates),
one render at a time. Requests 500m / 1Gi (1500m left it unschedulable, 47d86921),
limits 2000m / 3Gi; probes /test and /healthz/db.
livekit (PROD)
Apps/PROD/livekit/ (2026-11-05, f6587fc2): self-hosted livekit/livekit-server:v1.9.11
for the widget's Gemini speech-to-speech media leg (WebRTC instead of the Socket.IO
audio relay). livekit-signal-service (ClusterIP 7880, IngressRoute
livekit.knockknockapp.ai) and livekit-media-service (Azure LoadBalancer on the
static IP kk-livekit-prod-ip, TCP 7881 / UDP 7882 — rtc.node_ip in livekit-cm
must match it). API keys come from livekit-secret (LIVEKIT_KEYS). ms-communication
turns it on with VOICE_DIRECT_LIVEKIT: "true", LIVEKIT_URL (in-cluster ws),
LIVEKIT_PUBLIC_URL (wss://livekit.knockknockapp.ai) and LIVEKIT_API_KEY /
LIVEKIT_API_SECRET in its secret. See Voice Engine.
White-label hosts (*.nextlevelai.site)
Every public-facing service has a second ingress for the white-label apex,
added 2026-08-13: Apps/PROD/<service>/ingress-nextlevelai.yaml for widget,
ms-ai, ms-sessions, and ms-communication (widget.nextlevelai.site,
etc., each with its own <service>-nextlevelai-tls-cert). The backend's
whiteLabelHost lib rewrites recording links, the widget video proxy URL, and
the install snippet host to this apex for script-white-label agencies; served
widget scripts on that host carry no Knock Knock brand strings. Adding a new
public service means adding its nextlevelai twin too, or white-label tenants
get a mixed-brand URL.
Recent config notes (2026-10-20 → 11-16)
- Credential vault —
CREDENTIAL_VAULT_DUAL_WRITE: 'false'in the PROD backend, schedular and ms-ai configmaps (1a2f8c40, window closed 2026-11-12; no code reads it any more).SECRET_ENCRYPTION_KEYre-set across the backend, schedular, ms-communication and ms-ai secrets 2026-11-11/13 (d5071078,0e007f25,839b4de6). FILE_LINK_SECRETadded to the backend and ms-communication secrets (ada394c5) — signed private-file links.- Product updates (PROD only) —
PRODUCT_UPDATES_MESSAGE_STREAM,PRODUCT_UPDATES_TEST_TOin backend + schedular configmaps;POSTMARK_WEBHOOK_USER/_PASSWORD,PRODUCT_UPDATES_UNSUBSCRIBE_SECRETin both secrets;WP_CHANGELOG_SITE/_STATUS/_CATEGORY_ID(configmap) andWP_CHANGELOG_USER/_APP_PASSWORD,OPS_VIDEO_WORKER_KEY(secret) in the backend only. See Ops workspace. - Partner Hub —
PARTNER_HUB_TOKEN,SERPER_API_KEY,DATAFORSEO_LOGIN/_PASSWORD,BRAVE_SEARCH_API_KEY,ANTHROPIC_API_KEYin the backend secret only (0e40a888,6f6a81f5) — the scheduled agents run in the schedular. See Partner Hub. - xAI / Grok voice —
XAI_API_KEYin the ms-ai and ms-communication secrets;XAI_SIP_ENABLED: "true"in the ms-communication configmap (d88deed1,43e353a7). - ms-communication —
APN_SANDBOX_FALLBACK: "false"(2026-11-16): a production APNsBadDeviceTokenis no longer retried against the sandbox gateway.SOCKET_LEGACY_DASHBOARD_AUTH: "",WHATSAPP_ALLOW_UNSIGNED: "true". - ms-ai — Nox v2 for every company (
NOX_ONLY_COMPANY_IDSremoved 2026-11-03),NOX_JEV_SHADOW+TYPESAFE_API_KEY(2026-11-12),TOOL_AUTH_LEGACY_ALLOWcleared in DEV to match PROD. - Alertmanager — an
alertmindwebhook receiver (AlertMind / PagerMind,send_resolved: true) mirrors every real alert withcontinue: truebefore Slack (a4b93da2). See Observability. - Secret rotations — PROD
PARTNER_API_KEY(4e03d3c4), Stripe key (b7228953), DEV secrets and PROD mismatches (9cfff5b2), ms-aiANTHROPIC_API_KEY(efe01ccb).
Recent config notes (2026-10)
GO_HIGH_LEVEL_V2_SSO_KEYin both backend and schedular configmaps (2026-10-07,b9ade1c7) — decrypts the GHL marketplace Custom Page SSO payload; unset = the SSO login path can't work.- HubSpot tenant app —
HUBSPOT_CLIENT_IDin both backend and schedular configmaps;HUBSPOT_CLIENT_SECRETmoved from the configmaps into bothsecret.sops.yamlfiles (9a0409b2→987a527a). There are no in-code defaults; without them the HubSpot service isn't loaded. - Alertmanager — the
hermes-knockknockwebhook receiver and itscontinue: trueroute were removed fromInfrastructure/Monitoring/Prom-stack/helmrelease.yaml(6898c723). - PROD ms-communication secret:
OPENAI_API_KEYrotated (6d02d345).
Recent config notes (2026-08/09)
-
Speech-to-speech voice (2026-09) — see Voice Engine. PROD ms-ai:
VOICE_S2S_MODE: "direct"(wasvapifrom 2026-09-28; direct since 2026-09-30),VOICE_DIRECT_WEB_VENDORS: "openai,gemini"(Gemini added 2026-10-13 withVOICE_DIRECT_GEMINI_API_KEYin the ms-ai sops secret,acd25916),VAPI_PHONE_ENABLED: "false",VOICE_ENGINE_ROLES: "web,inbound,outbound",VOICE_ENGINE_RECORDING_ENABLED: "true"withVOICE_ENGINE_RECORDING_DISCLOSURE_CHECK: "off"(must be explicit), Vapi config (VAPI_ORG_ID,VAPI_DENOISE, secretsVAPI_PRIVATE_KEY/VAPI_WEBHOOK_TOKEN). Also in PROD ms-ai (2026-10):VOICE_DIRECT_TRACE_COMPANIES: "*",VOICE_ENGINE_RECORDING_S3_BUCKET/_REGION(6877a982),VOICE_DIRECT_OPENAI_NOISE_REDUCTION: "near_field"andVOICE_DIRECT_OPENAI_EAGERNESS: "auto"(62c33207; was briefly"low"— reverted ind7c5e8b9because OpenAI then waited up to ~8 s after the caller finished).VOICE_ENGINE_RECORDING_API_KEYmust be the same value in the backend, ms-communication and ms-voice-engine secrets; ms-comm also needsBACKEND_INTERNAL_URL. ms-voice-engineALLOWED_OPENAI_MODELSis held empty since 2026-09-18 (empty allowlist = provider unreachable). No Reloader — every one of these needs a rollout restart of the consuming service. -
2026-09-09 → 09-30 —
IP_HASH_SECRET,ORIGAMI_API_KEYandWHITE_LABEL_PORTAL_URLadded to both backend and schedular configmaps (rotatingIP_HASH_SECRETorphans every storedip_hash);STRIPE_CONNECT_APPLICATION_FEE_PERCENTis in the backend secret only. The schedular livenessProbe is now/healthz/db(Mongo watchdog); startup/readiness and the API pods stay on/test. -
Jobber OAuth —
JOBBER_CLIENT_ID,JOBBER_CLIENT_SECRET,JOBBER_REDIRECT_URIin bothApps/PROD/backend/configmap.yamlandApps/PROD/schedular/configmap.yaml(2026-09-09); since 2026-10-19 PRODJOBBER_REDIRECT_URIishttps://api.nextlevelai.site/jobber/install(the Jobber app's registered redirect must match). Without them the Jobber service isn't loaded and the card says "not set up on this server". -
Salesforce —
SALESFORCE_CLIENT_ID(configmap) /SALESFORCE_CLIENT_SECRET(now in the secret) in PROD backend (2026-08-20); the service is gated bySERVICE_SALESFORCEinhooks.js, so the keys alone don't enable it. -
PARTNER_API_KEY(2026-08-24; PROD value now in the backend secret) — the server-to-server key the marketing site's certification apply page uses to post toPOST /certification/apply. -
CRAWLER_PROVIDERadded to the ms-ai configmaps (DEV + PROD) — selects the website-crawler backend for knowledge-base scraping. -
DEV ms-communication sets
EMAIL_QUEUE_REDIS_HOST: redis-backend-service(Apps/DEV/ms-communication/configmap.yaml) — the email BullMQ queue lives on backend's Redis. PROD shares one Redis, so no override there. -
FIRECRAWL_API_KEYadded toApps/DEV/ms-ai/configmap.yaml— must be added to the PROD ms-ai configmap at go-live. -
UNIPILE_DSNupdated in bothApps/PROD/backend/configmap.yamlandApps/PROD/schedular/configmap.yaml— the both-configmaps rule: provider keys/endpoints used by cron feeders must exist in backend AND schedular, or crons silently break. -
New encryption keys added to the backend
secret.sops.yaml. -
A region-aware phone validation lib landed across services in this window — keep service images in step when deploying phone-touching changes.