Skip to main content

Deployment & GitOps

Deployment is GitOps: the desired cluster state lives in the flux-deployments repo, and FluxCD continuously reconciles the cluster to match it. Engineers don't kubectl apply to prod — they merge manifest changes, and Flux (plus image automation) does the rest.

flux-deployments layout​

Apps/
DEV/ PROD/ CONFIG-MANAGER/
<service>/ deployment.yaml, service.yaml, configmap.yaml,
secret.sops.yaml, ingress.yaml, kustomization.yaml
Infrastructure/
cert-manager/ Traefik/ Monitoring/ ImageAutoUpdator/ cloudflare/
clusters/
my-cluster/flux-system/ (gotk-components.yaml, gotk-sync.yaml, kustomization.yaml)
  • Apps/ — one folder per service per environment (DEV → kk-ns-dev, PROD → kk-ns-prod), each a Kustomize bundle. Deployed services: backend, frontend (PROD also frontend-beta), ms-ai, ms-communication, ms-sessions, ms-voice-engine, widget, knock-agents, schedular, video-worker (2026-11-13), PROD-only livekit (2026-11-05), plus DEV per-service Redis (redis-backend, redis-communication, redis-sessions). CONFIG-MANAGER/ holds shared config.
  • Infrastructure/ — cluster-wide components (below).
  • clusters/my-cluster/ — Flux bootstrap: the GitRepository source and root Kustomization that pull in Apps + Infrastructure.

Environments​

A single cluster (my-cluster) with DEV (kk-ns-dev) and PROD (kk-ns-prod) namespaces. Container images are published to Azure Container Registry (knockknockacr.azurecr.io), e.g. kk-backend-repo-prod:1.0.510. (Note: some service buildspec.yml files also target AWS ECR/EKS — the platform spans AWS and Azure; the Flux image automation tracks ACR tags.)

Infrastructure components​

ComponentRole
Traefikingress controller (LoadBalancer, 2 replicas, HTTP→HTTPS, metrics)
cert-managerTLS via Let's Encrypt (staging + prod ClusterIssuers, wildcard)
MonitoringPrometheus + Grafana + AlertManager (Prom-stack), Loki (logs), Grafana dashboards
ImageAutoUpdatorFlux ImageUpdateAutomation per service — auto-bumps image tags on new pushes
cloudflareDNS / WAF

The deploy flow​

1. merge code → CI (buildspec) builds image → push to ACR/ECR with new tag
2. Flux ImageAutoUpdator polls the registry (~1 min) → detects the new tag
3. it rewrites the image: tag in the target deployment YAML (Kustomize image
policy marker), commits & pushes to flux-deployments
4. Flux GitRepository syncs → Kustomization reconciles → rolling update
(maxSurge 1, maxUnavailable 0)

So shipping is usually just "merge and let CI + Flux carry it" — no manual cluster ops. See CI/CD for the build half.

Secrets management​

Secrets are SOPS-encrypted (Age key) and committed as secret.sops.yaml files; Flux decrypts them in-cluster during reconciliation. Never commit plaintext secrets — encrypt with the project SOPS config (.sops.yaml).

In the 2026-10-27 security pass every provider key that had been in a configmap (APIFY_TOKEN, APOLLO_API_KEY, BACKEND_API_KEY, GO_HIGH_LEVEL_V2_SSO_KEY, JOBBER_CLIENT_SECRET, ORIGAMI_API_KEY, PARTNER_API_KEY, RB2B_API_KEY, SALESFORCE_CLIENT_SECRET, SNITCHER_API_KEY, UNIPILE_*, RECORDING_LINK_SECRET, …) moved into secret.sops.yaml — check the secret, not the configmap, before assuming a key is missing. Values that must be identical across services:

ValueMust match in
SECRET_ENCRYPTION_KEY (and SECRET_ENCRYPTION_KEYS / _CURRENT if used)backend, schedular, ms-communication (ms-ai has its own) — see Security & Secrets
FILE_LINK_SECRETbackend, schedular, ms-communication. The schedular signs links too (ChatTranscriptEmailJob, CRM sync notes); without the key it falls back to RECORDING_LINK_SECRET, and links it signs then fail on the API pods. PROD schedular has no FILE_LINK_SECRET as of 2026-11-16.
VOICE_ENGINE_RECORDING_API_KEYbackend, ms-communication, ms-voice-engine

Notable per-service deployment facts​

  • backend runs API replicas plus a separate scheduler pod (SERVICE_CRON=true, 1 replica). CronMonitor takes a per-minute Redis claim per job firing (3d508d45d), so scaling the schedular no longer double-runs crons.
  • ms-sessions runs an HTTP deployment plus an autoscaling recording-worker deployment (HPA on queue depth).
  • ms-ai migrations run as an init step with npm run migration:run:prod (switched from migration:run on 2026-09-02 so the compiled, production-mode migration runner is used — the dev runner needs ts-node and the source tree).
  • ms-voice-engine is a StatefulSet (not a Deployment): PROD 2 replicas, HPA 2–6, PDB minAvailable: 1, terminationGracePeriodSeconds: 900, port 7860, IngressRoute voice-engine.knock-knockapp.com, image kk-ms-voice-engine-repo-prod (Flux image policy kk-ms-voice-engine-prd). DRAIN_TIMEOUT_S (600) must outlast the longest call or an unattended Flux roll cuts live calls. Its REDIS_PASSWORD lives only in its own sops secret; without it /ready stays 503. Currently dormant (ms-ai VOICE_S2S_MODE=direct).
  • schedular jobs fire on the wall clock with a fixed per-job offset (2026-10-26, c566fd28f): each interval job gets hash(class name) % interval seconds, so 5/10/15/60-minute jobs no longer all start in the same second (13 jobs at 04:02 UTC on 2026-10-24 took Cosmos from 27% to 99% CPU). A job runs when its wall-clock slot changes, so dropped node-cron ticks no longer drift it, and an outage longer than an interval yields one run, not a burst. CronMonitor's per-minute claim slot is centred on the job's own offset.
  • DEV schedular memory limit raised 400Mi → 768Mi (request 256Mi → 384Mi) on 2026-09-03 after OOMKills on startup — the scheduler constructs every job in its constructor, so its boot footprint is the whole job set.

video-worker​

Apps/{PROD,DEV}/video-worker/ (2026-11-13, 1615bd7f): video-worker-deployment, 1 replica, image kk-backend-video-repo-{prod,dev} built from backend Dockerfile.video (Debian + Chromium + ffmpeg; image policies kk-backend-video-prd / -dev). Same code and envFrom (backend-cm, backend-secret) as the backend, with SERVICE_VIDEO='true' and SERVICE_CRON='': it only consumes the BullMQ product-video queue (release videos for product updates), one render at a time. Requests 500m / 1Gi (1500m left it unschedulable, 47d86921), limits 2000m / 3Gi; probes /test and /healthz/db.

livekit (PROD)​

Apps/PROD/livekit/ (2026-11-05, f6587fc2): self-hosted livekit/livekit-server:v1.9.11 for the widget's Gemini speech-to-speech media leg (WebRTC instead of the Socket.IO audio relay). livekit-signal-service (ClusterIP 7880, IngressRoute livekit.knockknockapp.ai) and livekit-media-service (Azure LoadBalancer on the static IP kk-livekit-prod-ip, TCP 7881 / UDP 7882 — rtc.node_ip in livekit-cm must match it). API keys come from livekit-secret (LIVEKIT_KEYS). ms-communication turns it on with VOICE_DIRECT_LIVEKIT: "true", LIVEKIT_URL (in-cluster ws), LIVEKIT_PUBLIC_URL (wss://livekit.knockknockapp.ai) and LIVEKIT_API_KEY / LIVEKIT_API_SECRET in its secret. See Voice Engine.

White-label hosts (*.nextlevelai.site)​

Every public-facing service has a second ingress for the white-label apex, added 2026-08-13: Apps/PROD/<service>/ingress-nextlevelai.yaml for widget, ms-ai, ms-sessions, and ms-communication (widget.nextlevelai.site, etc., each with its own <service>-nextlevelai-tls-cert). The backend's whiteLabelHost lib rewrites recording links, the widget video proxy URL, and the install snippet host to this apex for script-white-label agencies; served widget scripts on that host carry no Knock Knock brand strings. Adding a new public service means adding its nextlevelai twin too, or white-label tenants get a mixed-brand URL.

Recent config notes (2026-10-20 → 11-16)​

  • Credential vault — CREDENTIAL_VAULT_DUAL_WRITE: 'false' in the PROD backend, schedular and ms-ai configmaps (1a2f8c40, window closed 2026-11-12; no code reads it any more). SECRET_ENCRYPTION_KEY re-set across the backend, schedular, ms-communication and ms-ai secrets 2026-11-11/13 (d5071078, 0e007f25, 839b4de6).
  • FILE_LINK_SECRET added to the backend and ms-communication secrets (ada394c5) — signed private-file links.
  • Product updates (PROD only) — PRODUCT_UPDATES_MESSAGE_STREAM, PRODUCT_UPDATES_TEST_TO in backend + schedular configmaps; POSTMARK_WEBHOOK_USER/_PASSWORD, PRODUCT_UPDATES_UNSUBSCRIBE_SECRET in both secrets; WP_CHANGELOG_SITE/_STATUS/_CATEGORY_ID (configmap) and WP_CHANGELOG_USER/_APP_PASSWORD, OPS_VIDEO_WORKER_KEY (secret) in the backend only. See Ops workspace.
  • Partner Hub — PARTNER_HUB_TOKEN, SERPER_API_KEY, DATAFORSEO_LOGIN/_PASSWORD, BRAVE_SEARCH_API_KEY, ANTHROPIC_API_KEY in the backend secret only (0e40a888, 6f6a81f5) — the scheduled agents run in the schedular. See Partner Hub.
  • xAI / Grok voice — XAI_API_KEY in the ms-ai and ms-communication secrets; XAI_SIP_ENABLED: "true" in the ms-communication configmap (d88deed1, 43e353a7).
  • ms-communication — APN_SANDBOX_FALLBACK: "false" (2026-11-16): a production APNs BadDeviceToken is no longer retried against the sandbox gateway. SOCKET_LEGACY_DASHBOARD_AUTH: "", WHATSAPP_ALLOW_UNSIGNED: "true".
  • ms-ai — Nox v2 for every company (NOX_ONLY_COMPANY_IDS removed 2026-11-03), NOX_JEV_SHADOW + TYPESAFE_API_KEY (2026-11-12), TOOL_AUTH_LEGACY_ALLOW cleared in DEV to match PROD.
  • Alertmanager — an alertmind webhook receiver (AlertMind / PagerMind, send_resolved: true) mirrors every real alert with continue: true before Slack (a4b93da2). See Observability.
  • Secret rotations — PROD PARTNER_API_KEY (4e03d3c4), Stripe key (b7228953), DEV secrets and PROD mismatches (9cfff5b2), ms-ai ANTHROPIC_API_KEY (efe01ccb).

Recent config notes (2026-10)​

  • GO_HIGH_LEVEL_V2_SSO_KEY in both backend and schedular configmaps (2026-10-07, b9ade1c7) — decrypts the GHL marketplace Custom Page SSO payload; unset = the SSO login path can't work.
  • HubSpot tenant app — HUBSPOT_CLIENT_ID in both backend and schedular configmaps; HUBSPOT_CLIENT_SECRET moved from the configmaps into both secret.sops.yaml files (9a0409b2 → 987a527a). There are no in-code defaults; without them the HubSpot service isn't loaded.
  • Alertmanager — the hermes-knockknock webhook receiver and its continue: true route were removed from Infrastructure/Monitoring/Prom-stack/helmrelease.yaml (6898c723).
  • PROD ms-communication secret: OPENAI_API_KEY rotated (6d02d345).

Recent config notes (2026-08/09)​

  • Speech-to-speech voice (2026-09) — see Voice Engine. PROD ms-ai: VOICE_S2S_MODE: "direct" (was vapi from 2026-09-28; direct since 2026-09-30), VOICE_DIRECT_WEB_VENDORS: "openai,gemini" (Gemini added 2026-10-13 with VOICE_DIRECT_GEMINI_API_KEY in the ms-ai sops secret, acd25916), VAPI_PHONE_ENABLED: "false", VOICE_ENGINE_ROLES: "web,inbound,outbound", VOICE_ENGINE_RECORDING_ENABLED: "true" with VOICE_ENGINE_RECORDING_DISCLOSURE_CHECK: "off" (must be explicit), Vapi config (VAPI_ORG_ID, VAPI_DENOISE, secrets VAPI_PRIVATE_KEY / VAPI_WEBHOOK_TOKEN). Also in PROD ms-ai (2026-10): VOICE_DIRECT_TRACE_COMPANIES: "*", VOICE_ENGINE_RECORDING_S3_BUCKET / _REGION (6877a982), VOICE_DIRECT_OPENAI_NOISE_REDUCTION: "near_field" and VOICE_DIRECT_OPENAI_EAGERNESS: "auto" (62c33207; was briefly "low" — reverted in d7c5e8b9 because OpenAI then waited up to ~8 s after the caller finished). VOICE_ENGINE_RECORDING_API_KEY must be the same value in the backend, ms-communication and ms-voice-engine secrets; ms-comm also needs BACKEND_INTERNAL_URL. ms-voice-engine ALLOWED_OPENAI_MODELS is held empty since 2026-09-18 (empty allowlist = provider unreachable). No Reloader — every one of these needs a rollout restart of the consuming service.

  • 2026-09-09 → 09-30 — IP_HASH_SECRET, ORIGAMI_API_KEY and WHITE_LABEL_PORTAL_URL added to both backend and schedular configmaps (rotating IP_HASH_SECRET orphans every stored ip_hash); STRIPE_CONNECT_APPLICATION_FEE_PERCENT is in the backend secret only. The schedular livenessProbe is now /healthz/db (Mongo watchdog); startup/readiness and the API pods stay on /test.

  • Jobber OAuth — JOBBER_CLIENT_ID, JOBBER_CLIENT_SECRET, JOBBER_REDIRECT_URI in both Apps/PROD/backend/configmap.yaml and Apps/PROD/schedular/configmap.yaml (2026-09-09); since 2026-10-19 PROD JOBBER_REDIRECT_URI is https://api.nextlevelai.site/jobber/install (the Jobber app's registered redirect must match). Without them the Jobber service isn't loaded and the card says "not set up on this server".

  • Salesforce — SALESFORCE_CLIENT_ID (configmap) / SALESFORCE_CLIENT_SECRET (now in the secret) in PROD backend (2026-08-20); the service is gated by SERVICE_SALESFORCE in hooks.js, so the keys alone don't enable it.

  • PARTNER_API_KEY (2026-08-24; PROD value now in the backend secret) — the server-to-server key the marketing site's certification apply page uses to post to POST /certification/apply.

  • CRAWLER_PROVIDER added to the ms-ai configmaps (DEV + PROD) — selects the website-crawler backend for knowledge-base scraping.

  • DEV ms-communication sets EMAIL_QUEUE_REDIS_HOST: redis-backend-service (Apps/DEV/ms-communication/configmap.yaml) — the email BullMQ queue lives on backend's Redis. PROD shares one Redis, so no override there.

  • FIRECRAWL_API_KEY added to Apps/DEV/ms-ai/configmap.yaml — must be added to the PROD ms-ai configmap at go-live.

  • UNIPILE_DSN updated in both Apps/PROD/backend/configmap.yaml and Apps/PROD/schedular/configmap.yaml — the both-configmaps rule: provider keys/endpoints used by cron feeders must exist in backend AND schedular, or crons silently break.

  • New encryption keys added to the backend secret.sops.yaml.

  • A region-aware phone validation lib landed across services in this window — keep service images in step when deploying phone-touching changes.