Deployment & GitOps

Deployment is GitOps: the desired cluster state lives in the flux-deployments repo, and FluxCD continuously reconciles the cluster to match it. Engineers don't kubectl apply to prod — they merge manifest changes, and Flux (plus image automation) does the rest.

flux-deployments layout

Apps/
  DEV/   PROD/   CONFIG-MANAGER/
    <service>/   deployment.yaml, service.yaml, configmap.yaml,
                 secret.sops.yaml, ingress.yaml, kustomization.yaml
Infrastructure/
  cert-manager/  Traefik/  Monitoring/  ImageAutoUpdator/  cloudflare/
clusters/
  my-cluster/flux-system/  (gotk-components.yaml, gotk-sync.yaml, kustomization.yaml)

• Apps/ — one folder per service per environment (DEV → kk-ns-dev, PROD → kk-ns-prod), each a Kustomize bundle. Deployed services: backend, frontend, ms-ai, ms-communication, ms-sessions, widget, knock-agents, scheduler, plus per-service Redis (redis-backend, redis-communication, redis-sessions). CONFIG-MANAGER/ holds shared config.
• Infrastructure/ — cluster-wide components (below).
• clusters/my-cluster/ — Flux bootstrap: the GitRepository source and root Kustomization that pull in Apps + Infrastructure.

Environments

A single cluster (my-cluster) with DEV (kk-ns-dev) and PROD (kk-ns-prod) namespaces. Container images are published to Azure Container Registry (knockknockacr.azurecr.io), e.g. kk-backend-repo-prod:1.0.510. (Note: some service buildspec.yml files also target AWS ECR/EKS — the platform spans AWS and Azure; the Flux image automation tracks ACR tags.)

Infrastructure components

Component	Role
Traefik	ingress controller (LoadBalancer, 2 replicas, HTTP→HTTPS, metrics)
cert-manager	TLS via Let's Encrypt (staging + prod ClusterIssuers, wildcard)
Monitoring	Prometheus + Grafana + AlertManager (Prom-stack), Loki (logs), Grafana dashboards
ImageAutoUpdator	Flux ImageUpdateAutomation per service — auto-bumps image tags on new pushes
cloudflare	DNS / WAF

The deploy flow

1. merge code → CI (buildspec) builds image → push to ACR/ECR with new tag
2. Flux ImageAutoUpdator polls the registry (~1 min) → detects the new tag
3. it rewrites the image: tag in the target deployment YAML (Kustomize image
   policy marker), commits & pushes to flux-deployments
4. Flux GitRepository syncs → Kustomization reconciles → rolling update
   (maxSurge 1, maxUnavailable 0)

So shipping is usually just "merge and let CI + Flux carry it" — no manual cluster ops. See CI/CD for the build half.

Secrets management

Secrets are SOPS-encrypted (Age key) and committed as secret.sops.yaml files; Flux decrypts them in-cluster during reconciliation. Never commit plaintext secrets — encrypt with the project SOPS config (.sops.yaml).

Notable per-service deployment facts

• backend runs API replicas plus a separate scheduler pod (cron leader, SERVICE_CRON=true).
• ms-sessions runs an HTTP deployment plus an autoscaling recording-worker deployment (HPA on queue depth).
• ms-ai migrations run as an init step with npm run migration:run:prod (switched from migration:run on 2026-09-02 so the compiled, production-mode migration runner is used — the dev runner needs ts-node and the source tree).
• DEV schedular memory limit raised 400Mi → 768Mi (request 256Mi → 384Mi) on 2026-09-03 after OOMKills on startup — the scheduler constructs every job in its constructor, so its boot footprint is the whole job set.

White-label hosts (*.nextlevelai.site)

Every public-facing service has a second ingress for the white-label apex, added 2026-08-13: Apps/PROD/<service>/ingress-nextlevelai.yaml for widget, ms-ai, ms-sessions, and ms-communication (widget.nextlevelai.site, etc., each with its own <service>-nextlevelai-tls-cert). The backend's whiteLabelHost lib rewrites recording links, the widget video proxy URL, and the install snippet host to this apex for script-white-label agencies; served widget scripts on that host carry no Knock Knock brand strings. Adding a new public service means adding its nextlevelai twin too, or white-label tenants get a mixed-brand URL.

Recent config notes (2026-08/09)

• Jobber OAuth — JOBBER_CLIENT_ID, JOBBER_CLIENT_SECRET, JOBBER_REDIRECT_URI in both Apps/PROD/backend/configmap.yaml and Apps/PROD/schedular/configmap.yaml (2026-09-09). Without them the Jobber service isn't loaded and the card says "not set up on this server".

• Salesforce — SALESFORCE_CLIENT_ID / SALESFORCE_CLIENT_SECRET in the PROD backend configmap (2026-08-20); the service is gated by SERVICE_SALESFORCE in hooks.js, so the keys alone don't enable it.

• PARTNER_API_KEY in the backend configmaps for DEV and PROD (2026-08-24) — the server-to-server key the marketing site's certification apply page uses to post to POST /certification/apply.

• CRAWLER_PROVIDER added to the ms-ai configmaps (DEV + PROD) — selects the website-crawler backend for knowledge-base scraping.

• DEV ms-communication sets EMAIL_QUEUE_REDIS_HOST: redis-backend-service (Apps/DEV/ms-communication/configmap.yaml) — the email BullMQ queue lives on backend's Redis. PROD shares one Redis, so no override there.

• FIRECRAWL_API_KEY added to Apps/DEV/ms-ai/configmap.yaml — must be added to the PROD ms-ai configmap at go-live.

• UNIPILE_DSN updated in both Apps/PROD/backend/configmap.yaml and Apps/PROD/schedular/configmap.yaml — the both-configmaps rule: provider keys/endpoints used by cron feeders must exist in backend AND schedular, or crons silently break.

• New encryption keys added to the backend secret.sops.yaml.

• A region-aware phone validation lib landed across services in this window — keep service images in step when deploying phone-touching changes.