Observability
Each service initializes observability first (the otel.js / otel.ts
required at the top of the entry file) so instrumentation wraps everything.
Tracing & metrics
- OpenTelemetry — all Node services export OTLP traces.
ms-aiadds auto-instrumentation plus Traceloop instrumentation for Anthropic/OpenAI calls. - Prometheus + Grafana + AlertManager — cluster metrics and alerting
(
Infrastructure/Monitoring/Prom-stackin flux-deployments), including custom alerting and probe (health-check) rules. Thehermes-knockknockAlertmanager webhook receiver was removed on 2026-09-30 (6898c723). Since 2026-11-10 (a4b93da2) every real alert is first mirrored to thealertmindwebhook receiver (hooks.pagermind.com/webhooks/alertmanager-knockknockapp,send_resolved: true,continue: true) and then routed to Slack as before. - Server-Timing (backend) — mounted when
DEBUGis true (it is in the PROD backend configmap), so every response carriestotal,db;dur=<ms>;desc="<n> queries"anddb.slowest(duration +collection.opof the slowest Mongo query) —lib/dbTiming.js(bd38a0dae,5a3dd3401). Read it in the browser's Network → Timing tab to tell Mongo time from pod/queue time; overlapping queries are summed, sodbcan exceedtotal. - Grafana dashboards — pre-built dashboards shipped via Flux.
LLM-specific (ms-ai)
- Grafana Pyroscope — continuous profiling (
GRAFANA_PYROSCOPE_*). - Grafana Sigil — LLM tracing (token usage, prompt/response spans), used by the
Knox/agent code paths (
sigilGenerateText). - AI cost tracking —
cost-trackingmodule records per-provider token usage and pricing; daily reconciliation; queryable viaGET /token-usage/:company_id.
Logs
- Winston in the Nodevel services (backend exposes a basic-auth
/logsviewer; cleanup cron prunes oldEmailLog/ErrorLog). - Loki aggregates logs cluster-wide (
Infrastructure/Monitoring/Loki);winston-lokiships logs from services. - PostHog Logs are no longer used by the backend — the last
posthog_helper.logcall sites were removed on 2026-08-25 (f59478165). The helper class stays for the next module that needs prod-visible debugging; the policy (add for the module under debugging, remove afterwards) is unchanged.
Azure database diagnostic logs → Loki
Cosmos Mongo and Postgres emit diagnostic logs to the kkdblogsprod storage
account (insights-logs-vcoremongorequests,
insights-logs-postgresqlflexquerystoreruntime). They reach Loki through
Infrastructure/Monitoring/LogShipper: a CronJob (db-log-shipper, every
minute, azure-cli image running ship.py from the configmap) that walks
each source forward one hour-blob at a time from its checkpoint to the live
hour, range-GETting CHUNK_READ_BYTES (15 MB) at a time and pushing to Loki in
≤ 2 MB chunks under job="azure-db-diagnostics" with db, category, and
operationName labels. The checkpoint ({year, month, day, hour, offset} per
source in the log-shipper-checkpoints container; the legacy
{blob, offset} shape is migrated on read) advances after every chunk, so
a crash loses at most one chunk and a Loki outage replays rather than drops.
Each run self-limits to RUNTIME_BUDGET_SECONDS (240, under the pod's
activeDeadlineSeconds 300, 1 Gi memory), split evenly across sources so a
big Mongo backlog can't starve Postgres.
History (2026-09-18): the old version read a whole hour-blob into memory and
OOM-crash-looped for 12 days (one hour was 547 MB). Loki rejects samples older
than its ~7-day reject_old_samples_max_age (separate from the 30-day blob
retention), so a "timestamp too old" 400 is treated as handled and the
checkpoint jumps straight to the cutoff parsed from Loki's response, rounded
up to the next hour. Running every minute is cost-neutral — storage
transactions scale with bytes moved, not run frequency.
The first attempt — a Grafana Alloy deployment tailing the blobs through a
blobfuse2 CSI mount (Infrastructure/Monitoring/Alloy) — is kept in the repo
but suspended (suspend: true, 2026-08-20): the mount's directory polling
generated enough storage API calls to cost $700+ NZD per week. Don't
un-suspend it; extend the CronJob instead.
Grafana's Loki datasource is configured by hand, not via GitOps (as are the
Azure Monitor and two Postgres datasources) — a sidecar-provisioned "Loki"
datasource was added and reverted on 2026-09-18 because the name clash broke the
provisioning reload. Check /api/datasources, not the Helm configmap, before
adding one.
Query in Grafana with {job="azure-db-diagnostics", db="cosmos-mongo"} — this is
where connection N to <ip> timed out style Mongo timeouts and slow-query
runtime records are visible.
Error reporting
- Sentry (
@sentry/node+ profiling) in backend, initialized inotel.jswithBUGSINK_DSN(PROD backend configmap; the name is historical — it is a Sentry ingest DSN). Unset, orENVnot staging/production → Sentry is not initialised. Staging sends toBUGSINK_DSN; production sends to a DSN hardcoded inotel.js(soBUGSINK_DSNthere only acts as the on switch). Socket.io and signed-recording paths are never traced. Tracing/profiling enabled in prod;SENTRY_ENABLEDis set in the backend, schedular, ms-sessions and ms-communication configmaps. - No personal data (
a4a132393, GDPR):sendDefaultPii: false(no automatic IP, cookies or AI prompt/response content) andapp/lib/sentryScrub.jsdrops request bodies, the client IP andevent.userfrom every event.
Job & queue visibility
- CronMonitor — backend's Redis-backed cron run history at
/admin/cron. Each firing (and manual trigger) first claimscron:claim:<job>:<minute>in Redis, so only one schedular replica runs it; a missing claim is skipped silently, and without Redis it fails open. Interval jobs fire at a fixed per-job offset inside their interval (not all on :00), and the claim slot is centred on that offset — so "next run" times in the monitor are staggered by design (see Deployment › per-service facts). - Bull Board — BullMQ queue dashboards at
/admin/queues(backend and ms-ai, basic auth). Backend queues now includetenant-webhooks(custom webhook + Zapier delivery, 4 attempts) alongside email and CRM sync. - Socket.IO admin UI —
@socket.io/admin-uion the realtime services (dev).
Health checks
Each service exposes liveness/readiness probes (e.g. ms-ai GET /health/live,
/health/ready checking DB + Redis; the Nodevel services use a static GET /
or /test). Prometheus probe-alerts watch these.
Backend MongoDB self-healing
fd3ce2873. On 2026-09-19 the schedular pod's driver pool sat in
MongoServerSelectionError: connection N to <host> closed for ~20 h: mongoose
emitted no event, readyState stayed 1, the static /test probe stayed green,
and every cron run failed. Three layers now catch it:
GET /healthz/db(app/routes/api.js) — a realpinground-trip (HEALTHZ_DB_TIMEOUT_MS, default 5 s) returning 200/503 withping_ms,ready_stateand the watchdog counters. It is the schedular's livenessProbe; startup/readiness stay on/teston purpose so a DB blip doesn't pull the pod from the Service. API pods are still on/test.- Watchdog —
MongoService.startWatchdog()pings everyMONGO_WATCHDOG_INTERVAL_MS(30 s, timeoutMONGO_WATCHDOG_PING_TIMEOUT_MS10 s) andprocess.exit(1)s afterMONGO_WATCHDOG_MAX_FAILURES(5) consecutive failures. Kill switchMONGO_WATCHDOG_ENABLED=false. - CronMonitor streak — consecutive DB-connectivity fatals across all
jobs (
CronMonitor.isDbConnectivityError); any successful run resets it; atCRON_DB_FATAL_EXIT_THRESHOLD(12, deliberately above the watchdog's) the process exits. Kill switchCRON_DB_FATAL_EXIT=false.
Where to look when…
| Symptom | First place |
|---|---|
| Request errors in backend | Sentry, then /logs |
| "Encryption key … differs" / tenant secrets unreadable | backend logs [vault] key-mismatch line — a pod's SECRET_ENCRYPTION_KEY differs from the one recorded in credential_keys (Security & Secrets) |
| AI answer wrong/slow | Grafana Sigil traces + ms-ai logs (Loki) |
| Recording not processed | ms-sessions Bull Board (recording-analysis queue) |
| CRM sync gap | backend CronMonitor (*GhlSyncJob) + CrmSyncQueueService |
| Webhook not delivered | Bull Board tenant-webhooks + webhook_logs (attempt N, disabled_reason) |
| Cost spike | GET /token-usage, cost-tracking dashboards |
| Mongo timeouts / slow ops | Loki {job="azure-db-diagnostics"} (db-log-shipper) |
| Every cron failing | schedular restarts / GET /healthz/db on the pod; CronMonitor db-fatal streak lines |
| "Who changed this setting?" | Admin Audit Trail (/admin/audit-logs, 180 days) — see Audit Trail |
| A call misbehaved (ringing, join, hang-up) | Admin Call Logs (/admin/call-logs) — per-call step log from server, sockets, widget and dashboard |
| Slow endpoint: Mongo or pod? | Server-Timing db / db.slowest response header, then Loki azure-db-diagnostics |
| Dashboard stopped getting live events | socket handshake missing auth.token → never joined the company room; a rejected token gets an unauthorized event and the dashboard reconnects once a minute (Socket room auth) |
| Live session never loads | browser console [LiveSession] lines: "Server has no stored snapshot" = the widget's snapshot never reached ms-sessions; "Large live event could not be fetched" = pointer expired (120 s) — see Session Recording › Replay |
| Gemini S2S call went silent | ms-comm STALL / commit log lines (vad_state, ws_buffered_bytes, gemini_heard) — see Voice Engine › Direct mode |
List shorter than total | X-Result-Limit-Capped response header — see Backend |