Skip to main content

Observability

Each service initializes observability first (the otel.js / otel.ts required at the top of the entry file) so instrumentation wraps everything.

Tracing & metrics​

  • OpenTelemetry — all Node services export OTLP traces. ms-ai adds auto-instrumentation plus Traceloop instrumentation for Anthropic/OpenAI calls.
  • Prometheus + Grafana + AlertManager — cluster metrics and alerting (Infrastructure/Monitoring/Prom-stack in flux-deployments), including custom alerting and probe (health-check) rules. The hermes-knockknock Alertmanager webhook receiver was removed on 2026-09-30 (6898c723). Since 2026-11-10 (a4b93da2) every real alert is first mirrored to the alertmind webhook receiver (hooks.pagermind.com/webhooks/alertmanager-knockknockapp, send_resolved: true, continue: true) and then routed to Slack as before.
  • Server-Timing (backend) — mounted when DEBUG is true (it is in the PROD backend configmap), so every response carries total, db;dur=<ms>;desc="<n> queries" and db.slowest (duration + collection.op of the slowest Mongo query) — lib/dbTiming.js (bd38a0dae, 5a3dd3401). Read it in the browser's Network → Timing tab to tell Mongo time from pod/queue time; overlapping queries are summed, so db can exceed total.
  • Grafana dashboards — pre-built dashboards shipped via Flux.

LLM-specific (ms-ai)​

  • Grafana Pyroscope — continuous profiling (GRAFANA_PYROSCOPE_*).
  • Grafana Sigil — LLM tracing (token usage, prompt/response spans), used by the Knox/agent code paths (sigilGenerateText).
  • AI cost tracking — cost-tracking module records per-provider token usage and pricing; daily reconciliation; queryable via GET /token-usage/:company_id.

Logs​

  • Winston in the Nodevel services (backend exposes a basic-auth /logs viewer; cleanup cron prunes old EmailLog/ErrorLog).
  • Loki aggregates logs cluster-wide (Infrastructure/Monitoring/Loki); winston-loki ships logs from services.
  • PostHog Logs are no longer used by the backend — the last posthog_helper.log call sites were removed on 2026-08-25 (f59478165). The helper class stays for the next module that needs prod-visible debugging; the policy (add for the module under debugging, remove afterwards) is unchanged.

Azure database diagnostic logs → Loki​

Cosmos Mongo and Postgres emit diagnostic logs to the kkdblogsprod storage account (insights-logs-vcoremongorequests, insights-logs-postgresqlflexquerystoreruntime). They reach Loki through Infrastructure/Monitoring/LogShipper: a CronJob (db-log-shipper, every minute, azure-cli image running ship.py from the configmap) that walks each source forward one hour-blob at a time from its checkpoint to the live hour, range-GETting CHUNK_READ_BYTES (15 MB) at a time and pushing to Loki in ≤ 2 MB chunks under job="azure-db-diagnostics" with db, category, and operationName labels. The checkpoint ({year, month, day, hour, offset} per source in the log-shipper-checkpoints container; the legacy {blob, offset} shape is migrated on read) advances after every chunk, so a crash loses at most one chunk and a Loki outage replays rather than drops. Each run self-limits to RUNTIME_BUDGET_SECONDS (240, under the pod's activeDeadlineSeconds 300, 1 Gi memory), split evenly across sources so a big Mongo backlog can't starve Postgres.

History (2026-09-18): the old version read a whole hour-blob into memory and OOM-crash-looped for 12 days (one hour was 547 MB). Loki rejects samples older than its ~7-day reject_old_samples_max_age (separate from the 30-day blob retention), so a "timestamp too old" 400 is treated as handled and the checkpoint jumps straight to the cutoff parsed from Loki's response, rounded up to the next hour. Running every minute is cost-neutral — storage transactions scale with bytes moved, not run frequency.

The first attempt — a Grafana Alloy deployment tailing the blobs through a blobfuse2 CSI mount (Infrastructure/Monitoring/Alloy) — is kept in the repo but suspended (suspend: true, 2026-08-20): the mount's directory polling generated enough storage API calls to cost $700+ NZD per week. Don't un-suspend it; extend the CronJob instead.

Grafana's Loki datasource is configured by hand, not via GitOps (as are the Azure Monitor and two Postgres datasources) — a sidecar-provisioned "Loki" datasource was added and reverted on 2026-09-18 because the name clash broke the provisioning reload. Check /api/datasources, not the Helm configmap, before adding one.

Query in Grafana with {job="azure-db-diagnostics", db="cosmos-mongo"} — this is where connection N to <ip> timed out style Mongo timeouts and slow-query runtime records are visible.

Error reporting​

  • Sentry (@sentry/node + profiling) in backend, initialized in otel.js with BUGSINK_DSN (PROD backend configmap; the name is historical — it is a Sentry ingest DSN). Unset, or ENV not staging/production → Sentry is not initialised. Staging sends to BUGSINK_DSN; production sends to a DSN hardcoded in otel.js (so BUGSINK_DSN there only acts as the on switch). Socket.io and signed-recording paths are never traced. Tracing/profiling enabled in prod; SENTRY_ENABLED is set in the backend, schedular, ms-sessions and ms-communication configmaps.
  • No personal data (a4a132393, GDPR): sendDefaultPii: false (no automatic IP, cookies or AI prompt/response content) and app/lib/sentryScrub.js drops request bodies, the client IP and event.user from every event.

Job & queue visibility​

  • CronMonitor — backend's Redis-backed cron run history at /admin/cron. Each firing (and manual trigger) first claims cron:claim:<job>:<minute> in Redis, so only one schedular replica runs it; a missing claim is skipped silently, and without Redis it fails open. Interval jobs fire at a fixed per-job offset inside their interval (not all on :00), and the claim slot is centred on that offset — so "next run" times in the monitor are staggered by design (see Deployment › per-service facts).
  • Bull Board — BullMQ queue dashboards at /admin/queues (backend and ms-ai, basic auth). Backend queues now include tenant-webhooks (custom webhook + Zapier delivery, 4 attempts) alongside email and CRM sync.
  • Socket.IO admin UI — @socket.io/admin-ui on the realtime services (dev).

Health checks​

Each service exposes liveness/readiness probes (e.g. ms-ai GET /health/live, /health/ready checking DB + Redis; the Nodevel services use a static GET / or /test). Prometheus probe-alerts watch these.

Backend MongoDB self-healing​

fd3ce2873. On 2026-09-19 the schedular pod's driver pool sat in MongoServerSelectionError: connection N to <host> closed for ~20 h: mongoose emitted no event, readyState stayed 1, the static /test probe stayed green, and every cron run failed. Three layers now catch it:

  • GET /healthz/db (app/routes/api.js) — a real ping round-trip (HEALTHZ_DB_TIMEOUT_MS, default 5 s) returning 200/503 with ping_ms, ready_state and the watchdog counters. It is the schedular's livenessProbe; startup/readiness stay on /test on purpose so a DB blip doesn't pull the pod from the Service. API pods are still on /test.
  • Watchdog — MongoService.startWatchdog() pings every MONGO_WATCHDOG_INTERVAL_MS (30 s, timeout MONGO_WATCHDOG_PING_TIMEOUT_MS 10 s) and process.exit(1)s after MONGO_WATCHDOG_MAX_FAILURES (5) consecutive failures. Kill switch MONGO_WATCHDOG_ENABLED=false.
  • CronMonitor streak — consecutive DB-connectivity fatals across all jobs (CronMonitor.isDbConnectivityError); any successful run resets it; at CRON_DB_FATAL_EXIT_THRESHOLD (12, deliberately above the watchdog's) the process exits. Kill switch CRON_DB_FATAL_EXIT=false.

Where to look when…​

SymptomFirst place
Request errors in backendSentry, then /logs
"Encryption key … differs" / tenant secrets unreadablebackend logs [vault] key-mismatch line — a pod's SECRET_ENCRYPTION_KEY differs from the one recorded in credential_keys (Security & Secrets)
AI answer wrong/slowGrafana Sigil traces + ms-ai logs (Loki)
Recording not processedms-sessions Bull Board (recording-analysis queue)
CRM sync gapbackend CronMonitor (*GhlSyncJob) + CrmSyncQueueService
Webhook not deliveredBull Board tenant-webhooks + webhook_logs (attempt N, disabled_reason)
Cost spikeGET /token-usage, cost-tracking dashboards
Mongo timeouts / slow opsLoki {job="azure-db-diagnostics"} (db-log-shipper)
Every cron failingschedular restarts / GET /healthz/db on the pod; CronMonitor db-fatal streak lines
"Who changed this setting?"Admin Audit Trail (/admin/audit-logs, 180 days) — see Audit Trail
A call misbehaved (ringing, join, hang-up)Admin Call Logs (/admin/call-logs) — per-call step log from server, sockets, widget and dashboard
Slow endpoint: Mongo or pod?Server-Timing db / db.slowest response header, then Loki azure-db-diagnostics
Dashboard stopped getting live eventssocket handshake missing auth.token → never joined the company room; a rejected token gets an unauthorized event and the dashboard reconnects once a minute (Socket room auth)
Live session never loadsbrowser console [LiveSession] lines: "Server has no stored snapshot" = the widget's snapshot never reached ms-sessions; "Large live event could not be fetched" = pointer expired (120 s) — see Session Recording › Replay
Gemini S2S call went silentms-comm STALL / commit log lines (vad_state, ws_buffered_bytes, gemini_heard) — see Voice Engine › Direct mode
List shorter than totalX-Result-Limit-Capped response header — see Backend