Skip to main content

Observability

Each service initializes observability first (the otel.js / otel.ts required at the top of the entry file) so instrumentation wraps everything.

Tracing & metrics​

  • OpenTelemetry — all Node services export OTLP traces. ms-ai adds auto-instrumentation plus Traceloop instrumentation for Anthropic/OpenAI calls.
  • Prometheus + Grafana + AlertManager — cluster metrics and alerting (Infrastructure/Monitoring/Prom-stack in flux-deployments), including custom alerting and probe (health-check) rules.
  • Grafana dashboards — pre-built dashboards shipped via Flux.

LLM-specific (ms-ai)​

  • Grafana Pyroscope — continuous profiling (GRAFANA_PYROSCOPE_*).
  • Grafana Sigil — LLM tracing (token usage, prompt/response spans), used by the Knox/agent code paths (sigilGenerateText).
  • AI cost tracking — cost-tracking module records per-provider token usage and pricing; daily reconciliation; queryable via GET /token-usage/:company_id.

Logs​

  • Winston in the Nodevel services (backend exposes a basic-auth /logs viewer; cleanup cron prunes old EmailLog/ErrorLog).
  • Loki aggregates logs cluster-wide (Infrastructure/Monitoring/Loki); winston-loki ships logs from services.
  • PostHog Logs are no longer used by the backend — the last posthog_helper.log call sites were removed on 2026-08-25 (f59478165). The helper class stays for the next module that needs prod-visible debugging; the policy (add for the module under debugging, remove afterwards) is unchanged.

Azure database diagnostic logs → Loki​

Cosmos Mongo and Postgres emit diagnostic logs to the kkdblogsprod storage account (insights-logs-vcoremongorequests, insights-logs-postgresqlflexquerystoreruntime). They reach Loki through Infrastructure/Monitoring/LogShipper: a CronJob (db-log-shipper, every 5 minutes, azure-cli image running ship.py) that does a HEAD + range-GET per source for the current hour's blob, pushes the new bytes to Loki in rate-limit-sized chunks under job="azure-db-diagnostics" with db, category, and operationName labels, and stores its byte offset in the log-shipper-checkpoints container. A checkpoint only advances after a successful push, so a Loki outage replays rather than drops.

The first attempt — a Grafana Alloy deployment tailing the blobs through a blobfuse2 CSI mount (Infrastructure/Monitoring/Alloy) — is kept in the repo but suspended (suspend: true, 2026-08-20): the mount's directory polling generated enough storage API calls to cost $700+ NZD per week. Don't un-suspend it; extend the CronJob instead.

Query in Grafana with {job="azure-db-diagnostics", db="cosmos-mongo"} — this is where connection N to <ip> timed out style Mongo timeouts and slow-query runtime records are visible.

Error reporting​

  • Sentry (@sentry/node + profiling) in backend, initialized in otel.js with BUGSINK_DSN; tracing/profiling enabled in prod.

Job & queue visibility​

  • CronMonitor — backend's Redis-backed cron run history at /admin/cron.
  • Bull Board — BullMQ queue dashboards at /admin/queues (backend and ms-ai, basic auth). Backend queues now include tenant-webhooks (custom webhook + Zapier delivery, 4 attempts) alongside email and CRM sync.
  • Socket.IO admin UI — @socket.io/admin-ui on the realtime services (dev).

Health checks​

Each service exposes liveness/readiness probes (e.g. ms-ai GET /health/live, /health/ready checking DB + Redis; the Nodevel services use GET /). Prometheus probe-alerts watch these.

Where to look when…​

SymptomFirst place
Request errors in backendSentry, then /logs
AI answer wrong/slowGrafana Sigil traces + ms-ai logs (Loki)
Recording not processedms-sessions Bull Board (recording-analysis queue)
CRM sync gapbackend CronMonitor (*GhlSyncJob) + CrmSyncQueueService
Webhook not deliveredBull Board tenant-webhooks + webhook_logs (attempt N, disabled_reason)
Cost spikeGET /token-usage, cost-tracking dashboards
Mongo timeouts / slow opsLoki {job="azure-db-diagnostics"} (db-log-shipper)