Observability

Each service initializes observability first (the otel.js / otel.ts required at the top of the entry file) so instrumentation wraps everything.

Tracing & metrics

• OpenTelemetry — all Node services export OTLP traces. ms-ai adds auto-instrumentation plus Traceloop instrumentation for Anthropic/OpenAI calls.
• Prometheus + Grafana + AlertManager — cluster metrics and alerting (Infrastructure/Monitoring/Prom-stack in flux-deployments), including custom alerting and probe (health-check) rules.
• Grafana dashboards — pre-built dashboards shipped via Flux.

LLM-specific (ms-ai)

• Grafana Pyroscope — continuous profiling (GRAFANA_PYROSCOPE_*).
• Grafana Sigil — LLM tracing (token usage, prompt/response spans), used by the Knox/agent code paths (sigilGenerateText).
• AI cost tracking — cost-tracking module records per-provider token usage and pricing; daily reconciliation; queryable via GET /token-usage/:company_id.

Logs

• Winston in the Nodevel services (backend exposes a basic-auth /logs viewer; cleanup cron prunes old EmailLog/ErrorLog).
• Loki aggregates logs cluster-wide (Infrastructure/Monitoring/Loki); winston-loki ships logs from services.

Error reporting

• Sentry (@sentry/node + profiling) in backend, initialized in otel.js with BUGSINK_DSN; tracing/profiling enabled in prod.

Job & queue visibility

• CronMonitor — backend's Redis-backed cron run history at /admin/cron.
• Bull Board — BullMQ queue dashboards at /admin/queues (backend and ms-ai, basic auth).
• Socket.IO admin UI — @socket.io/admin-ui on the realtime services (dev).

Health checks

Each service exposes liveness/readiness probes (e.g. ms-ai GET /health/live, /health/ready checking DB + Redis; the Nodevel services use GET /). Prometheus probe-alerts watch these.

Where to look when…

Symptom	First place
Request errors in backend	Sentry, then /logs
AI answer wrong/slow	Grafana Sigil traces + ms-ai logs (Loki)
Recording not processed	ms-sessions Bull Board (recording-analysis queue)
CRM sync gap	backend CronMonitor (*GhlSyncJob) + CrmSyncQueueService
Cost spike	GET /token-usage, cost-tracking dashboards