Skip to main content

ms-ai — AI Service

ms-ai is the AI brain of the platform. Unlike the other Node services it is NestJS + TypeScript + Postgres/pgvector, not Nodevel/Mongo.

  • Stack: NestJS 11, TypeORM + Postgres 17 + pgvector, Redis + BullMQ, Socket.IO for streaming.
  • LLMs: Anthropic, OpenAI (incl. @openai/agents), Mistral, Cohere (rerank), ElevenLabs (voice). Multi-provider routing.
  • Observability: OpenTelemetry + Traceloop, Grafana Pyroscope (profiling), Grafana Sigil (LLM tracing).

Responsibilities​

  • AI chat agent — streaming/non-streaming answers grounded in RAG + identity context; auto-trigger messages; chat summaries; contact extraction.
  • AI voice agent (Knox Voice) — ElevenLabs telephony with real-time tool use, post-call webhooks, transcript/summary; plus native speech-to-speech calls (Voice Engine).
  • Nox/Knox admin assistant — the operator-facing assistant (intent → sub-agent → tools). See Nox Assistant.
  • Knowledge base / RAG — document ingestion, dual embeddings, pgvector + Cohere rerank. See Knowledge-Base RAG.
  • Integrations — Shopify/WooCommerce product sync, Google/Calendly/GHL calendars, fact extraction.
  • AI cost tracking — per-provider token usage + pricing.

Module map (src/)​

ModulePurpose
agents/ (KnockAgentsModule)Chat agent: ask-question (v1/v2 streaming), auto-trigger, summary, contact extraction
knox-agent/ (KnoxAgentModule)The Nox admin assistant — intent classifier → 6 sub-agents → tools
knox-v2/ (KnoxV2Module)Nox engine v2 — planner → researchers/writers → checked actions, runs on knoxV2RunQueue; see Nox engine v2
knox-voice/ (KnoxVoiceModule)Voice agent orchestration, ElevenLabs tool bridge, call context store
elevenlabs/Telephony: start calls, webhooks, phone numbers, audio/summary
voice-engine/Speech-to-speech (S2S) calls: routing, company gate, direct/ vendor minting + sideband, vapi/, ms-voice-engine sessions, recording policy — see Voice Engine
integrations/Shopify, WooCommerce, Google Calendar, Calendly, GHL calendar, Simpro
queues/ (QueueModule)BullMQ processors: scraping, embeddings, PDF, product sync, Knox docs/facts, narration
company/Company context, embeddings, RAG queries (Cohere rerank)
cost-tracking/Token usage, pricing, daily reconciliation
fact-extraction/Structured facts from chat/call transcripts
identity-graph/gRPC client to backend for visitor identity
knox-signals/Recording-analysis insight signals
auth/JWT, API-key (secret), webhook HMAC guards; throttler
core/health, core/cache, core/ai-providerProbes, Redis cache, provider/pricing config

API surface​

~45 endpoints (see API_ENDPOINTS.md). Highlights:

AreaEndpoints
AgentsPOST /agents/ask-question, POST /agents/v2/ask-question (SSE), /agents/auto-trigger-message, /agents/extract-contact-details
KnowledgePOST /context/.../:company_id, POST /create-embeddings, POST /enqueue-embeddings
Calendar toolsPOST /tools/calendar/:id/availability, /tools/calendar/:id/booking (public, rate-limited)
VoicePOST /elevenlabs/start-audio-call, /start-outbound-call, POST /elevenlabs/webhook/post-call
S2S voicePOST /voice-direct/{tool,heartbeat,webrtc-offer,refresh,setup,usage,session-end}, POST /voice-engine/telephony/session, POST /voice-engine/session-callback, GET /voice-engine/sessions/:id/recording, POST /voice-engine/company-gate/:company_id/revoke-inbound, POST /voice-vapi/webhook
IntegrationsPOST /integrations/shopify, /integrations/woocommerce, /integrations/actions/google-calendar
BillingGET /token-usage/:company_id, POST /ai-model-pricing
AdminGET /docs (Swagger), GET /admin/queues (Bull Board) — basic auth

Auth types: jwt (browser), secret (service-to-service via x-api-key/ Bearer), public (rate-limited), shopify/elevenlabs (HMAC over raw body).

Background processing (BullMQ)​

QueuePurpose
scrappingQueuefetch & parse URLs
embeddingQueuegenerate embeddings
shopiProductsQueue / wooProductsQueueproduct catalog sync
pdfExtractionQueuePDF text extraction
knoxDocumentQueueKB ingestion → chunk → embed
factExtractionQueuefacts from transcripts
knoxNarrationQueuenarration/audio summaries
knoxV2RunQueueNox engine v2 runs (knox-v2/run/) — see Nox engine v2

Scheduled jobs (@nestjs/schedule): daily cost estimate refresh, 5-min webhook retry, daily ElevenLabs catch-up, weekly cost-table prune. Telephony billing crons (src/elevenlabs/number-rental.scheduler.ts, kill switch NUMBER_RENTAL_CRON_ENABLED=false): number-rental:meter-due (daily 06:00 UTC, monthly number rental, idempotent on (number, period)), twilio-call-cost:meter (10 min, idempotent on CallSid), number-rental:suspension-sweep (15 min, suspend → warn → release after 7 days).

Recent additions (2026-07)​

  • Cost-tracking: POST /api/cost-tracking/internal/provider-usage — flat-USD provider usage from the backend's AgentKUsageEmitJob, converted to credits at $1 = 100, idempotent on agentk:<co>:<provider>:<day>; priced by src/cost-tracking/agent-k-provider-pricing.ts (AGENT_K_PROVIDER_USD env override; apollo_match $0.05, predictleads $0.02, unipile $0.01, apollo_search free). recordFlatUsdCharge is the generic mechanism — telephony charges use it too, under separate providers. New agent_k usage agent + knox_intent category split; Baseten provider added with GLM-4.7 at 0 credits (platform-absorbed); company_id: 'platform' rows are never emitted. Drift calculation now follows the user-selected window. See cost-tracking.
  • Telephony wallet integration: ms-ai gates number purchase/calls on the backend wallet (GET /internal/telephony/wallet/:company_id, marked-up prices) — see Wallet.
  • White-label Nox voice: agency nox_voice_id applied as a per-session TTS override on the shared ElevenLabs agent.

Recent additions (2026-08)​

  • SMS agent: POST /elevenlabs/send-sms + /elevenlabs/compose-sms, SMS phone-number import/buy (/elevenlabs/import-phone-number/twilio/sms); KnownAgent.SMS cost bucket; meterTwilioSmsCosts sweep meters Twilio SMS alongside call costs (elevenlabs.service.ts).
  • New Knox tools: knox_engage_visitor (message a live visitor), knox_schedule_sms, knox_list_scheduled_actions (renamed from knox_list_scheduled_calls), knox_brief_offers, knox_agent_k_weights (src/knox-agent/tools/).
  • Tool markdown: src/knox-agent/services/knox-tool-markdown.ts custom renderers surface ids/URLs/friction/local time verbatim; fixes nested arrays being dropped from tool output (tests in __tests__/knox-tool-markdown.spec.ts).
  • First message: FirstMessageAgent (src/modules/agents/definitions/first-message.agent.ts) + POST /agents/generate-first-message (public, throttled).
  • Transcript translation: POST /translate-transcript (app.controller.ts) — chunked translation, costed under KnownAgent.TRANSCRIPT_TRANSLATE.
  • Agent sync skip-cache: unchanged ElevenLabs agent syncs are skipped (AGENT_SYNC_CACHE_TTL_MS, default 5 min; invalidateAgentSyncCache on settings writes). Also: AMD request params (_appendAmdParams), multi-calendar calendar_id tool param + sync-calendar-tools, multilingual preset propagation fixed in updateAgent, native background sound.
  • Knox voice / iOS: signed-URL response now carries a conversation_token for the native iOS SDK.
  • KB: collection delete cascades to documents + chunks; creating with the same slug revives a soft-deleted collection. Mistral OCR verbatim PDF transcription (src/pdf-extraction/providers/mistral-ocr.service.ts).
  • Cost-tracking leader locks: Postgres advisory locks 4823001/4823002 (src/cost-tracking/schedulers/cost-tracking.scheduler.ts) so refresh-estimates (03:00 UTC) and elevenlabs-catchup (04:15 UTC) run once across replicas.
  • Agent K: POST /agent-k/classify-reply — reply stop-intent classifier (agent-k-reply.service.ts, model via AGENT_K_STOP_MODEL).
  • Internal endpoints: narration read endpoint for the recorded-sessions UI (recording/recording-api.controller.ts), internal action-events query + tenant conversation-thread endpoints (knox-agent.controller.ts), Nox activity-history feed, and engage delivery receipts.

Recent additions (2026-08-11 → 09-09)​

  • Nox settings-by-chat: src/knox-agent/config-registry/ (56 ops, backend catalog resolver, lexical search, Redis preview tokens) + knox_config_describe/apply/invoke tools; npm run nox:catalog:sync. See Nox › Settings-by-chat.
  • Morning Standup: agents/knox-standup.agent.ts, tools/knox-standup.tools.ts (knox_standup_tasks, knox_confirm_run, knox_cancel_task, knox_approve_task, knox_resolve_reference), services/knox-run-state.store.ts (Redis run state, 2h TTL); day-keyed run ids. Canvas: services/knox-canvas.ts typed cards (priorities, stage_contacts, meetings, pipelines, opportunities), per-turn KnoxActionRegistry with timestamp-seeded ids.
  • New tools: knox_visitor_forms, knox_call_status, knox_agent_k_pipeline, knox_discard_followup; knox_schedule_sms GHL route; 57 tools total.
  • Caller authentication tool company_user_authentication (src/elevenlabs/company-auth-tool.*), prompt placement first + last, redact-auth-transcript.ts at the top of postCallWebhook. See Telephony › Caller authentication.
  • Custom LLM for ElevenLabs: src/elevenlabs/custom-llm.service.ts + call-models.ts (NATIVE vs CUSTOM ids; Baseten via ElevenLabs workspace secret secretId, never a key in ms-ai; never throws). Env ELEVENLABS_CUSTOM_LLM_PROVIDER / ELEVENLABS_CUSTOM_LLM_MODEL. GET /elevenlabs/agent/:company_id?include_prompt=1.
  • Memory gating: use_memory in the agent-config hash; memory-off withholds upsert_memory / delete_memory_slot (memory-tool-gating.spec.ts).
  • Voice hardening: assertVoiceCallable() in agent sync, standup allowlist enforced in the tool webhook, "Never executed" post-call marking, tool-status labels on the signed-URL response.
  • Call events: knox-signals posts duration_seconds on /call-events/ended and exposes GET /knox-signals/transcripts/durations for the backfill.
  • Crawler / scraping: CRAWL_JOB_CONCURRENCY (default 3), scrapping-queue concurrency 10 → 50, sitemap parser + 4-phase native crawler (12b33293e), ScrapingBee tier state (CRAWLER_PROVIDER default native, CRAWL_STEALTH_RENDER_TIMEOUT_MS / SCRAPE_STEALTH_TIMEOUT_MS 240 s, SCRAPE_STEALTH_BUDGET_PER_JOB 5), POST /cancel-scrapping.
  • Google Calendar creds: removeIntegration revokes + emits removed; refreshed tokens persisted, re-mirrored, Redis cache invalidated; tokens masked in webhook logs and DLQ (1495b356e).
  • Jobber accepted as a CRM + calendar provider (cb9cd8414): CalendarType / pickCalendarType / hosted tools include 'jobber'; book_meeting accepts a Jobber client id.
  • Contact extraction: corrected value wins, spelled letters join (54d120273); Simpro create_simpro_job carries call_sid and asks for a comma-separated address (re-push the ElevenLabs tool).
  • Build: buildx + ACR :buildcache, prod --push without --load, npm run migration:run:prod.

Recent additions (2026-09-09 → 09-30)​

  • Speech-to-speech voice (src/voice-engine/) — native S2S models (OpenAI Realtime gpt-realtime / -2 / -2.1 / -2.1-mini, Gemini Live 2.5 / 3.1 / 3.8 / 3.8 extended thinking, xAI Grok Voice) as a second voice stack beside ElevenLabs. VOICE_S2S_MODE picks the path: direct (PROD — ms-ai mints a short-lived vendor credential, the widget or ms-comm connects straight to the vendor; OpenAI web via WebRTC relay + server sideband that runs tools), vapi (per-call transient Vapi assistant, at-cost billing, 5-min poll/reconcile, held-for-review), engine (self-hosted ms-voice-engine, dormant), off. Routing: voice-engine.routing.ts (per-channel model, VOICE_ENGINE_ROLES kill switch, company gate voice_engine_enabled + voice_engine_providers); every refusal falls back to an ElevenLabs standby agent. Recording policy (voice-engine-recording-policy.ts, fail-closed). Direct billing per vendor token usage (vd:<session_id>, voice-model-billing.ts). Full write-up: Voice Engine.
  • Pricing picker — GET /ai-model-pricing/estimates/:company_id drops S2S models the current mode can't serve (isS2SModelServed: direct needs the vendor key) and marks the rest s2s: true; vapi mode adds a ~per_minute estimate.
  • GLM 5.3 Flash (baseten-glm-5.3-flash, zai-org/GLM-5.3-Flash) replaces GLM 4.7 as the Baseten custom LLM (ELEVENLABS_CUSTOM_LLM_MODEL default); seeder baseten-glm-5-3-flash.seeder.ts.
  • SDK voice agents — each dashboard SDK voice agent syncs its own ElevenLabs web agent sdk:<agent_key>; per-call instructions from the Headless SDK are merged into the call prompt (ms-comm builds it, never sent to the browser).
  • SMS numbers — one Twilio number can back several SMS agents in the same company: SMS rows use number_id = <sid>:<agent_key> (legacy bare-SID rows still match); a number already connected to another company is refused with 409 (e990b845c).
  • Bounded bodies — the global JSON/urlencoded limit is 20 MB (was 2 GB), sized for the ms-sessions recording handoff; POST /agents/classify-call-tags now requires the internal secret (only backend post-call jobs call it). See request body limits.
  • Seeders — the OpenAI Realtime, Grok, Gemini 3.1/3.8, per-minute S2S, whisper-1 / gpt-4o-transcribe transcription and Vapi billing-catalog seeders are registered in run-seeders.ts (an unregistered seeder never runs).

Data model​

Postgres + pgvector. Knox knowledge (knox_knowledge_chunks with dual 1536-dim embeddings + HNSW, knox_knowledge_documents, knox_knowledge_collections), Knox conversations/memory, visitor insight (knox_visitor_profile, knox_session_analysis), action audit, plus Company/Integration/AgentMemory/ PdfContent. Schema is migration-managed (migrations/, npm run migration:run). See Data Stores.

Cross-service​

  • backend → ms-ai: HTTP with Bearer MS_AI_SECRET_TOKEN for Knox sync, fact extraction, voice handoff.
  • ms-sessions → ms-ai: analyzed recordings handed off for enrichment.
  • ms-ai → backend: identity-graph lookups (gRPC client) + callbacks.

Config & deployment​

  • Key env: DATABASE_URL/POSTGRES_*, REDIS_*, ANTHROPIC_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, ELEVENLABS_API_KEY, DEFAULT_CHAT_MODEL, AUTO_TRIGGER_MODEL, BASIC_AUTH_*, OTLP/Pyroscope creds. S2S voice: VOICE_S2S_MODE (unset = off), VOICE_S2S_WEB_MODE, VOICE_ENGINE_ROLES, VOICE_DIRECT_WEB_VENDORS, VOICE_DIRECT_WEB_GEMINI_RELAY, VOICE_DIRECT_GEMINI_API_KEY (falls back to GEMINI_API_KEY) / XAI_API_KEY (a vendor without a key is never offered), VOICE_DIRECT_OPENAI_NOISE_REDUCTION, VOICE_DIRECT_OPENAI_EAGERNESS, VOICE_DIRECT_TRACE_COMPANIES, VOICE_ENGINE_RECORDING_S3_*, VAPI_*, VOICE_ENGINE_* — reference table in Voice Engine. Secrets at rest: SECRET_ENCRYPTION_KEYS / SECRET_ENCRYPTION_KEY_CURRENT (legacy SECRET_ENCRYPTION_KEY = v1, in the ms-ai secret) — ms-ai's own keyring (src/core/crypto/secret-crypto.ts, same blob format as the backend's SecretCrypto, never the backend's keys). It encrypts Shopify/WooCommerce credentials (integrations.credentials) and Twilio/Telnyx keys (elabs_phone_number.meta) via column transformers (AAD <table>.<column>.<key>); POST /credentials/encrypt-at-rest (encrypt-at-rest.controller.ts, called in a loop by the backend's Encrypt ms-ai secrets card) encrypts leftovers, re-encrypts ones on an older key and drops the Google tokens ms-ai no longer uses. See Security & Secrets. CREDENTIAL_VAULT_DUAL_WRITE: 'false' in the PROD configmap is a leftover of the closed vault window. Since 2026-11-10 Nox platform switches (daily quality check, server-written call reports) come from the backend's GET /internal/knox/platform-flags (Admin → Settings → Feature Flags), not NOX_QUALITY_ENABLED / NOX_VOICE_SERVER_REPORTS; resolved support tickets arrive at POST /knox-v2/internal/product-notes/from-ticket. Configmap changes need kubectl rollout restart deployment/ms-ai-deployment.
  • CORS preflights are cached for 2 h (maxAge: 7200 in main.ts, 1fc15916d) — same in backend, ms-communication and ms-sessions.
  • Docker: 3-stage build (deps → build → runtime as non-root, port 3000). entrypoint.sh starts the app directly — migrations run externally (CI), not in the container, to avoid multi-replica races.
  • Local: docker-compose.yml brings up Postgres(+pgvector) + Redis.