ms-ai — AI Service
ms-ai is the AI brain of the platform. Unlike the other Node services it is
NestJS + TypeScript + Postgres/pgvector, not Nodevel/Mongo.
- Stack: NestJS 11, TypeORM + Postgres 17 + pgvector, Redis + BullMQ, Socket.IO for streaming.
- LLMs: Anthropic, OpenAI (incl.
@openai/agents), Mistral, Cohere (rerank), ElevenLabs (voice). Multi-provider routing. - Observability: OpenTelemetry + Traceloop, Grafana Pyroscope (profiling), Grafana Sigil (LLM tracing).
Responsibilities
- AI chat agent — streaming/non-streaming answers grounded in RAG + identity context; auto-trigger messages; chat summaries; contact extraction.
- AI voice agent (Knox Voice) — ElevenLabs telephony with real-time tool use, post-call webhooks, transcript/summary; plus native speech-to-speech calls (Voice Engine).
- Nox/Knox admin assistant — the operator-facing assistant (intent → sub-agent → tools). See Nox Assistant.
- Knowledge base / RAG — document ingestion, dual embeddings, pgvector + Cohere rerank. See Knowledge-Base RAG.
- Integrations — Shopify/WooCommerce product sync, Google/Calendly/GHL calendars, fact extraction.
- AI cost tracking — per-provider token usage + pricing.
Module map (src/)
| Module | Purpose |
|---|---|
agents/ (KnockAgentsModule) | Chat agent: ask-question (v1/v2 streaming), auto-trigger, summary, contact extraction |
knox-agent/ (KnoxAgentModule) | The Nox admin assistant — intent classifier → 6 sub-agents → tools |
knox-v2/ (KnoxV2Module) | Nox engine v2 — planner → researchers/writers → checked actions, runs on knoxV2RunQueue; see Nox engine v2 |
knox-voice/ (KnoxVoiceModule) | Voice agent orchestration, ElevenLabs tool bridge, call context store |
elevenlabs/ | Telephony: start calls, webhooks, phone numbers, audio/summary |
voice-engine/ | Speech-to-speech (S2S) calls: routing, company gate, direct/ vendor minting + sideband, vapi/, ms-voice-engine sessions, recording policy — see Voice Engine |
integrations/ | Shopify, WooCommerce, Google Calendar, Calendly, GHL calendar, Simpro |
queues/ (QueueModule) | BullMQ processors: scraping, embeddings, PDF, product sync, Knox docs/facts, narration |
company/ | Company context, embeddings, RAG queries (Cohere rerank) |
cost-tracking/ | Token usage, pricing, daily reconciliation |
fact-extraction/ | Structured facts from chat/call transcripts |
identity-graph/ | gRPC client to backend for visitor identity |
knox-signals/ | Recording-analysis insight signals |
auth/ | JWT, API-key (secret), webhook HMAC guards; throttler |
core/health, core/cache, core/ai-provider | Probes, Redis cache, provider/pricing config |
API surface
~45 endpoints (see API_ENDPOINTS.md). Highlights:
| Area | Endpoints |
|---|---|
| Agents | POST /agents/ask-question, POST /agents/v2/ask-question (SSE), /agents/auto-trigger-message, /agents/extract-contact-details |
| Knowledge | POST /context/.../:company_id, POST /create-embeddings, POST /enqueue-embeddings |
| Calendar tools | POST /tools/calendar/:id/availability, /tools/calendar/:id/booking (public, rate-limited) |
| Voice | POST /elevenlabs/start-audio-call, /start-outbound-call, POST /elevenlabs/webhook/post-call |
| S2S voice | POST /voice-direct/{tool,heartbeat,webrtc-offer,refresh,setup,usage,session-end}, POST /voice-engine/telephony/session, POST /voice-engine/session-callback, GET /voice-engine/sessions/:id/recording, POST /voice-engine/company-gate/:company_id/revoke-inbound, POST /voice-vapi/webhook |
| Integrations | POST /integrations/shopify, /integrations/woocommerce, /integrations/actions/google-calendar |
| Billing | GET /token-usage/:company_id, POST /ai-model-pricing |
| Admin | GET /docs (Swagger), GET /admin/queues (Bull Board) — basic auth |
Auth types: jwt (browser), secret (service-to-service via x-api-key/
Bearer), public (rate-limited), shopify/elevenlabs (HMAC over raw body).
Background processing (BullMQ)
| Queue | Purpose |
|---|---|
scrappingQueue | fetch & parse URLs |
embeddingQueue | generate embeddings |
shopiProductsQueue / wooProductsQueue | product catalog sync |
pdfExtractionQueue | PDF text extraction |
knoxDocumentQueue | KB ingestion → chunk → embed |
factExtractionQueue | facts from transcripts |
knoxNarrationQueue | narration/audio summaries |
knoxV2RunQueue | Nox engine v2 runs (knox-v2/run/) — see Nox engine v2 |
Scheduled jobs (@nestjs/schedule): daily cost estimate refresh, 5-min webhook
retry, daily ElevenLabs catch-up, weekly cost-table prune. Telephony billing
crons (src/elevenlabs/number-rental.scheduler.ts, kill switch
NUMBER_RENTAL_CRON_ENABLED=false): number-rental:meter-due (daily 06:00
UTC, monthly number rental, idempotent on (number, period)),
twilio-call-cost:meter (10 min, idempotent on CallSid),
number-rental:suspension-sweep (15 min, suspend → warn → release after 7 days).
Recent additions (2026-07)
- Cost-tracking:
POST /api/cost-tracking/internal/provider-usage— flat-USD provider usage from the backend'sAgentKUsageEmitJob, converted to credits at $1 = 100, idempotent onagentk:<co>:<provider>:<day>; priced bysrc/cost-tracking/agent-k-provider-pricing.ts(AGENT_K_PROVIDER_USDenv override; apollo_match $0.05, predictleads $0.02, unipile $0.01, apollo_search free).recordFlatUsdChargeis the generic mechanism — telephony charges use it too, under separate providers. Newagent_kusage agent +knox_intentcategory split; Baseten provider added with GLM-4.7 at 0 credits (platform-absorbed);company_id: 'platform'rows are never emitted. Drift calculation now follows the user-selected window. See cost-tracking. - Telephony wallet integration: ms-ai gates number purchase/calls on the
backend wallet (
GET /internal/telephony/wallet/:company_id, marked-up prices) — see Wallet. - White-label Nox voice: agency
nox_voice_idapplied as a per-session TTS override on the shared ElevenLabs agent.
Recent additions (2026-08)
- SMS agent:
POST /elevenlabs/send-sms+/elevenlabs/compose-sms, SMS phone-number import/buy (/elevenlabs/import-phone-number/twilio/sms);KnownAgent.SMScost bucket;meterTwilioSmsCostssweep meters Twilio SMS alongside call costs (elevenlabs.service.ts). - New Knox tools:
knox_engage_visitor(message a live visitor),knox_schedule_sms,knox_list_scheduled_actions(renamed fromknox_list_scheduled_calls),knox_brief_offers,knox_agent_k_weights(src/knox-agent/tools/). - Tool markdown:
src/knox-agent/services/knox-tool-markdown.tscustom renderers surface ids/URLs/friction/local time verbatim; fixes nested arrays being dropped from tool output (tests in__tests__/knox-tool-markdown.spec.ts). - First message:
FirstMessageAgent(src/modules/agents/definitions/first-message.agent.ts) +POST /agents/generate-first-message(public, throttled). - Transcript translation:
POST /translate-transcript(app.controller.ts) — chunked translation, costed underKnownAgent.TRANSCRIPT_TRANSLATE. - Agent sync skip-cache: unchanged ElevenLabs agent syncs are skipped
(
AGENT_SYNC_CACHE_TTL_MS, default 5 min;invalidateAgentSyncCacheon settings writes). Also: AMD request params (_appendAmdParams), multi-calendarcalendar_idtool param +sync-calendar-tools, multilingual preset propagation fixed inupdateAgent, native background sound. - Knox voice / iOS: signed-URL response now carries a
conversation_tokenfor the native iOS SDK. - KB: collection delete cascades to documents + chunks; creating with the
same slug revives a soft-deleted collection. Mistral OCR verbatim PDF
transcription (
src/pdf-extraction/providers/mistral-ocr.service.ts). - Cost-tracking leader locks: Postgres advisory locks
4823001/4823002(src/cost-tracking/schedulers/cost-tracking.scheduler.ts) sorefresh-estimates(03:00 UTC) andelevenlabs-catchup(04:15 UTC) run once across replicas. - Agent K:
POST /agent-k/classify-reply— reply stop-intent classifier (agent-k-reply.service.ts, model viaAGENT_K_STOP_MODEL). - Internal endpoints: narration read endpoint for the recorded-sessions UI
(
recording/recording-api.controller.ts), internal action-events query + tenant conversation-thread endpoints (knox-agent.controller.ts), Nox activity-history feed, and engage delivery receipts.
Recent additions (2026-08-11 → 09-09)
- Nox settings-by-chat:
src/knox-agent/config-registry/(56 ops, backend catalog resolver, lexical search, Redis preview tokens) +knox_config_describe/apply/invoketools;npm run nox:catalog:sync. See Nox › Settings-by-chat. - Morning Standup:
agents/knox-standup.agent.ts,tools/knox-standup.tools.ts(knox_standup_tasks,knox_confirm_run,knox_cancel_task,knox_approve_task,knox_resolve_reference),services/knox-run-state.store.ts(Redis run state, 2h TTL); day-keyed run ids. Canvas:services/knox-canvas.tstyped cards (priorities,stage_contacts,meetings,pipelines,opportunities), per-turnKnoxActionRegistrywith timestamp-seeded ids. - New tools:
knox_visitor_forms,knox_call_status,knox_agent_k_pipeline,knox_discard_followup;knox_schedule_smsGHL route; 57 tools total. - Caller authentication tool
company_user_authentication(src/elevenlabs/company-auth-tool.*), prompt placement first + last,redact-auth-transcript.tsat the top ofpostCallWebhook. See Telephony › Caller authentication. - Custom LLM for ElevenLabs:
src/elevenlabs/custom-llm.service.ts+call-models.ts(NATIVE vs CUSTOM ids; Baseten via ElevenLabs workspace secretsecretId, never a key in ms-ai; never throws). EnvELEVENLABS_CUSTOM_LLM_PROVIDER/ELEVENLABS_CUSTOM_LLM_MODEL.GET /elevenlabs/agent/:company_id?include_prompt=1. - Memory gating:
use_memoryin the agent-config hash; memory-off withholdsupsert_memory/delete_memory_slot(memory-tool-gating.spec.ts). - Voice hardening:
assertVoiceCallable()in agent sync, standup allowlist enforced in the tool webhook, "Never executed" post-call marking, tool-status labels on the signed-URL response. - Call events:
knox-signalspostsduration_secondson/call-events/endedand exposesGET /knox-signals/transcripts/durationsfor the backfill. - Crawler / scraping:
CRAWL_JOB_CONCURRENCY(default 3), scrapping-queue concurrency 10 → 50, sitemap parser + 4-phase native crawler (12b33293e), ScrapingBee tier state (CRAWLER_PROVIDERdefaultnative,CRAWL_STEALTH_RENDER_TIMEOUT_MS/SCRAPE_STEALTH_TIMEOUT_MS240 s,SCRAPE_STEALTH_BUDGET_PER_JOB5),POST /cancel-scrapping. - Google Calendar creds:
removeIntegrationrevokes + emitsremoved; refreshed tokens persisted, re-mirrored, Redis cache invalidated; tokens masked in webhook logs and DLQ (1495b356e). - Jobber accepted as a CRM + calendar provider (
cb9cd8414):CalendarType/pickCalendarType/ hosted tools include'jobber';book_meetingaccepts a Jobber client id. - Contact extraction: corrected value wins, spelled letters join
(
54d120273); Simprocreate_simpro_jobcarriescall_sidand asks for a comma-separated address (re-push the ElevenLabs tool). - Build: buildx + ACR
:buildcache, prod--pushwithout--load,npm run migration:run:prod.
Recent additions (2026-09-09 → 09-30)
- Speech-to-speech voice (
src/voice-engine/) — native S2S models (OpenAI Realtimegpt-realtime/-2/-2.1/-2.1-mini, Gemini Live 2.5 / 3.1 / 3.8 / 3.8 extended thinking, xAI Grok Voice) as a second voice stack beside ElevenLabs.VOICE_S2S_MODEpicks the path:direct(PROD — ms-ai mints a short-lived vendor credential, the widget or ms-comm connects straight to the vendor; OpenAI web via WebRTC relay + server sideband that runs tools),vapi(per-call transient Vapi assistant, at-cost billing, 5-min poll/reconcile, held-for-review),engine(self-hostedms-voice-engine, dormant),off. Routing:voice-engine.routing.ts(per-channel model,VOICE_ENGINE_ROLESkill switch, company gatevoice_engine_enabled+voice_engine_providers); every refusal falls back to an ElevenLabs standby agent. Recording policy (voice-engine-recording-policy.ts, fail-closed). Direct billing per vendor token usage (vd:<session_id>,voice-model-billing.ts). Full write-up: Voice Engine. - Pricing picker —
GET /ai-model-pricing/estimates/:company_iddrops S2S models the current mode can't serve (isS2SModelServed: direct needs the vendor key) and marks the rests2s: true;vapimode adds a~per_minuteestimate. - GLM 5.3 Flash (
baseten-glm-5.3-flash,zai-org/GLM-5.3-Flash) replaces GLM 4.7 as the Baseten custom LLM (ELEVENLABS_CUSTOM_LLM_MODELdefault); seederbaseten-glm-5-3-flash.seeder.ts. - SDK voice agents — each dashboard SDK voice agent syncs its own ElevenLabs web agent
sdk:<agent_key>; per-callinstructionsfrom the Headless SDK are merged into the call prompt (ms-comm builds it, never sent to the browser). - SMS numbers — one Twilio number can back several SMS agents in the same company: SMS rows use
number_id = <sid>:<agent_key>(legacy bare-SID rows still match); a number already connected to another company is refused with 409 (e990b845c). - Bounded bodies — the global JSON/urlencoded limit is 20 MB (was 2 GB), sized for the ms-sessions recording handoff;
POST /agents/classify-call-tagsnow requires the internal secret (only backend post-call jobs call it). See request body limits. - Seeders — the OpenAI Realtime, Grok, Gemini 3.1/3.8, per-minute S2S, whisper-1 /
gpt-4o-transcribetranscription and Vapi billing-catalog seeders are registered inrun-seeders.ts(an unregistered seeder never runs).
Data model
Postgres + pgvector. Knox knowledge (knox_knowledge_chunks with dual 1536-dim
embeddings + HNSW, knox_knowledge_documents, knox_knowledge_collections),
Knox conversations/memory, visitor insight (knox_visitor_profile,
knox_session_analysis), action audit, plus Company/Integration/AgentMemory/
PdfContent. Schema is migration-managed (migrations/, npm run migration:run).
See Data Stores.
Cross-service
- backend → ms-ai: HTTP with
Bearer MS_AI_SECRET_TOKENfor Knox sync, fact extraction, voice handoff. - ms-sessions → ms-ai: analyzed recordings handed off for enrichment.
- ms-ai → backend: identity-graph lookups (gRPC client) + callbacks.
Config & deployment
- Key env:
DATABASE_URL/POSTGRES_*,REDIS_*,ANTHROPIC_API_KEY,OPENAI_API_KEY,MISTRAL_API_KEY,COHERE_API_KEY,ELEVENLABS_API_KEY,DEFAULT_CHAT_MODEL,AUTO_TRIGGER_MODEL,BASIC_AUTH_*, OTLP/Pyroscope creds. S2S voice:VOICE_S2S_MODE(unset =off),VOICE_S2S_WEB_MODE,VOICE_ENGINE_ROLES,VOICE_DIRECT_WEB_VENDORS,VOICE_DIRECT_WEB_GEMINI_RELAY,VOICE_DIRECT_GEMINI_API_KEY(falls back toGEMINI_API_KEY) /XAI_API_KEY(a vendor without a key is never offered),VOICE_DIRECT_OPENAI_NOISE_REDUCTION,VOICE_DIRECT_OPENAI_EAGERNESS,VOICE_DIRECT_TRACE_COMPANIES,VOICE_ENGINE_RECORDING_S3_*,VAPI_*,VOICE_ENGINE_*— reference table in Voice Engine. Secrets at rest:SECRET_ENCRYPTION_KEYS/SECRET_ENCRYPTION_KEY_CURRENT(legacySECRET_ENCRYPTION_KEY= v1, in the ms-ai secret) — ms-ai's own keyring (src/core/crypto/secret-crypto.ts, same blob format as the backend's SecretCrypto, never the backend's keys). It encrypts Shopify/WooCommerce credentials (integrations.credentials) and Twilio/Telnyx keys (elabs_phone_number.meta) via column transformers (AAD<table>.<column>.<key>);POST /credentials/encrypt-at-rest(encrypt-at-rest.controller.ts, called in a loop by the backend's Encrypt ms-ai secrets card) encrypts leftovers, re-encrypts ones on an older key and drops the Google tokens ms-ai no longer uses. See Security & Secrets.CREDENTIAL_VAULT_DUAL_WRITE: 'false'in the PROD configmap is a leftover of the closed vault window. Since 2026-11-10 Nox platform switches (daily quality check, server-written call reports) come from the backend'sGET /internal/knox/platform-flags(Admin → Settings → Feature Flags), notNOX_QUALITY_ENABLED/NOX_VOICE_SERVER_REPORTS; resolved support tickets arrive atPOST /knox-v2/internal/product-notes/from-ticket. Configmap changes needkubectl rollout restart deployment/ms-ai-deployment. - CORS preflights are cached for 2 h (
maxAge: 7200inmain.ts,1fc15916d) — same in backend, ms-communication and ms-sessions. - Docker: 3-stage build (deps → build → runtime as non-root, port 3000).
entrypoint.shstarts the app directly — migrations run externally (CI), not in the container, to avoid multi-replica races. - Local:
docker-compose.ymlbrings up Postgres(+pgvector) + Redis.