ms-ai — AI Service

ms-ai is the AI brain of the platform. Unlike the other Node services it is NestJS + TypeScript + Postgres/pgvector, not Nodevel/Mongo.

• Stack: NestJS 11, TypeORM + Postgres 17 + pgvector, Redis + BullMQ, Socket.IO for streaming.
• LLMs: Anthropic, OpenAI (incl. @openai/agents), Mistral, Cohere (rerank), ElevenLabs (voice). Multi-provider routing.
• Observability: OpenTelemetry + Traceloop, Grafana Pyroscope (profiling), Grafana Sigil (LLM tracing).

Responsibilities

• AI chat agent — streaming/non-streaming answers grounded in RAG + identity context; auto-trigger messages; chat summaries; contact extraction.
• AI voice agent (Knox Voice) — ElevenLabs telephony with real-time tool use, post-call webhooks, transcript/summary.
• Nox/Knox admin assistant — the operator-facing assistant (intent → sub-agent → tools). See Nox Assistant.
• Knowledge base / RAG — document ingestion, dual embeddings, pgvector + Cohere rerank. See Knowledge-Base RAG.
• Integrations — Shopify/WooCommerce product sync, Google/Calendly/GHL calendars, fact extraction.
• AI cost tracking — per-provider token usage + pricing.

Module map (src/)

Module	Purpose
agents/ (KnockAgentsModule)	Chat agent: ask-question (v1/v2 streaming), auto-trigger, summary, contact extraction
knox-agent/ (KnoxAgentModule)	The Nox admin assistant — intent classifier → 6 sub-agents → tools
knox-voice/ (KnoxVoiceModule)	Voice agent orchestration, ElevenLabs tool bridge, call context store
elevenlabs/	Telephony: start calls, webhooks, phone numbers, audio/summary
integrations/	Shopify, WooCommerce, Google Calendar, Calendly, GHL calendar, Simpro
queues/ (QueueModule)	BullMQ processors: scraping, embeddings, PDF, product sync, Knox docs/facts, narration
company/	Company context, embeddings, RAG queries (Cohere rerank)
cost-tracking/	Token usage, pricing, daily reconciliation
fact-extraction/	Structured facts from chat/call transcripts
identity-graph/	gRPC client to backend for visitor identity
knox-signals/	Recording-analysis insight signals
auth/	JWT, API-key (secret), webhook HMAC guards; throttler
core/health, core/cache, core/ai-provider	Probes, Redis cache, provider/pricing config

API surface

~45 endpoints (see API_ENDPOINTS.md). Highlights:

Area	Endpoints
Agents	POST /agents/ask-question, POST /agents/v2/ask-question (SSE), /agents/auto-trigger-message, /agents/extract-contact-details
Knowledge	POST /context/.../:company_id, POST /create-embeddings, POST /enqueue-embeddings
Calendar tools	POST /tools/calendar/:id/availability, /tools/calendar/:id/booking (public, rate-limited)
Voice	POST /elevenlabs/start-audio-call, /start-outbound-call, POST /elevenlabs/webhook/post-call
Integrations	POST /integrations/shopify, /integrations/woocommerce, /integrations/actions/google-calendar
Billing	GET /token-usage/:company_id, POST /ai-model-pricing
Admin	GET /docs (Swagger), GET /admin/queues (Bull Board) — basic auth

Auth types: jwt (browser), secret (service-to-service via x-api-key/ Bearer), public (rate-limited), shopify/elevenlabs (HMAC over raw body).

Background processing (BullMQ)

Queue	Purpose
scrappingQueue	fetch & parse URLs
embeddingQueue	generate embeddings
shopiProductsQueue / wooProductsQueue	product catalog sync
pdfExtractionQueue	PDF text extraction
knoxDocumentQueue	KB ingestion → chunk → embed
factExtractionQueue	facts from transcripts
knoxNarrationQueue	narration/audio summaries

Scheduled jobs (@nestjs/schedule): daily cost estimate refresh, 5-min webhook retry, daily ElevenLabs catch-up, weekly cost-table prune. Telephony billing crons (src/elevenlabs/number-rental.scheduler.ts, kill switch NUMBER_RENTAL_CRON_ENABLED=false): number-rental:meter-due (daily 06:00 UTC, monthly number rental, idempotent on (number, period)), twilio-call-cost:meter (10 min, idempotent on CallSid), number-rental:suspension-sweep (15 min, suspend → warn → release after 7 days).

Recent additions (2026-07)

• Cost-tracking: POST /api/cost-tracking/internal/provider-usage — flat-USD provider usage from the backend's AgentKUsageEmitJob, converted to credits at $1 = 100, idempotent on agentk:<co>:<provider>:<day>; priced by src/cost-tracking/agent-k-provider-pricing.ts (AGENT_K_PROVIDER_USD env override; apollo_match $0.05, predictleads $0.02, unipile $0.01, apollo_search free). recordFlatUsdCharge is the generic mechanism — telephony charges use it too, under separate providers. New agent_k usage agent + knox_intent category split; Baseten provider added with GLM-4.7 at 0 credits (platform-absorbed); company_id: 'platform' rows are never emitted. Drift calculation now follows the user-selected window. See cost-tracking.
• Telephony wallet integration: ms-ai gates number purchase/calls on the backend wallet (GET /internal/telephony/wallet/:company_id, marked-up prices) — see Wallet.
• White-label Nox voice: agency nox_voice_id applied as a per-session TTS override on the shared ElevenLabs agent.

Recent additions (2026-08)

• SMS agent: POST /elevenlabs/send-sms + /elevenlabs/compose-sms, SMS phone-number import/buy (/elevenlabs/import-phone-number/twilio/sms); KnownAgent.SMS cost bucket; meterTwilioSmsCosts sweep meters Twilio SMS alongside call costs (elevenlabs.service.ts).
• New Knox tools: knox_engage_visitor (message a live visitor), knox_schedule_sms, knox_list_scheduled_actions (renamed from knox_list_scheduled_calls), knox_brief_offers, knox_agent_k_weights (src/knox-agent/tools/).
• Tool markdown: src/knox-agent/services/knox-tool-markdown.ts custom renderers surface ids/URLs/friction/local time verbatim; fixes nested arrays being dropped from tool output (tests in __tests__/knox-tool-markdown.spec.ts).
• First message: FirstMessageAgent (src/modules/agents/definitions/first-message.agent.ts) + POST /agents/generate-first-message (public, throttled).
• Transcript translation: POST /translate-transcript (app.controller.ts) — chunked translation, costed under KnownAgent.TRANSCRIPT_TRANSLATE.
• Agent sync skip-cache: unchanged ElevenLabs agent syncs are skipped (AGENT_SYNC_CACHE_TTL_MS, default 5 min; invalidateAgentSyncCache on settings writes). Also: AMD request params (_appendAmdParams), multi-calendar calendar_id tool param + sync-calendar-tools, multilingual preset propagation fixed in updateAgent, native background sound.
• Knox voice / iOS: signed-URL response now carries a conversation_token for the native iOS SDK.
• KB: collection delete cascades to documents + chunks; creating with the same slug revives a soft-deleted collection. Mistral OCR verbatim PDF transcription (src/pdf-extraction/providers/mistral-ocr.service.ts).
• Cost-tracking leader locks: Postgres advisory locks 4823001/4823002 (src/cost-tracking/schedulers/cost-tracking.scheduler.ts) so refresh-estimates (03:00 UTC) and elevenlabs-catchup (04:15 UTC) run once across replicas.
• Agent K: POST /agent-k/classify-reply — reply stop-intent classifier (agent-k-reply.service.ts, model via AGENT_K_STOP_MODEL).
• Internal endpoints: narration read endpoint for the recorded-sessions UI (recording/recording-api.controller.ts), internal action-events query + tenant conversation-thread endpoints (knox-agent.controller.ts), Nox activity-history feed, and engage delivery receipts.

Recent additions (2026-08-11 → 09-09)

• Nox settings-by-chat: src/knox-agent/config-registry/ (56 ops, backend catalog resolver, lexical search, Redis preview tokens) + knox_config_describe/apply/invoke tools; npm run nox:catalog:sync. See Nox › Settings-by-chat.
• Morning Standup: agents/knox-standup.agent.ts, tools/knox-standup.tools.ts (knox_standup_tasks, knox_confirm_run, knox_cancel_task, knox_approve_task, knox_resolve_reference), services/knox-run-state.store.ts (Redis run state, 2h TTL); day-keyed run ids. Canvas: services/knox-canvas.ts typed cards (priorities, stage_contacts, meetings, pipelines, opportunities), per-turn KnoxActionRegistry with timestamp-seeded ids.
• New tools: knox_visitor_forms, knox_call_status, knox_agent_k_pipeline, knox_discard_followup; knox_schedule_sms GHL route; 57 tools total.
• Caller authentication tool company_user_authentication (src/elevenlabs/company-auth-tool.*), prompt placement first + last, redact-auth-transcript.ts at the top of postCallWebhook. See Telephony › Caller authentication.
• Custom LLM for ElevenLabs: src/elevenlabs/custom-llm.service.ts + call-models.ts (NATIVE vs CUSTOM ids; Baseten via ElevenLabs workspace secret secretId, never a key in ms-ai; never throws). Env ELEVENLABS_CUSTOM_LLM_PROVIDER / ELEVENLABS_CUSTOM_LLM_MODEL. GET /elevenlabs/agent/:company_id?include_prompt=1.
• Memory gating: use_memory in the agent-config hash; memory-off withholds upsert_memory / delete_memory_slot (memory-tool-gating.spec.ts).
• Voice hardening: assertVoiceCallable() in agent sync, standup allowlist enforced in the tool webhook, "Never executed" post-call marking, tool-status labels on the signed-URL response.
• Call events: knox-signals posts duration_seconds on /call-events/ended and exposes GET /knox-signals/transcripts/durations for the backfill.
• Crawler / scraping: CRAWL_JOB_CONCURRENCY (default 3), scrapping-queue concurrency 10 → 50, sitemap parser + 4-phase native crawler (12b33293e), ScrapingBee tier state (CRAWLER_PROVIDER default native, CRAWL_STEALTH_RENDER_TIMEOUT_MS / SCRAPE_STEALTH_TIMEOUT_MS 240 s, SCRAPE_STEALTH_BUDGET_PER_JOB 5), POST /cancel-scrapping.
• Google Calendar creds: removeIntegration revokes + emits removed; refreshed tokens persisted, re-mirrored, Redis cache invalidated; tokens masked in webhook logs and DLQ (1495b356e).
• Jobber accepted as a CRM + calendar provider (cb9cd8414): CalendarType / pickCalendarType / hosted tools include 'jobber'; book_meeting accepts a Jobber client id.
• Contact extraction: corrected value wins, spelled letters join (54d120273); Simpro create_simpro_job carries call_sid and asks for a comma-separated address (re-push the ElevenLabs tool).
• Build: buildx + ACR :buildcache, prod --push without --load, npm run migration:run:prod.

Data model

Postgres + pgvector. Knox knowledge (knox_knowledge_chunks with dual 1536-dim embeddings + HNSW, knox_knowledge_documents, knox_knowledge_collections), Knox conversations/memory, visitor insight (knox_visitor_profile, knox_session_analysis), action audit, plus Company/Integration/AgentMemory/ PdfContent. Schema is migration-managed (migrations/, npm run migration:run). See Data Stores.

Cross-service

• backend → ms-ai: HTTP with Bearer MS_AI_SECRET_TOKEN for Knox sync, fact extraction, voice handoff.
• ms-sessions → ms-ai: analyzed recordings handed off for enrichment.
• ms-ai → backend: identity-graph lookups (gRPC client) + callbacks.

Config & deployment

• Key env: DATABASE_URL/POSTGRES_*, REDIS_*, ANTHROPIC_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, ELEVENLABS_API_KEY, DEFAULT_CHAT_MODEL, AUTO_TRIGGER_MODEL, BASIC_AUTH_*, OTLP/Pyroscope creds.
• Docker: 3-stage build (deps → build → runtime as non-root, port 3000). entrypoint.sh starts the app directly — migrations run externally (CI), not in the container, to avoid multi-replica races.
• Local: docker-compose.yml brings up Postgres(+pgvector) + Redis.