ms-ai — AI Service

ms-ai is the AI brain of the platform. Unlike the other Node services it is NestJS + TypeScript + Postgres/pgvector, not Nodevel/Mongo.

• Stack: NestJS 11, TypeORM + Postgres 17 + pgvector, Redis + BullMQ, Socket.IO for streaming.
• LLMs: Anthropic, OpenAI (incl. @openai/agents), Mistral, Cohere (rerank), ElevenLabs (voice). Multi-provider routing.
• Observability: OpenTelemetry + Traceloop, Grafana Pyroscope (profiling), Grafana Sigil (LLM tracing).

Responsibilities

• AI chat agent — streaming/non-streaming answers grounded in RAG + identity context; auto-trigger messages; chat summaries; contact extraction.
• AI voice agent (Knox Voice) — ElevenLabs telephony with real-time tool use, post-call webhooks, transcript/summary.
• Nox/Knox admin assistant — the operator-facing assistant (intent → sub-agent → tools). See Nox Assistant.
• Knowledge base / RAG — document ingestion, dual embeddings, pgvector + Cohere rerank. See Knowledge-Base RAG.
• Integrations — Shopify/WooCommerce product sync, Google/Calendly/GHL calendars, fact extraction.
• AI cost tracking — per-provider token usage + pricing.

Module map (src/)

Module	Purpose
agents/ (KnockAgentsModule)	Chat agent: ask-question (v1/v2 streaming), auto-trigger, summary, contact extraction
knox-agent/ (KnoxAgentModule)	The Nox admin assistant — intent classifier → 6 sub-agents → tools
knox-voice/ (KnoxVoiceModule)	Voice agent orchestration, ElevenLabs tool bridge, call context store
elevenlabs/	Telephony: start calls, webhooks, phone numbers, audio/summary
integrations/	Shopify, WooCommerce, Google Calendar, Calendly, GHL calendar, Simpro
queues/ (QueueModule)	BullMQ processors: scraping, embeddings, PDF, product sync, Knox docs/facts, narration
company/	Company context, embeddings, RAG queries (Cohere rerank)
cost-tracking/	Token usage, pricing, daily reconciliation
fact-extraction/	Structured facts from chat/call transcripts
identity-graph/	gRPC client to backend for visitor identity
knox-signals/	Recording-analysis insight signals
auth/	JWT, API-key (secret), webhook HMAC guards; throttler
core/health, core/cache, core/ai-provider	Probes, Redis cache, provider/pricing config

API surface

~45 endpoints (see API_ENDPOINTS.md). Highlights:

Area	Endpoints
Agents	POST /agents/ask-question, POST /agents/v2/ask-question (SSE), /agents/auto-trigger-message, /agents/extract-contact-details
Knowledge	POST /context/.../:company_id, POST /create-embeddings, POST /enqueue-embeddings
Calendar tools	POST /tools/calendar/:id/availability, /tools/calendar/:id/booking (public, rate-limited)
Voice	POST /elevenlabs/start-audio-call, /start-outbound-call, POST /elevenlabs/webhook/post-call
Integrations	POST /integrations/shopify, /integrations/woocommerce, /integrations/actions/google-calendar
Billing	GET /token-usage/:company_id, POST /ai-model-pricing
Admin	GET /docs (Swagger), GET /admin/queues (Bull Board) — basic auth

Auth types: jwt (browser), secret (service-to-service via x-api-key/ Bearer), public (rate-limited), shopify/elevenlabs (HMAC over raw body).

Background processing (BullMQ)

Queue	Purpose
scrappingQueue	fetch & parse URLs
embeddingQueue	generate embeddings
shopiProductsQueue / wooProductsQueue	product catalog sync
pdfExtractionQueue	PDF text extraction
knoxDocumentQueue	KB ingestion → chunk → embed
factExtractionQueue	facts from transcripts
knoxNarrationQueue	narration/audio summaries

Scheduled jobs (@nestjs/schedule): daily cost estimate refresh, 5-min webhook retry, daily ElevenLabs catch-up, weekly cost-table prune. Telephony billing crons (src/elevenlabs/number-rental.scheduler.ts, kill switch NUMBER_RENTAL_CRON_ENABLED=false): number-rental:meter-due (daily 06:00 UTC, monthly number rental, idempotent on (number, period)), twilio-call-cost:meter (10 min, idempotent on CallSid), number-rental:suspension-sweep (15 min, suspend → warn → release after 7 days).

Recent additions (2026-07)

• Cost-tracking: POST /api/cost-tracking/internal/provider-usage — flat-USD provider usage from the backend's AgentKUsageEmitJob, converted to credits at $1 = 100, idempotent on agentk:<co>:<provider>:<day>; priced by src/cost-tracking/agent-k-provider-pricing.ts (AGENT_K_PROVIDER_USD env override; apollo_match $0.05, predictleads $0.02, unipile $0.01, apollo_search free). recordFlatUsdCharge is the generic mechanism — telephony charges use it too, under separate providers. New agent_k usage agent + knox_intent category split; Baseten provider added with GLM-4.7 at 0 credits (platform-absorbed); company_id: 'platform' rows are never emitted. Drift calculation now follows the user-selected window. See cost-tracking.
• Telephony wallet integration: ms-ai gates number purchase/calls on the backend wallet (GET /internal/telephony/wallet/:company_id, marked-up prices) — see Wallet.
• White-label Nox voice: agency nox_voice_id applied as a per-session TTS override on the shared ElevenLabs agent.

Data model

Postgres + pgvector. Knox knowledge (knox_knowledge_chunks with dual 1536-dim embeddings + HNSW, knox_knowledge_documents, knox_knowledge_collections), Knox conversations/memory, visitor insight (knox_visitor_profile, knox_session_analysis), action audit, plus Company/Integration/AgentMemory/ PdfContent. Schema is migration-managed (migrations/, npm run migration:run). See Data Stores.

Cross-service

• backend → ms-ai: HTTP with Bearer MS_AI_SECRET_TOKEN for Knox sync, fact extraction, voice handoff.
• ms-sessions → ms-ai: analyzed recordings handed off for enrichment.
• ms-ai → backend: identity-graph lookups (gRPC client) + callbacks.

Config & deployment

• Key env: DATABASE_URL/POSTGRES_*, REDIS_*, ANTHROPIC_API_KEY, OPENAI_API_KEY, MISTRAL_API_KEY, COHERE_API_KEY, ELEVENLABS_API_KEY, DEFAULT_CHAT_MODEL, AUTO_TRIGGER_MODEL, BASIC_AUTH_*, OTLP/Pyroscope creds.
• Docker: 3-stage build (deps → build → runtime as non-root, port 3000). entrypoint.sh starts the app directly — migrations run externally (CI), not in the container, to avoid multi-replica races.
• Local: docker-compose.yml brings up Postgres(+pgvector) + Redis.