Call Agent
Audience: Sales · Management · QA · Support · Engineering · Where in app: Settings → AI Agents → Call Agent (voice, personality, call behavior, routing); Settings → Calls → Preferences · Plan availability: All plans for web/inbound voice; outbound campaigns are admin-gated (ai_outbound_call_allowed). Usage bounded by AI Credits — see AI Credits
The Call Agent is the AI voice that handles audio calls — when a visitor clicks the audio-call button in the widget, when someone phones a provisioned AI number, or when an outbound campaign places a call. It speaks with a natural text-to-speech voice, listens, answers from your Knowledge Base, and can perform actions mid-call (book a meeting, look up a product, transfer to a human) using server-side tools.
What it does
- Answers inbound phone calls on a provisioned AI number.
- Answers browser/web voice calls started from the widget.
- Places outbound calls (campaigns, proactive callbacks, follow-ups).
- Holds a natural back-and-forth spoken conversation, grounded in the KB via RAG.
- Calls server-side tools/functions mid-conversation: book a meeting, search products/KB, transfer to a human, send a follow-up email, update a CRM contact.
- Verifies caller identity for inbound (caller-ID match) when enabled.
- Captures a transcript, quality score, sentiment, outcome, and termination reason after every call, and updates the identity graph with extracted facts.
How it works
Voice providers. Two stacks, chosen per channel by the model picked in the AI Model picker (ai_config.call_model for the widget agent):
| Stack | Notes |
|---|---|
| ElevenLabs Conversational AI | Default. STT → inner LLM → ElevenLabs TTS; high-quality voices, mature tool-use, native phone numbers. The picker is the inner LLM, from the curated agent_type: 'call' list in ms-ai/src/elevenlabs/call-models.ts (claude-sonnet-4-5 default, GPT-4.1/5-mini/5.1/5.2, Gemini rows); unsupported ids silently fall back to ELEVENLABS_INNER_LLM. Baseten custom LLMs back the agent via ElevenLabs' custom-LLM path — GLM 5.3 Flash (baseten-glm-5.3-flash, added 2026-09-25 when Baseten deprecated GLM 4.7; baseten-glm-4.7 was removed — a stale id falls back to the native default on next sync). Eleven Realtime Max (eleven-realtime-v1-max, 2026-10-23) is ElevenLabs' own speech-to-speech on the same agent: conversation_config.realtime_model = eleven_realtime_v1_max, ElevenLabs voices/tools/speech settings unchanged, explicit END CALL rules appended to the prompt; billed through ElevenLabs post-call credits ($0.08/min + tokens). Not an "(S2S)" row in the UI. The -mini variant was removed 2026-10-26 (announced tool actions without calling them). |
| Native speech-to-speech (S2S) | Picker rows suffixed "(S2S)": GPT Live 1 (S2S, full-duplex) (gpt-live-1), Gemini 3.8 Live (S2S) (gemini-3.8-live) and Gemini 3.8 Live Extended Thinking (S2S, slower replies) (gemini-3.8-live-extended-thinking, back 2026-11-02) — open to every company — plus Grok Voice (S2S) (grok-voice, 2026-11-06), opt-in per company via the super-admin Grok Voice switch (voice_engine_providers.xai). The GPT Realtime models and older Gemini ids are retired — refused at save (403 VOICE_ENGINE_MODEL_RETIRED). One realtime model hears and speaks; no ElevenLabs in the path. PROD serves all four direct on web, inbound and outbound (Grok phone over SIP). Full routing, modes, GPT-Live delegation, recording and billing: Voice Engine. |
Legacy: a widget company with ai_config.call_model_provider: 'fixie-ai/ultravox' still takes the Ultravox web path (ms-comm → ms-ai /ultravox/start-audio-call, per-call agent); it is not offered in the picker. The old direct "OpenAI Realtime" provider row is superseded by the S2S path.
Tool / function calling. During a call the agent can invoke server-side functions exposed by ms-ai's knox-voice/ module (knox-voice-tools.service.ts). Typical tools: book a meeting (Google/Calendly/GHL), look up a product (RAG over the product catalog), search the company knowledge base, transfer/take over to a human, send a follow-up email, update a CRM contact. Tool schemas are converted to the provider's format (zod-to-elevenlabs-schema.ts) and the calls flow provider → WebSocket → ms-ai → ms-communication. The public calendar tool endpoints are POST /tools/calendar/.../availability and .../booking (rate-limited).
RAG grounding. Same retrieval path as the Chat Agent: the question is embedded, top candidates are pulled from the company embeddings pgvector table and Cohere-reranked, and injected as context. Product lookups run over the product catalog (rag_products). See Knowledge Base.
Multilingual. ElevenLabs language_detection lets the agent auto-switch mid-call into any of ~70 preset languages (MULTILINGUAL_CODES in ms-ai/src/elevenlabs/elevenlabs.service.ts, built into languagePresets on every agent sync). The configured agent_language still governs the greeting; blank (the default since 2026-10-27, UI "Same as organization") inherits company.language, then en — see Voice Engine › Instructions.
Configuration & options
Voice selection & persona
Choose the voice (gender, tone, accent/style) and preview it before going live. The voice is stored as agent_voice_id (ElevenLabs voice ID, or a provider-native name such as marin for an S2S model); language as agent_language (BCP-47 code; '' = same as organization — the shared AgentLanguagePicker.vue; the widget agent's is ai_config.call_agent_language). The voice list follows the model (useProviderAwareVoices.js): an S2S model shows its vendor's small voice set with gender + description instead of the ElevenLabs gender/accent/language filters, and changing the model clears a voice the new model can't play.
Behavior tab
Agent Name, Greeting Message (moved here from its own tab), and the instructions. The old Quick mode (tone + response size) is gone — everything saves setting: 'advanced'. For an S2S model the tab shows the S2S copy (agent_instructions_s2s, "Use default S2S prompt") and the S2S greeting (ai_calling.greeting_message_s2s, "Greeting Message for speech-to-speech (S2S) models") instead of the ElevenLabs fields; on gpt-live-1 the instructions box becomes two — "Speech model instructions (GPT-Live 1)" (agent_instructions_gpt_live_speech) and "Reasoning model instructions (model label)" (agent_instructions_gpt_live_reasoning), each with a "Use default …" button and required when GPT-Live is selected; the AI Engine tab adds Reasoning Model (GPT-Live) (ai_config.call_gpt_live_reasoning_model). See Voice Engine › Instructions and GPT-Live. Agency Controls can hide the Behavior tab from the client.
Speech settings & background sound (widget agent)
ai_config.call_speech_settings— responsiveness (mapped to ElevenLabs turn-eagerness buckets),interruption_sensitivity(default 0.5), backchannel, audio normalization.ai_config.call_background_sound— ElevenLabs-native ambience: enable, preset track, volume (stored 0–100, sent/100), crossfade loop.- The per-telephony-agent equivalents are Mixed
speech_settings/background_soundblocks on the inbound/outbound agents — see Telephony › Speech settings.
Call-screen design (ai_calling.call_interface)
Widget voice-call screen theming (Settings → Call Agent → Call Interface): design (classic dark | modern light redesign), background_color, controller_bar_color, icon_color, accent gradient pair accent_color / accent_color_2 (mic pill, Schedule CTA, voice line, avatar mark), booking_bg_color / booking_slot_color (modern in-call booking calendar), hide_connecting_steps, hide_latest_message, avatar_image (file id), agent_name, and connect_messages (rotating connect-screen headlines, {name} substituted). Empty values fall back to widget defaults.
Outbound calling — companies.ai_calling
| Field | What |
|---|---|
enable | Master switch. |
setting | quick or advanced. |
agent_instructions | System prompt (advanced). |
agent_name | Voice persona name. |
tone | Persona tone. |
response_size | concise / detailed. |
greeting_message | Opening line. |
Inbound calling — companies.ai_inbound_call
| Field | What |
|---|---|
enable | Master switch — provisions an inbound number routed to the AI. |
model / model_provider | LLM + provider. |
agent_voice_id | ElevenLabs voice ID. |
agent_language | BCP-47 language code. Blank = same as organization: resolves own → company.language → en (outbound dial, inbound sync, Platform API); changing company.language redeploys inheriting inbound agents. New agents start blank (de172c75a). One-shot migration POST /admin/maintenance/inherit-agent-languages (also /migrations/inherit-agent-languages, background job) blanks stored codes that repeat what the call already speaks. |
Caller-ID may verify the visitor's identity if identity_graph_config.accept_inbound_call_verification is on.
Outbound campaigns (multi-agent)
companies.ai_outbound_call_enabled: true // requires admin gate ai_outbound_call_allowed
companies.ai_outbound_call_agents: [
{ name, prompt, voice_id, model_provider, model, webhook_secret, ... },
...
]
Each entry is a distinct persona/script. Launched from the dashboard's outbound campaign UI, or programmatically by Nox via knox-place-call (when enabled).
Call behavior & routing
- Call routing —
companies.ai_config.call_routing:ai(AI handles the call) /ai_to_human(AI as the first leg, then handoff) /human. - Duration limits / silence handling / hold behavior — cap call length to manage credits; configure re-prompts on silence ("Are you still there?").
- Provider data models (ms-ai):
elabs_agents(ElevenLabs agent config),elabs_phone_numbers(mapped inbound numbers).
Behaviors & edge cases
- Transport. Widget AI calls now run over WebRTC (LiveKit) — ms-communication hardcodes
calling_type: 'webrtc', ms-ai returns a LiveKit conversation token, and the widget uses the ElevenLabs SDK callbacks for transcripts/messages/tools (the websocket signed-URL path remains as fallback). PSTN calls run through the carrier bridges (Twilio + Telnyx) — see Telephony. - Connect latency — agent-sync skip cache. Web
start-audio-callno longer re-syncs the ElevenLabs agent when nothing changed (~6.2s → fast connect): per-replica in-memory cache keyed on a sha256 of the agent config, TTLAGENT_SYNC_CACHE_TTL_MS(5 min default), invalidated byinvalidateAgentSyncCache()on mutating paths. Telephony agents are not cached (they preheat). - Transcript translation. The Calls UI translates PSTN transcripts on demand via ms-ai
POST /translate-transcript— lines chunked 40 at a time (cap 400 lines, 2000 chars/line), index-aligned response, billed underKnownAgent.TRANSCRIPT_TRANSLATE. - S2S calls run outside ElevenLabs: no background sound/speech settings, recordings stored in our S3 keyed by a UUID session id (Grok SIP calls: Twilio recording, proxied). Live listening and takeover do work (phone: every vendor on the media bridge — not Grok over SIP; web: OpenAI, Gemini and xAI — Vapi refused). Gemini sessions use our own turn-taking (Soniox endpoints, near-mic gate, echo-aware barge-in) — see Voice Engine › Gemini turn-taking. Widget S2S calls get a summary + outcome from ms-ai
/voice-direct/web-analysisat call end (fills empty service-room fields; feeds the transcript email). See Voice Engine. - Takeover. A human agent joins and the AI hands off; audio routes through Twilio WebRTC. Visibility controlled by
companies.communication_options.take_over_permissions. - Monitor. A human can listen silently —
monitor_callsocket event, audio streamed viaaudio_chunk(on WebRTC calls the widget taps the agent's LiveKit audio track to feed the stream). - Recording. ElevenLabs/Ultravox transcripts are always captured; audio recording is optional per provider settings. Widget call recordings (Twilio compositions, Ultravox) are no longer public on ms-comm — the dashboard plays them through the backend's authed proxy (see Calls › Recording playback).
companies.always_twilio_calls = trueforces the Twilio path even for AI calls (useful for compliance environments that require Twilio's recording). - Post-call enrichment. After the call: transcript saved to
knox_call_transcripts; heuristic quality score computed; high-stakes calls re-scored by an LLM-judge (quality_method: llm_judge); sentiment, outcome, topics, and termination reason populated; booking confirmations sent; identity graph updated with extracted facts. PSTN calls additionally getcall_sentiment+call_outcomeclassification, voicemail detection, and a signedrecording_urlin the post-call webhook — see Telephony › Post-call pipeline. - Credit exhaustion. Voice consumes credits per minute (faster than chat). Inbound/web falls back to a human if available, otherwise a missed-call. Outbound campaigns pause until refilled. See AI Credits.
Quality scoring
knox_call_transcripts.quality_score (0..1) with quality_method:
| Method | When |
|---|---|
heuristic | Fast — based on duration, turn count, agent-failed-to-answer signals. |
llm_judge | Slower — LLM scores the transcript across multiple dimensions (high-stakes calls). |
skipped | Too short to score meaningfully. |
quality_breakdown is available via knox-aggregate. Nox can report: which provider is configured, the agent's prompt, call/booking counts, quality breakdown, top termination reasons, and whether outbound is enabled. Tools: knox-call-transcripts, knox-call-transcript-full, knox-recent-bookings, knox-aggregate(metric=quality_breakdown).
Plan & limits
- Web and inbound voice are on all plans; outbound campaigns require the
ai_outbound_call_allowedadmin gate. - Voice usage is bounded by AI Credits. ElevenLabs calls bill per minute (provider voice cost) plus the inner LLM; S2S calls bill the vendor's own usage — Gemini (both models) per token (+ the external caller transcriber,
gpt-transcribeper minute), GPT-Live $0.05/min per second plus the reasoning model's tokens, Grok Voice $0.08/min (its SIP leg + Twilio recording go to the telephony wallet) — see Voice Engine › Billing. ElevenLabs voice is priced as a credit-based plan rate; the bundled inner LLM is aLLM_PASSTHROUGHcharge on top. See AI Credits. - Voice usage is tracked by location (
voice_inbound/voice_outbound/voice_web) so management can see, e.g., whether browser voice is costing more than phone calls. - Verify per-tenant phone-number provisioning and any region/compliance constraints.
Technical implementation
- Service: ms-ai, modules
knox-voice/(KnoxVoiceModule— orchestration, tool bridge, context/registry stores, post-call service) andelevenlabs/(telephony: start calls, webhooks, phone numbers). - Endpoints:
POST /elevenlabs/start-audio-call(returns a LiveKittokenforcalling_type: 'webrtc',signedUrlfor websocket),GET /elevenlabs/voices,POST /elevenlabs/create-tools,POST /elevenlabs/webhook/post-call(HMAC-verifiedelevenlabsauth), number endpoints (import-phone-number/{twilio,telnyx},buy-phone-number/twilio/*,GET /elevenlabs/phone-numbers/:company_id— see Telephony), and calendar toolsPOST /tools/calendar/.../availability·.../booking. Seems-ai/API_ENDPOINTS.md. - Tool schema bridge:
knox-voice/zod-to-elevenlabs-schema.ts; tool impls inknox-voice/knox-voice-tools.service.ts. - Prompts:
knox-voice/knox-voice.prompt.ts,knox-voice-brief.prompt.ts. - Post-call:
knox-voice/knox-voice-post-call.service.ts; transcripts inknox_call_transcripts. - Inner LLM:
ELEVENLABS_INNER_LLMenv (defaultclaude-sonnet-4-5) inelevenlabs/elevenlabs.service.ts. - Widget: the browser audio-call flow lives in the widget (
aiCall.ts/ ChatWidget) → ms-communication → ms-aistart-audio-call; LiveKit transport with a 30s token wait and a ringback loop (on-call-ringing.mp3) during connect. - Cost:
cost-tracking/books per-minuteusage_charge(provider) +LLM_PASSTHROUGHfor the inner LLM. See AI Credits.
Related
- Chat Agent — the text equivalent (shares RAG + identity context).
- Knowledge Base — what the agent reads, incl. product catalog.
- AI Credits — per-minute billing and exhaustion behavior.
- Voice Engine — speech-to-speech models, modes, billing.
- Telephony subsystem — carriers, number lifecycle, wallet, transports, escalation.
- ms-ai service · Knowledge-Base RAG · Identity Graph · Nox Assistant