Skip to main content

Call Agent

Audience: Sales · Management · QA · Support · Engineering · Where in app: Settings → AI Agents → Call Agent (voice, personality, call behavior, routing); Settings → Calls → Preferences · Plan availability: All plans for web/inbound voice; outbound campaigns are admin-gated (ai_outbound_call_allowed). Usage bounded by AI Credits — see AI Credits

The Call Agent is the AI voice that handles audio calls — when a visitor clicks the audio-call button in the widget, when someone phones a provisioned AI number, or when an outbound campaign places a call. It speaks with a natural text-to-speech voice, listens, answers from your Knowledge Base, and can perform actions mid-call (book a meeting, look up a product, transfer to a human) using server-side tools.

What it does​

  • Answers inbound phone calls on a provisioned AI number.
  • Answers browser/web voice calls started from the widget.
  • Places outbound calls (campaigns, proactive callbacks, follow-ups).
  • Holds a natural back-and-forth spoken conversation, grounded in the KB via RAG.
  • Calls server-side tools/functions mid-conversation: book a meeting, search products/KB, transfer to a human, send a follow-up email, update a CRM contact.
  • Verifies caller identity for inbound (caller-ID match) when enabled.
  • Captures a transcript, quality score, sentiment, outcome, and termination reason after every call, and updates the identity graph with extracted facts.

How it works​

Voice providers. Two stacks, chosen per channel by the model picked in the AI Model picker (ai_config.call_model for the widget agent):

StackNotes
ElevenLabs Conversational AIDefault. STT → inner LLM → ElevenLabs TTS; high-quality voices, mature tool-use, native phone numbers. The picker is the inner LLM, from the curated agent_type: 'call' list in ms-ai/src/elevenlabs/call-models.ts (claude-sonnet-4-5 default, GPT-4.1/5-mini/5.1/5.2, Gemini rows); unsupported ids silently fall back to ELEVENLABS_INNER_LLM. Baseten custom LLMs back the agent via ElevenLabs' custom-LLM path — GLM 5.3 Flash (baseten-glm-5.3-flash, added 2026-09-25 when Baseten deprecated GLM 4.7; baseten-glm-4.7 was removed — a stale id falls back to the native default on next sync). Eleven Realtime Max (eleven-realtime-v1-max, 2026-10-23) is ElevenLabs' own speech-to-speech on the same agent: conversation_config.realtime_model = eleven_realtime_v1_max, ElevenLabs voices/tools/speech settings unchanged, explicit END CALL rules appended to the prompt; billed through ElevenLabs post-call credits ($0.08/min + tokens). Not an "(S2S)" row in the UI. The -mini variant was removed 2026-10-26 (announced tool actions without calling them).
Native speech-to-speech (S2S)Picker rows suffixed "(S2S)": GPT Live 1 (S2S, full-duplex) (gpt-live-1), Gemini 3.8 Live (S2S) (gemini-3.8-live) and Gemini 3.8 Live Extended Thinking (S2S, slower replies) (gemini-3.8-live-extended-thinking, back 2026-11-02) — open to every company — plus Grok Voice (S2S) (grok-voice, 2026-11-06), opt-in per company via the super-admin Grok Voice switch (voice_engine_providers.xai). The GPT Realtime models and older Gemini ids are retired — refused at save (403 VOICE_ENGINE_MODEL_RETIRED). One realtime model hears and speaks; no ElevenLabs in the path. PROD serves all four direct on web, inbound and outbound (Grok phone over SIP). Full routing, modes, GPT-Live delegation, recording and billing: Voice Engine.

Legacy: a widget company with ai_config.call_model_provider: 'fixie-ai/ultravox' still takes the Ultravox web path (ms-comm → ms-ai /ultravox/start-audio-call, per-call agent); it is not offered in the picker. The old direct "OpenAI Realtime" provider row is superseded by the S2S path.

Tool / function calling. During a call the agent can invoke server-side functions exposed by ms-ai's knox-voice/ module (knox-voice-tools.service.ts). Typical tools: book a meeting (Google/Calendly/GHL), look up a product (RAG over the product catalog), search the company knowledge base, transfer/take over to a human, send a follow-up email, update a CRM contact. Tool schemas are converted to the provider's format (zod-to-elevenlabs-schema.ts) and the calls flow provider → WebSocket → ms-ai → ms-communication. The public calendar tool endpoints are POST /tools/calendar/.../availability and .../booking (rate-limited).

RAG grounding. Same retrieval path as the Chat Agent: the question is embedded, top candidates are pulled from the company embeddings pgvector table and Cohere-reranked, and injected as context. Product lookups run over the product catalog (rag_products). See Knowledge Base.

Multilingual. ElevenLabs language_detection lets the agent auto-switch mid-call into any of ~70 preset languages (MULTILINGUAL_CODES in ms-ai/src/elevenlabs/elevenlabs.service.ts, built into languagePresets on every agent sync). The configured agent_language still governs the greeting; blank (the default since 2026-10-27, UI "Same as organization") inherits company.language, then en — see Voice Engine › Instructions.

Configuration & options​

Voice selection & persona​

Choose the voice (gender, tone, accent/style) and preview it before going live. The voice is stored as agent_voice_id (ElevenLabs voice ID, or a provider-native name such as marin for an S2S model); language as agent_language (BCP-47 code; '' = same as organization — the shared AgentLanguagePicker.vue; the widget agent's is ai_config.call_agent_language). The voice list follows the model (useProviderAwareVoices.js): an S2S model shows its vendor's small voice set with gender + description instead of the ElevenLabs gender/accent/language filters, and changing the model clears a voice the new model can't play.

Behavior tab​

Agent Name, Greeting Message (moved here from its own tab), and the instructions. The old Quick mode (tone + response size) is gone — everything saves setting: 'advanced'. For an S2S model the tab shows the S2S copy (agent_instructions_s2s, "Use default S2S prompt") and the S2S greeting (ai_calling.greeting_message_s2s, "Greeting Message for speech-to-speech (S2S) models") instead of the ElevenLabs fields; on gpt-live-1 the instructions box becomes two — "Speech model instructions (GPT-Live 1)" (agent_instructions_gpt_live_speech) and "Reasoning model instructions (model label)" (agent_instructions_gpt_live_reasoning), each with a "Use default …" button and required when GPT-Live is selected; the AI Engine tab adds Reasoning Model (GPT-Live) (ai_config.call_gpt_live_reasoning_model). See Voice Engine › Instructions and GPT-Live. Agency Controls can hide the Behavior tab from the client.

Speech settings & background sound (widget agent)​

  • ai_config.call_speech_settings — responsiveness (mapped to ElevenLabs turn-eagerness buckets), interruption_sensitivity (default 0.5), backchannel, audio normalization.
  • ai_config.call_background_sound — ElevenLabs-native ambience: enable, preset track, volume (stored 0–100, sent /100), crossfade loop.
  • The per-telephony-agent equivalents are Mixed speech_settings / background_sound blocks on the inbound/outbound agents — see Telephony › Speech settings.

Call-screen design (ai_calling.call_interface)​

Widget voice-call screen theming (Settings → Call Agent → Call Interface): design (classic dark | modern light redesign), background_color, controller_bar_color, icon_color, accent gradient pair accent_color / accent_color_2 (mic pill, Schedule CTA, voice line, avatar mark), booking_bg_color / booking_slot_color (modern in-call booking calendar), hide_connecting_steps, hide_latest_message, avatar_image (file id), agent_name, and connect_messages (rotating connect-screen headlines, {name} substituted). Empty values fall back to widget defaults.

Outbound calling — companies.ai_calling​

FieldWhat
enableMaster switch.
settingquick or advanced.
agent_instructionsSystem prompt (advanced).
agent_nameVoice persona name.
tonePersona tone.
response_sizeconcise / detailed.
greeting_messageOpening line.

Inbound calling — companies.ai_inbound_call​

FieldWhat
enableMaster switch — provisions an inbound number routed to the AI.
model / model_providerLLM + provider.
agent_voice_idElevenLabs voice ID.
agent_languageBCP-47 language code. Blank = same as organization: resolves own → company.language → en (outbound dial, inbound sync, Platform API); changing company.language redeploys inheriting inbound agents. New agents start blank (de172c75a). One-shot migration POST /admin/maintenance/inherit-agent-languages (also /migrations/inherit-agent-languages, background job) blanks stored codes that repeat what the call already speaks.

Caller-ID may verify the visitor's identity if identity_graph_config.accept_inbound_call_verification is on.

Outbound campaigns (multi-agent)​

companies.ai_outbound_call_enabled: true // requires admin gate ai_outbound_call_allowed
companies.ai_outbound_call_agents: [
{ name, prompt, voice_id, model_provider, model, webhook_secret, ... },
...
]

Each entry is a distinct persona/script. Launched from the dashboard's outbound campaign UI, or programmatically by Nox via knox-place-call (when enabled).

Call behavior & routing​

  • Call routing — companies.ai_config.call_routing: ai (AI handles the call) / ai_to_human (AI as the first leg, then handoff) / human.
  • Duration limits / silence handling / hold behavior — cap call length to manage credits; configure re-prompts on silence ("Are you still there?").
  • Provider data models (ms-ai): elabs_agents (ElevenLabs agent config), elabs_phone_numbers (mapped inbound numbers).

Behaviors & edge cases​

  • Transport. Widget AI calls now run over WebRTC (LiveKit) — ms-communication hardcodes calling_type: 'webrtc', ms-ai returns a LiveKit conversation token, and the widget uses the ElevenLabs SDK callbacks for transcripts/messages/tools (the websocket signed-URL path remains as fallback). PSTN calls run through the carrier bridges (Twilio + Telnyx) — see Telephony.
  • Connect latency — agent-sync skip cache. Web start-audio-call no longer re-syncs the ElevenLabs agent when nothing changed (~6.2s → fast connect): per-replica in-memory cache keyed on a sha256 of the agent config, TTL AGENT_SYNC_CACHE_TTL_MS (5 min default), invalidated by invalidateAgentSyncCache() on mutating paths. Telephony agents are not cached (they preheat).
  • Transcript translation. The Calls UI translates PSTN transcripts on demand via ms-ai POST /translate-transcript — lines chunked 40 at a time (cap 400 lines, 2000 chars/line), index-aligned response, billed under KnownAgent.TRANSCRIPT_TRANSLATE.
  • S2S calls run outside ElevenLabs: no background sound/speech settings, recordings stored in our S3 keyed by a UUID session id (Grok SIP calls: Twilio recording, proxied). Live listening and takeover do work (phone: every vendor on the media bridge — not Grok over SIP; web: OpenAI, Gemini and xAI — Vapi refused). Gemini sessions use our own turn-taking (Soniox endpoints, near-mic gate, echo-aware barge-in) — see Voice Engine › Gemini turn-taking. Widget S2S calls get a summary + outcome from ms-ai /voice-direct/web-analysis at call end (fills empty service-room fields; feeds the transcript email). See Voice Engine.
  • Takeover. A human agent joins and the AI hands off; audio routes through Twilio WebRTC. Visibility controlled by companies.communication_options.take_over_permissions.
  • Monitor. A human can listen silently — monitor_call socket event, audio streamed via audio_chunk (on WebRTC calls the widget taps the agent's LiveKit audio track to feed the stream).
  • Recording. ElevenLabs/Ultravox transcripts are always captured; audio recording is optional per provider settings. Widget call recordings (Twilio compositions, Ultravox) are no longer public on ms-comm — the dashboard plays them through the backend's authed proxy (see Calls › Recording playback). companies.always_twilio_calls = true forces the Twilio path even for AI calls (useful for compliance environments that require Twilio's recording).
  • Post-call enrichment. After the call: transcript saved to knox_call_transcripts; heuristic quality score computed; high-stakes calls re-scored by an LLM-judge (quality_method: llm_judge); sentiment, outcome, topics, and termination reason populated; booking confirmations sent; identity graph updated with extracted facts. PSTN calls additionally get call_sentiment + call_outcome classification, voicemail detection, and a signed recording_url in the post-call webhook — see Telephony › Post-call pipeline.
  • Credit exhaustion. Voice consumes credits per minute (faster than chat). Inbound/web falls back to a human if available, otherwise a missed-call. Outbound campaigns pause until refilled. See AI Credits.

Quality scoring​

knox_call_transcripts.quality_score (0..1) with quality_method:

MethodWhen
heuristicFast — based on duration, turn count, agent-failed-to-answer signals.
llm_judgeSlower — LLM scores the transcript across multiple dimensions (high-stakes calls).
skippedToo short to score meaningfully.

quality_breakdown is available via knox-aggregate. Nox can report: which provider is configured, the agent's prompt, call/booking counts, quality breakdown, top termination reasons, and whether outbound is enabled. Tools: knox-call-transcripts, knox-call-transcript-full, knox-recent-bookings, knox-aggregate(metric=quality_breakdown).

Plan & limits​

  • Web and inbound voice are on all plans; outbound campaigns require the ai_outbound_call_allowed admin gate.
  • Voice usage is bounded by AI Credits. ElevenLabs calls bill per minute (provider voice cost) plus the inner LLM; S2S calls bill the vendor's own usage — Gemini (both models) per token (+ the external caller transcriber, gpt-transcribe per minute), GPT-Live $0.05/min per second plus the reasoning model's tokens, Grok Voice $0.08/min (its SIP leg + Twilio recording go to the telephony wallet) — see Voice Engine › Billing. ElevenLabs voice is priced as a credit-based plan rate; the bundled inner LLM is a LLM_PASSTHROUGH charge on top. See AI Credits.
  • Voice usage is tracked by location (voice_inbound / voice_outbound / voice_web) so management can see, e.g., whether browser voice is costing more than phone calls.
  • Verify per-tenant phone-number provisioning and any region/compliance constraints.

Technical implementation​

  • Service: ms-ai, modules knox-voice/ (KnoxVoiceModule — orchestration, tool bridge, context/registry stores, post-call service) and elevenlabs/ (telephony: start calls, webhooks, phone numbers).
  • Endpoints: POST /elevenlabs/start-audio-call (returns a LiveKit token for calling_type: 'webrtc', signedUrl for websocket), GET /elevenlabs/voices, POST /elevenlabs/create-tools, POST /elevenlabs/webhook/post-call (HMAC-verified elevenlabs auth), number endpoints (import-phone-number/{twilio,telnyx}, buy-phone-number/twilio/*, GET /elevenlabs/phone-numbers/:company_id — see Telephony), and calendar tools POST /tools/calendar/.../availability · .../booking. See ms-ai/API_ENDPOINTS.md.
  • Tool schema bridge: knox-voice/zod-to-elevenlabs-schema.ts; tool impls in knox-voice/knox-voice-tools.service.ts.
  • Prompts: knox-voice/knox-voice.prompt.ts, knox-voice-brief.prompt.ts.
  • Post-call: knox-voice/knox-voice-post-call.service.ts; transcripts in knox_call_transcripts.
  • Inner LLM: ELEVENLABS_INNER_LLM env (default claude-sonnet-4-5) in elevenlabs/elevenlabs.service.ts.
  • Widget: the browser audio-call flow lives in the widget (aiCall.ts / ChatWidget) → ms-communication → ms-ai start-audio-call; LiveKit transport with a 30s token wait and a ringback loop (on-call-ringing.mp3) during connect.
  • Cost: cost-tracking/ books per-minute usage_charge (provider) + LLM_PASSTHROUGH for the inner LLM. See AI Credits.