Skip to main content

Voice Engine (Speech-to-Speech)

Every AI voice call (widget, inbound phone, outbound phone) runs on one of two stacks:

  • ElevenLabs Convai — the default. ElevenLabs does STT → inner LLM → TTS; the "AI Model" pick is the inner LLM. See Call Agent and Telephony.
  • Native speech-to-speech (S2S) — one realtime model hears the caller and speaks back (offered today: OpenAI GPT Live 1, Google Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, open to every company; xAI Grok Voice, opt-in per company). Chosen by picking an S2S model in the same AI Model picker; everything below is about this path.

Eleven Realtime (eleven-realtime-v1-max, picker label "Eleven Realtime Max", ms-ai abf56f790, 2026-10-23) is not this path: it stays an ordinary ElevenLabs agent with conversation_config.realtime_model set (ElevenlabsService.applyRealtimeModel, cleared on the next sync when the model changes), keeps ElevenLabs voices/tools/speech settings and bills through the ElevenLabs post-call credits. ms-ai appends explicit END CALL rules to its prompt (elevenRealtimePrompt). eleven-realtime-v1-mini was removed 2026-10-26 (62d88daa2) — on real calls it announced tool actions without calling them; a stale id falls back to the native default LLM.

The routing decision, the model catalog, and billing all live in ms-ai (src/voice-engine/). The audio path depends on VOICE_S2S_MODE.

Model catalog​

Source of truth: ms-ai/src/voice-engine/voice-engine.models.ts (VOICE_ENGINE_MODELS, OPEN_TO_ALL_COMPANIES_MODELS, RETIRED_VOICE_MODELS). Mirrored — add ids in the same change — in backend app/lib/voiceEngineModels.js, ms-comm app/helpers/VoiceProviderHelper.js, frontend src/utils/voiceEngineModels.js (an ms-ai drift spec parses the frontend file), and the picker catalog ms-ai/src/elevenlabs/call-models.ts.

Offered (since 2026-10-13, 0da8c0b6f / backend e0c575e86 / frontend 105bfaac; Extended Thinking back 2026-11-02, Grok 2026-11-06):

Selector id (model_name)Permission keyVendor model / APIPicker labelVoices
gpt-live-1openai_livegpt-live-1 on OpenAI's Live API (api: 'live', /v1/live/sessions), full duplexGPT Live 1 (S2S, full-duplex)17: marin (default), cedar, 12 locale voices (quartz, ripple, vesper, willow, stone, gleam, meridian, delta, cinder, beacon, bossa, tempo), alloy, echo, shimmer
gemini-3.8-livegooglemodels/gemini-3.8-live (Live API, v1beta)Gemini 3.8 Live (S2S)12 Gemini Live voices: Puck (default), Charon, Kore, Fenrir, Aoede, Zephyr, Erinome, Achernar, Laomedeia, Rasalgethi, Alnilam, Schedar
gemini-3.8-live-extended-thinkinggooglemodels/gemini-3.8-live-extended-thinkingGemini 3.8 Live Extended Thinking (S2S, slower replies)same 12 Gemini voices
grok-voicexaigrok-voice-3.0 (xAI realtime, OpenAI-Realtime wire format)Grok Voice (S2S)4: Eve (default), Ara, Rex, Leo

The three OpenAI/Gemini rows are in OPEN_TO_ALL_COMPANIES_MODELS: every company may use them — no voice_engine_enabled, no provider key (routing, save check, S2S prompt and greeting selection all short-circuit on it). Grok stays on the per-company permission (opt-in xai, below).

  • Gemini 3.8 Live now refuses any thinking setting (Google 1007 "Thinking budget is not supported", since 2026-11-04) — the minter sends none (d6b2d11d2, 24408bebe).
  • Extended Thinking (isGeminiThinkingModel) rejects a setup without a thinking level, so it is sent thinkingLevel: LOW (accepts LOW/MEDIUM/HIGH, not MINIMAL; 6ed636544). Google runs it differently: only NON_BLOCKING tool declarations, it speaks its own fillers while reasoning, and turnComplete no longer means done (interactionStatus does). Tool results go back with WHEN_IDLE scheduling (ms-comm VOICE_DIRECT_GEMINI_TOOL_SCHEDULING=off rolls back). On end_call it is allowed one goodbye before the hang-up, and lookup results still in flight are dropped (975d963, c87f959). Same per-token prices as 3.8 Live (Google's output price includes thinking tokens).

Retired (RETIRED_VOICE_MODELS): gpt-realtime, gpt-realtime-2, gpt-realtime-2.1, gpt-realtime-2.1-mini, gemini-2.5-flash-native-audio, gemini-3.1-flash-live. Still catalogued so an in-flight or historical session resolves for billing/recordings, and still recognised as S2S (an old saved id keeps its S2S instructions), but removed from the picker catalog and refused at save for every company — backend 403 VOICE_ENGINE_MODEL_RETIRED ("This speech-to-speech model is no longer available. Choose GPT Live 1 or Gemini 3.8 Live."). Re-sending an unchanged stored value is still allowed so the rest of the form saves; the picker shows a saved retired id as "(Not available — runs on ElevenLabs)". (The backend message still names only "GPT Live 1 or Gemini 3.8 Live".) Note ms-ai routing does not check RETIRED_VOICE_MODELS — a company still holding a retired id with voice_engine_enabled on would keep routing S2S.

Voice ids are provider-native and never shared across providers — a voice from one vendor sent to another is refused at the handshake (a dead call); GPT-Live voices must never be defaulted onto a Realtime model or vice versa. Changing the model in the UI clears a voice the new model can't play. Gemini 3.8 tool calling is not in Google's Live tool matrix — it works on the direct path but smoke-test new tools on stage.

Which models a tenant can actually pick is the intersection of:

  1. the frontend allowlist VAPI_SERVED_MODELS (gpt-live-1, gemini-3.8-live, gemini-3.8-live-extended-thinking, grok-voice);
  2. ms-ai GET /ai-model-pricing/estimates/:company_id, which drops any S2S model the current mode can't serve (isS2SModelServed: in direct mode, only vendors whose API key is present in ms-ai);
  3. for anything not open-to-all, the company permission (below).

Production (2026-11-06): ms-ai PROD has the OpenAI key, VOICE_DIRECT_GEMINI_API_KEY (flux acd25916) and XAI_API_KEY, with VOICE_DIRECT_WEB_VENDORS: 'openai,gemini,xai' (flux d88deed1); ms-comm PROD has XAI_API_KEY + XAI_SIP_ENABLED: "true" (flux 43e353a7). So the three open models are offered to every tenant on web, inbound and outbound, and Grok to companies with xai on.

Picker behaviour (buildCallModelOptions): in direct mode S2S rows carry no credits/minute price (only vapi mode shows a trailing-30-day estimate). The "No live listening or takeover on web calls" hint is shown for providers outside the picker's MONITORABLE_PROVIDERS (openai, openai_live, google) — i.e. on the Grok row, although xAI web calls are in fact monitorable now (MONITORABLE_WEB_VENDORS includes xai); see Operational limits. A saved S2S model the company can't use stays visible, suffixed "(Requires speech-to-speech access)", "(Provider not enabled)" or "(Not available — runs on ElevenLabs)". For an S2S model the AI Engine tab hides background sound and speech settings (ElevenLabs-only; values kept). On gpt-live-1 the AI Engine tab adds a Reasoning Model (GPT-Live) select (see GPT-Live). Voice previews (since 2026-11-06): pickers play the clip saved in the super-admin Voice Previews page first (frontend views/admin/voice-previews/List.vue, 01cba755), then a static file. Backend AdminVoicePreviewController (GET /voice-previews for pickers; GET /admin/voice-previews, POST /admin/voice-previews/:provider/:voice_id/{generate,upload}, POST /admin/voice-previews/generate-missing → background job polled at GET /admin/voice-previews/jobs/:job_id, one retry per failure for Gemini's rate limit) stores clips as files rows on S3 linked per voice in voice_previews (provider, voice_id, file, source generated|uploaded, unique on provider+voice). Generation goes through ms-ai GET /voice-engine/previews/catalog + POST /voice-engine/previews/generate (a4f24bef8) with the provider keys ms-ai holds. GPT-Live voices have no TTS twin, so they are not generatable — only their clips stay in the repo (frontend/public/voice-previews/openai/); all other static clips were removed.

Company permission​

The general provider panel is gone (frontend 105bfaac deleted VoiceEngineProviders.vue and useVoiceEngineGate.js). Grok is the one offered model still gated, so the super-admin company page (admin/companies/Detail.vue) has a single Grok Voice switch (GrokVoiceToggle.vue, 34997d6c, "Web, inbound and outbound calls"): on sends PUT /company/:_id/voice-engine { enabled: true, providers: { xai: true } } (sets the main flag too), off sends { providers: { xai: false } }. If the off-save fails with a 5xx, the toggle offers Retry revoke — stored phone agents may still reach Grok until the revoke (below) succeeds.

  • company.voice_engine_enabled — main flag (default true for companies created since 2026-09-30); writable only through toggleVoiceEngine (route strips + the CompanyModel stripVoiceEngineFlagFieldsFromUpdate hook, incl. $set/$rename and dotted paths).
  • company.voice_engine_providers — { google?, openai?, xai?, openai_live? }. Vendor key openai: absent = allowed, only explicit false denies. Opt-in keys google (since 1f0bafbbb, 2026-10-12), openai_live and xai (since 303d70e3b / backend 5c164b0a6, 2026-11-06): only explicit true permits; openai_live additionally needs openai not false. One rule in four places: backend voiceEngineModelAllowed, ms-ai isVoiceProviderPermitted, ms-comm VoiceProviderHelper, frontend providerEnabledFor. Open-to-all models skip all of it.
  • Backend refuses newly saving a non-permitted S2S model — 403 VOICE_ENGINE_NOT_ENABLED / VOICE_ENGINE_PROVIDER_NOT_ENABLED / VOICE_ENGINE_MODEL_RETIRED (voiceEngineRejectionFor). UX only; ms-ai routing re-checks per session.
  • Turning the gate off calls ms-ai POST /voice-engine/company-gate/:company_id/revoke-inbound so inbound agents are re-synced onto ElevenLabs.

Routing decision​

ms-ai/src/voice-engine/voice-engine.routing.ts → voiceRoutingReason() is the one place the provider is chosen. A session goes S2S only when all hold:

  1. the channel's model is an S2S id (not_s2s_model otherwise);
  2. the role (web / inbound / outbound) is enabled — VOICE_ENGINE_ROLES (default web,inbound,outbound; none/off = everything on ElevenLabs, web = pull telephony back) and VOICE_S2S_MODE != off (role_disabled);
  3. voice_engine_enabled === true (company_gate) — skipped for an open-to-all model;
  4. the model's provider isn't switched off (company_gate_provider) — also skipped for open-to-all.

Model selection is per channel and must stay that way: web = ai_config.call_model; inbound = ai_inbound_call.model / ai_inbound_call_agents[].model; outbound = the dialled agent's model in ai_outbound_call_agents (resolved by agent_key, not index 0); SDK voice agents = their own model (blank = the web agent's). New inbound/outbound agents default to claude-sonnet-4-5 (must stay in CALL_MODEL_CATALOG, or sync silently swaps in ELEVENLABS_INNER_LLM).

Fallback is always ElevenLabs. Both telephony roles keep a live ElevenLabs standby agent (inbound re-synced on every save with allowS2SFallback; outbound provisioned lazily on the first S2S dial), so a declined or failed S2S session degrades instead of dropping. Deliberate web fallbacks log a DirectWebFallbackReason: direct_mint_failed, direct_web_unbound, direct_web_disabled, direct_concurrency_limited, vapi_unsupported_model, vapi_call_failed, vapi_not_configured. Gotcha (2026-09-16 incident): narrowing VOICE_ENGINE_ROLES sends a company whose voice is also provider-native (e.g. Gemini "Puck") to an ElevenLabs agent with no such voice → sync 400 voice_not_found, widget stuck on "Connecting to AI...".

Serving modes (VOICE_S2S_MODE)​

ModeAudio pathStatus
directms-ai mints a short-lived vendor credential; the peer (widget, or ms-comm's Twilio/Telnyx bridge) talks straight to the vendorPROD + DEV
vapiVapi runs the call with a transient assistant built per call — OpenAI realtime models onlybuilt, not active
engineself-hosted ms-voice-engine (Pipecat) sits in the media path; doesn't serve GPT-Livedeployed, dormant
offno S2S — every call on ElevenLabs. Unset/blank/unknown resolves to offkill switch

VOICE_S2S_WEB_MODE (direct|vapi) can move the widget alone; off still wins. All read per call — change the configmap and restart ms-ai (no Reloader in the cluster). A Pipecat Cloud backend was added (024ff0c26) and removed again the next day (7825ce5de) — there is no Pipecat Cloud / Daily path; Gemini stays on direct.

Direct mode​

ms-ai/src/voice-engine/direct/ (voice-direct.service.ts, one minter per vendor: openai-realtime.minter.ts, gemini-live.minter.ts, xai-realtime.minter.ts; GPT-Live session building in openai-live.session.ts). The session response (VoiceDirectSessionResponse, provider: 'voice-direct') carries connect_protocol:

PeerVendorProtocol
widgetOpenAI (Realtime + GPT-Live)webrtc-relay — the browser sends its SDP offer via ms-comm to ms-ai POST /voice-direct/webrtc-offer; ms-ai holds the key, so no vendor credential reaches the browser. ms-ai also joins the call on a server sideband socket (voice-direct-sideband.ts, VOICE_DIRECT_SIDEBAND default on; GPT-Live attaches via liveAttachUrl) and runs server tools itself; the browser relay is the fallback.
widgetGeminiws-relay — browser streams PCM16 to ms-comm VoiceDirectWebMediaRelay, which holds the Gemini socket and is billed from its exact token count. On by default since 7825ce5de (VOICE_DIRECT_WEB_GEMINI_RELAY=false is the kill switch). Widget plays the agent through a local RTCPeerConnection loopback into the <audio> element (keeps the echo-canceller reference; ?vd_out=element or ?vd_out=ctx selects the old routes).
widgetGemini over WebRTCSame relay, but the media leg runs over our self-hosted LiveKit (ms-comm LiveKitMediaLeg.js, 137f2b9; widget bd354a2, livekit-client 2.16.1) when ms-comm advertises media_transport: 'livekit': the widget asks voice-direct-livekit-join and joins the session's room (mic track in, agent track out at 24 kHz, relay events on the vd data channel); ms-comm joins as agent and hands VoiceDirectWebMediaRelay a socket-shaped shim, so the relay is unchanged. Opus/UDP with jitter buffer + loss recovery instead of PCM over TCP. On when VOICE_DIRECT_LIVEKIT=true + LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET (LIVEKIT_PUBLIC_URL = what the browser dials); ?vd_transport=ws forces the socket. Needs a glibc image — ms-comm moved to node:22-bookworm-slim (8f3fc7d) because @livekit/rtc-node has no musl binding.
widgetxAIws-relay through VoiceDirectWebMediaRelay, always (303d70e3b): xAI's token binds no prompt or tools, so they must travel with whoever holds the socket — never a browser. ms-comm runs no caller transcriber for xAI (Grok's own transcription). VOICE_DIRECT_WEB_ALLOW_UNBOUND is gone; Grok web needs xai in VOICE_DIRECT_WEB_VENDORS.
ms-comm (phone)xAI, XAI_SIP_ENABLED=trueGrok over SIP (ms-comm b158b29, XaiSipCall.js): the voice webhook answers with grokSipTwiml — <Dial answerOnBridge record="record-from-answer-dual"> to sip.voice.x.ai with signed x-kk-* SIP headers (company, agent, agent key, direction, call, exp/sig). xAI posts a Standard-Webhooks-signed realtime.call.incoming to POST /xai-sip/incoming/:number; ms-comm joins the call (wss://api.x.ai/v1/realtime?call_id=), sends the session (S2S prompt, tools, disclosure, CRM keyterms) and runs the tools. Caller audio never passes through ms-comm — no live listen/takeover on this leg. Numbers register at xAI on first use (paged lookup, Redis per-number lock); the per-number webhook secret xAI returns only at registration is kept in Mongo xai_sip_numbers (number, phone_number_id, webhook_secret). Twilio callbacks: POST /xai-sip/leg-ended (statusCallback → end the session) and POST /xai-sip/recording (stores twilio_recording_sid on the call row), both HMAC-signed ?call=&sig=. XAI_SIP_ENABLED=false keeps Grok agents on the media bridge (RealtimeAdapter).
ms-comm (phone)Realtime / Gemini / xAIwebsocket; ms-comm adapters RealtimeAdapter.js (OpenAI/xAI) and GeminiLiveAdapter.js (streaming upsampler, opener retry; with GEMINI_API_KEY in ms-comm it opens with its own key + POST /voice-direct/setup, else the single-use token path) in app/services/voiceDirect/.
ms-comm (phone)GPT-LiveLiveAdapter.js — the Live API mints no ephemeral credential, so ms-comm opens wss://api.openai.com/v1/live/sessions with its own OpenAI key (VOICE_DIRECT_OPENAI_API_KEY → OPENAI_API_KEY) and sends session.start from POST /voice-direct/setup (answer carries client_secret: 'api-key'). ms-comm advertises live_capable: true; a peer without it is declined live_phone_unsupported → ElevenLabs.
  • Widget: widget/src/services/directS2SCall.ts (mic with echo-cancel/NS/AGC, agent audio only through an <audio> element, playout jitter buffer 150 ms). s2sSession.ts holds the shared session interface; vapiCall.ts is the Vapi path.
  • Phone pre-warm: the vendor session is opened while the phone rings; the direct answer (a credential) is parked in Redis vd:warm:<callSid> and taken with GETDEL, so whichever ms-comm replica Twilio delivers the media stream to adopts it exactly once. While minting, the pre-warm publishes vd:warming:<callSid> (ms-comm 5a0323a): a media stream that lands on another replica ~300 ms after the webhook and finds the marker polls for the credential (bounded by the start budget, stopped by caller hang-up) instead of starting cold — which was refused call_sid_in_use and fell back to ElevenLabs. No marker, no wait.
  • ms-ai endpoints (secret auth): POST /voice-direct/{tool,heartbeat,webrtc-offer,refresh,setup,usage,session-end,web-analysis,trace}. A sweeper closes sessions whose heartbeats stop (VOICE_DIRECT_SWEEP_GRACE_S).
  • Limits: credential TTL ≤ 120 s (hard clamp), VOICE_DIRECT_MAX_CONCURRENT_WEB_PER_COMPANY (10), per-session tool caps (VOICE_DIRECT_TOOL_MAX_PER_SESSION 60, _MAX_PER_MIN 30, _TIMEOUT_MS 30000; ms-comm waits 35 s for a tool).
  • Tools on direct calls = ENGINE_HOSTED_TOOLS (voice-engine.tools.ts): KB, products, orders, calendar (check/book/find/reschedule/cancel), the phone tools from elevenlabs/phone-tool-configs.ts (create_simpro_job, check_simpro_availability, validate_address, send_sms) and — inbound only — caller verification (company_user_authentication, 027255494): run in-process by VoiceDirectToolRunner with this session's company + agent_key (no HTTP, no secret); the model supplies only username/password, which VoiceDirectTrace scrubs. Prompts advertise exactly these tools (engineHostedTools).
  • Turn-taking (read per session): Gemini's own detection is off on every Gemini Live session ms-comm can drive — see Gemini turn-taking; the Gemini VAD knobs below (VOICE_DIRECT_GEMINI_START_SENSITIVITY, default low, PROD high; _END_SENSITIVITY default high; _PREFIX_PADDING_MS 300; _SILENCE_MS 400; _TEMPERATURE) only apply when manual turns are rolled back or the phone peer isn't manual_turns_capable. Grok: VOICE_DIRECT_XAI_SILENCE_DURATION_MS (unset = xAI's tuning), VOICE_DIRECT_XAI_IDLE_TIMEOUT_MS (10000, server check-in on a silent caller; 0 = off), VOICE_DIRECT_XAI_REASONING (high default | none); OpenAI Realtime VOICE_DIRECT_OPENAI_EAGERNESS (semantic VAD, default auto; PROD auto — low made callers wait up to ~8 s), _NOISE_REDUCTION (near_field/far_field, default off; PROD near_field since flux d7c5e8b9, coughs were cutting the agent off). First-turn VAD (ea0afac4f): the caller's first turn ends on plain server VAD after VOICE_DIRECT_OPENAI_FIRST_TURN_SILENCE_MS (600, 0 = off), then session_update_after_first_turn switches to semantic VAD — web via the sideband, phone only when ms-comm says first_turn_vad. Realtime only (GPT-Live is full duplex, no turn detection), so it's dormant now the Realtime models are retired.
  • Greeting guard — stops a "Hello?" over the first words from making the vendor VAD cancel the greeting. Phone: ElevenLabsBridgeService._startGreetingGuard holds caller audio from the opener request until the greeting has played out (DIRECT_GREETING_GUARD_LIMITS maxMs 8000, noAudioMs 3000, tailMs 300). Web OpenAI Realtime: the widget does the same (not armed on GPT-Live — Live never sends the event it waits for). Gemini web relay: mic not forwarded until the opener turn ends (VOICE_DIRECT_OPENER_MIC_HOLD=false disables).
  • Call guard (phone) — ms-comm DirectCallGuard: end-intent classifier (VOICE_DIRECT_END_INTENT, model VOICE_DIRECT_END_INTENT_MODEL) plus an idle guard measured from the last activity of either side or the last tool run (so a long lookup isn't a dead line): one check-in after VOICE_DIRECT_IDLE_CHECKIN_S (40) of silence, hang-up VOICE_DIRECT_IDLE_HANGUP_S (35) after it. VOICE_DIRECT_CALL_GUARD=false turns it off.
  • Gemini guards (ms-comm GeminiLiveAdapter, 7463f08 → 22a93d0; prod 2026-10-16/17: Gemini 3.8 Live web calls went silent after the caller finished while the socket stayed healthy):
    • Turn commit — 1.2 s after our transcriber ended the caller's turn with no Gemini commit, send audioStreamEnd (Google's hybrid-VAD finalize prompt; doesn't provoke a reply to an answered turn). VOICE_DIRECT_GEMINI_COMMIT_MS (1200, 0 = off). The phone direct leg feeds the transcriber final into the same guard.
    • Stall nudge — no model output VOICE_DIRECT_STALL_NUDGE_MS (4000, 0 = nudge off, log stays) after the turn → one text nudge per turn, max 3 per call. Neither fires while the caller is audibly talking (frame RMS > 200, 800 ms hang).
    • Backpressure — caller frames are dropped while our socket backlog to Google exceeds ~96 KB (~2 s of audio); a backlog that persists 4 s resumes on a fresh socket.
    • Echo gate (web relay) — while the agent is playing (+250 ms tail), mic frames below VOICE_DIRECT_ECHO_GATE_RMS are silenced so its own voice leaking into the mic doesn't barge in; real speech still interrupts. Default 0 (off); PROD 600.
    • Fast recovery — VOICE_DIRECT_FAST_RECOVERY_MS (default off) nudges a turn our energy detector heard end but Gemini never registered.
    • Logs Gemini voiceActivity; STALL / commit lines carry vad_state, ws_buffered_bytes, frame flow, event-loop lag, gemini_heard, commit_tried.
  • External caller transcriber (Gemini) — Gemini's own input transcription invents words, so ms-comm CallerTranscriber.js feeds the caller audio to an OpenAI realtime transcription session (server VAD) and uses its finals. The model is the one ms-ai names as transcription_model in the start/refresh answer (since 6ed636544, 2026-11-02: gpt-transcribe, $0.0045 per audio minute, billed per minute; rollback VOICE_DIRECT_GEMINI_CALLER_TRANSCRIBE_MODEL=gpt-4o-transcribe or gpt-4o-mini-transcribe, both per token) and bills under, so they can't drift. Agent and company names go in as transcription hints (51e74c8). On a manual-turn session with VOICE_DIRECT_TURN_DETECTOR=soniox, the Soniox detector supplies the caller transcript instead (see below); without it ms-comm keeps its default (deploy ms-comm first). Lines are stamped with when they were spoken (caller: transcriber VAD speech start; agent: first fragment), emitted in that order with ts on the widget message, and held only while an earlier-started caller utterance is still transcribing (max 3 s). A caller transcript with no audio above the silence floor just before it is dropped (invented from silence); Gemini's text is the fallback if it never connects. Runs on the Gemini web relay (12a4423) and Gemini phone calls (8634235, mu-law in); agent echo is filtered and agent lines ordered after the caller line. ms-ai hands the agent language over as transcription_language on the session/refresh answer. Env: VOICE_DIRECT_CALLER_TRANSCRIBER=false (off), VOICE_DIRECT_CALLER_TRANSCRIBE_MODEL; needs the OpenAI key in ms-comm. Billed — see Billing.
  • Web call summary (a1c980b) — at the end of a widget S2S call ms-comm asks ms-ai POST /voice-direct/web-analysis for a summary + outcome and fills call_summary / call_outcome on the service room (only empty fields); the transcript email carries it. ElevenLabs web calls are untouched.
  • Keys: VOICE_DIRECT_<VENDOR>_API_KEY wins over the shared OPENAI_API_KEY / GEMINI_API_KEY / XAI_API_KEY, so voice spend can sit on its own vendor project. VOICE_DIRECT_WEB_VENDORS (unset = openai,gemini, empty = none) is the no-deploy kill switch for the widget; PROD = openai,gemini,xai.
  • Debug: ms-comm VOICE_DIRECT_DEBUG_CAPTURE / widget DirectAudioDebug trace audio quality (DEV). VOICE_DIRECT_TRACE_COMPANIES (* in PROD) writes one JSON trace per direct OpenAI web call (exact session body + every server event, caller-auth args scrubbed) to the recordings bucket under voice-engine/traces/<company>/<session>.json; read it with secret-auth POST /voice-direct/trace. Holds the full prompt and the caller's words.

Gemini turn-taking (manual turns)​

Since ms-ai c8beccb2e (2026-11-05) every Gemini Live model — 3.8 Live and Extended Thinking, web and phone — runs with Gemini's automaticActivityDetection disabled; ms-comm GeminiLiveAdapter sends activityStart / activityEnd itself. Reason: speaker echo at ~2 % made Gemini's detector cut the agent off mid-sentence, and it cut callers off in pauses (harness: 3.8 Live 6/10 → 8/10). ms-ai stores manual_turns on the session snapshot and returns it on mint/refresh; web is always manual through the relay, phone only when ms-comm's bridge sends manual_turns_capable (an older ms-comm keeps Gemini's detection). VOICE_DIRECT_GEMINI_MANUAL_TURNS=false (ms-ai) rolls back to Gemini's detector. ms-comm pieces (51e74c8, 137f2b9, a335746, 975d963, c87f959):

  • Turn controller — opens a turn from our own level detector with the agent's playback masked (echo-aware), replays ~1.5 s of the caller's masked first words on barge-in (pre-roll, in order), recovers false barge-ins; barge-in bar at 40 % of the caller's learned level. Phone tracks agent playback end from mu-law bytes. Silence check-ins at 15 s then 10 s; promise-stall guard (VOICE_DIRECT_GEMINI_PROMISE_MS) when the agent says it will check and goes quiet.
  • Soniox end-of-turn (SonioxTurnDetector.js, VOICE_DIRECT_TURN_DETECTOR=soniox
    • SONIOX_API_KEY; without it, the loudness detector) — streaming transcript + <end> endpoints; a turn opens only on words with near-mic speech; endpoint wait depends on utterance shape (question / statement / trailing word) and grows when the caller resumes. Off-mic filter: a final word counts as the caller's only if the audio around it was ≥25 % of their learned level, so a TV or the room stays out of the transcript and the endpoint text. Runs at 8 kHz on carrier mu-law for phone. Diarization is log-only (VOICE_DIRECT_SONIOX_DIARIZE); VOICE_DIRECT_SONIOX_MODEL / _URL / _MAX_ENDPOINT_DELAY_MS. Soniox usage is a platform cost — nothing bills it to the tenant.
  • Near-mic gate (NearMicGate.js, VOICE_DIRECT_NEAR_MIC_GATE=true) — mutes audio far quieter than the caller before it reaches Gemini / Soniox; adaptive bar (VOICE_DIRECT_NEAR_MIC_GATE_ADAPTIVE=false = fixed VOICE_DIRECT_NEAR_MIC_GATE_RMS, _HOLD_MS). The recording keeps the raw mic.
  • Wait fillers (WaitFiller.js) — soft hums in the session voice while a lookup runs (clips under app/assets/voice-fillers/<voice>, 8 kHz mu-law on phone; the real answer clears them). Off by default — VOICE_DIRECT_WAIT_FILLERS=true, and only for English sessions. ms-ai's /voice-direct/refresh returns the session voice for them.
  • Prompt side (ms-ai d6b2d11d2, c8beccb2e): both Gemini models share one S2S platform prompt — answer the caller mid-lookup, one lookup per question, always the latest request, and always say what you are checking before a lookup, even where tenant instructions ask for very short lookup lines.

PROD (flux): ms-comm VOICE_DIRECT_TURN_DETECTOR: soniox, VOICE_DIRECT_NEAR_MIC_GATE: "true", VOICE_DIRECT_LIVEKIT: "true" (LIVEKIT_URL in-cluster livekit-signal-service:7880, LIVEKIT_PUBLIC_URL wss://livekit.knockknockapp.ai), secrets SONIOX_API_KEY, LIVEKIT_API_KEY, LIVEKIT_API_SECRET. LiveKit itself is Apps/PROD/livekit/ (f6587fc2): livekit/livekit-server:v1.9.11, one replica (rooms in memory, no Redis, fixed UDP port), signal 7880 behind an IngressRoute, RTC TCP 7881 / UDP 7882.

GPT-Live​

gpt-live-1 is a different OpenAI product from Realtime: a full-duplex speech model that delegates every fact, lookup, tool call and booking to a backend text model (delegation.responses).

  • Reasoning model — per agent: ai_config.call_gpt_live_reasoning_model (widget) / gpt_live_reasoning_model on inbound/outbound agents; '' = env default VOICE_DIRECT_LIVE_BACKEND_MODEL (gpt-6-luna). Clamped to LIVE_BACKEND_MODELS (gpt-6-luna, gpt-6-sol, gpt-6.1-sol, gpt-5.6-terra) — a name outside it would bill at zero. gpt-6-astra was dropped (0da8c0b6f, ~100× Luna's price). VOICE_DIRECT_LIVE_REASONING_EFFORT (default low; none lifted to low for models that refuse it), VOICE_DIRECT_LIVE_MAX_OUTPUT_TOKENS (400).
  • Two instruction sets — agent_instructions_gpt_live_speech + agent_instructions_gpt_live_reasoning on ai_calling and on each inbound/outbound agent (strict subdocs — declared). Backend gptLiveInstructionsField sends gpt_live_instructions: { speech, reasoning }
    • gpt_live_reasoning_model only for a permitted GPT-Live agent; ms-comm does the same for the widget. ms-ai ElevenlabsService.getLiveAgentInstructions assembles both prompts in code (elevenlabs/live-agent-prompts.ts: voice delivery, delegation policy, guardrails, end-call rules; backend capabilities, tool rules, booking/transfer/timezone/company sections). An empty speech half falls back to the agent's S2S/standard prompt; an empty reasoning half = platform blocks only. No LLM on the call path.
  • LivePromptSplitService (direct/live-prompt-split.service.ts, table voice_live_prompt_split) — LLM migration of one authored prompt into the two halves (cleanup of ElevenLabs artefacts, restructure, split, audit report), cached by sha256 of the text, SPLITTER_VERSION 2, model VOICE_DIRECT_LIVE_SPLIT_MODEL (gpt-6-sol), timeout VOICE_DIRECT_LIVE_SPLIT_TIMEOUT_MS (240 s, runs in the background), VOICE_DIRECT_LIVE_SPLIT=false off. Only exposed as secret-auth helpers: POST /voice-engine/live-prompt-split (convert a custom prompt) and POST /voice-engine/live-prompt-preview (the two prompts a call would run on, {{vars}} unrendered). The dashboard's Use default speech/reasoning prompt buttons use frontend/src/utils/gptLiveDefaultPrompts.js instead; switching an agent to gpt-live-1 with nothing stored seeds both defaults on save (defaultGptLiveOnModelSave).
  • Widget specifics (aa00735) — no Realtime greeting request (Live rejects it as unknown_parameter), no greeting guard, "agent speaking" read from Live transcript chunks; the transcript is assembled from Live's deltas (widget/src/services/liveTranscript.ts).

Vapi mode​

ms-ai/src/voice-engine/vapi/: vapi-web-call.starter.ts (widget joins over Daily — widget vapiCall.ts), vapi-phone-call.starter.ts (numbers imported into Vapi via VapiTwilioNumberApi, gated by VAPI_PHONE_ENABLED; ms-comm VOICE_S2S_PHONE_MODE=vapi returns Vapi's TwiML from the voice webhook, and ms-comm POST /twilio/vapi-outbound-started/:company_id records an S2S outbound dial), vapi-assistant.builder.ts (transient assistant: first message, language, endpointing VAPI_ENDPOINTING, denoise VAPI_DENOISE, logging VAPI_LOGGING_ENABLED). Webhook POST /voice-vapi/webhook (VAPI_WEBHOOK_TOKEN, ≥ 24 chars). Billing is at cost — what Vapi charged — polled and reconciled every 5 min (voice-vapi:poll, voice-vapi:reconcile-costs); calls whose cost can't be trusted are held for review (GET /api/cost-tracking/admin/vapi/held, POST .../held/:session_id/release). Admin screen: Cost-Tracking › Vapi. PROD config (VAPI_ORG_ID, VAPI_PRIVATE_KEY, VAPI_WEBHOOK_TOKEN) is present but PROD runs direct with VAPI_PHONE_ENABLED=false.

Engine mode (ms-voice-engine)​

A separate Python/Pipecat service — no repo in this workspace; deployment only in flux-deployments/Apps/{DEV,PROD}/ms-voice-engine/. StatefulSet (image kk-ms-voice-engine-repo-prod, port 7860, HPA 2–6, PDB minAvailable: 1, terminationGracePeriodSeconds: 900), ingress voice-engine.knock-knockapp.com. ms-ai creates sessions (VOICE_ENGINE_URL, public VOICE_ENGINE_PUBLIC_URL), the engine calls back POST /voice-engine/session-callback (host must be in VE_CALLBACK_ALLOWED_HOSTS), ms-comm's bridge asks POST /voice-engine/telephony/session. Engine config: model allowlists ALLOWED_GEMINI_MODELS / ALLOWED_OPENAI_MODELS / ALLOWED_XAI_MODELS (an empty list makes the provider unreachable — OpenAI is held empty since 2026-09-18), Redis session registry on db 3 (REDIS_PASSWORD must be in its secret or /ready stays 503), MAX_CONCURRENT_SESSIONS 20 / MAX_SESSIONS_PER_POD 12 / WEB_RESERVED_SLOTS 4, DRAIN_TIMEOUT_S 600 (a deploy cuts calls longer than this), TURN_DETECTION=provider, SMART_TURN_STOP_SECS, VE_PARTIAL_REPORTS (keep false until ms-ai bills partial reports). Nothing routes here while VOICE_S2S_MODE=direct.

Instructions​

Every voice agent stores two prompts: agent_instructions (ElevenLabs, with [warmly]-style expression tags) and agent_instructions_s2s (same content, no tags — S2S models would read tags aloud). The S2S copy is used when the agent's model is S2S and the company is permitted it (agentInstructionsFor in backend for inbound/outbound; ms-comm CallsController._webCallSystemPrompt for the widget); an empty S2S copy falls back to the standard one.

  • Behavior tabs show one box at a time: "Agent Instructions for speech-to-speech (S2S) models" with a Use default S2S prompt button (frontend/src/utils/s2sDefaultPrompt.js — generic, company name baked in, booking workflow included only if a calendar is connected at the time). s2sInstructions.js derives a tag-free copy. Onboarding saves the generic S2S prompt for inbound/outbound agents.
  • Greeting: the same split for the opener — ai_calling.greeting_message_s2s (widget, read by ms-comm) and opening_script_s2s on inbound/outbound agents (backend openingScriptFor), used under the same rule; empty falls back to greeting_message / opening_script. Behavior tabs show "Greeting Message for speech-to-speech (S2S) models" / "Opening Script for speech-to-speech (S2S) models" (hint: leave out expression tags) in place of the standard field.
  • GPT-Live agents show two boxes instead of the S2S one — see GPT-Live; the S2S greeting still applies to them.
  • Default S2S prompt rules (frontend 105bfaac): every reply is two or three short sentences and must end with a question or a clear next step; a reaction opens a reply, never is the whole reply; emotion cues for impressed / empathetic / clear moments with worked examples. Since 9d79f6a1 (2026-11-05) a While you look something up section replaces the old "under five words, never explain what you're checking" lookup line (it made Gemini say "let me check" and go silent; 16 of 22 PROD Gemini companies carried it): say in one or two sentences what you're checking and why, answer the caller mid-lookup, never state a result early. Existing saved prompts are not rewritten — the platform prompt overrides the old line instead (next bullet).
  • S2S platform prompt (ms-ai elevenlabs/s2s-agent-prompts.ts, a40cc5cb6): Gemini, Grok and legacy Realtime sessions get their own platform layer built from the same workflow sections, every rule once (no language_detection line, no "filler before the tool" rule, one tool rule with human lookup lines; end-call, thinking and language folded in). Both Gemini models share it, differences in code; Extended Thinking adds "answer first, never promise a lookup" (57b511969); Grok is pinned on it plus Grok delivery rules. VOICE_S2S_PLATFORM_PROMPT=legacy restores the old assembly; ElevenLabs and GPT-Live prompts are untouched.
  • Honest AI disclosure (backend a5ebca159, frontend 9d2e906e, 2026-11-12): the default inbound/outbound/playground prompts say "If asked whether you are an AI, say yes honestly" and no longer call the agent a "human representative". Only defaults changed — saved prompts keep their text. The separate per-agent spoken disclosure line (call_disclosure) is in Telephony / Inbound Call Agent.
  • ms-comm prefixes exactly one You are <agent> (a prompt that already names [agent_name] is stored as written). ms-ai adds an explicit language-response mandate (voice-prompt-rules.ts) — native-audio models take no language code, so only the prompt can hold the agent language. Since ms-comm 7463f08 S2S web calls no longer get ms-comm's "switch to whatever you hear" directive (it contradicted ms-ai and let the model drift); ms-ai's languageRules own it, built from agent_language: the default language for the whole call, a switch only on an explicit request (next sentence already in the new language), otherwise offer once. The ElevenLabs pipeline keeps its directive.
  • Agent language inherits (backend de172c75a, ms-comm 6fa0900, frontend 3ad468dd, 2026-10-27): a blank agent_language / ai_config.call_agent_language means "same as organization". Resolution: own value → company.language → en (backend: outbound dial, inbound sync, Platform API; ms-comm callLanguage(company, sdkEngine) for both web paths: SDK agent → widget voice agent → organization → en, used for the language hint, directive and default greeting). Changing company.language re-deploys inheriting inbound agents. Schema defaults and new agents start blank. The shared AgentLanguagePicker.vue shows Same as organization (<language>). One-shot admin migration Call agents follow the organization language (lib/inheritAgentLanguages, in Integrations Maintenance) clears stored languages that repeat what the call already speaks; it never changes a spoken language. Deploy order: backend → migration → ms-comm (or non-English widget voices fall back to the stored en).

Recording​

  • Policy (voice-engine-recording-policy.ts), every input fails closed: VOICE_ENGINE_RECORDING_ENABLED=true AND package allows recording AND agent hasn't opted out AND visitor hasn't refused consent AND the disclosure check passes (VOICE_ENGINE_RECORDING_DISCLOSURE_CHECK=off must be set explicitly, else the policy is invalid and records nothing). PROD: enabled, check off (product decision 2026-09-16 — disclosure handled like ElevenLabs calls).
  • Direct calls are recorded by the peer (there is no media hop to tap): ms-comm DirectCallRecorder on phone and Gemini web relay; the widget (src/services/recording/callRecorder.ts, MP3 via lamejs in a worker) on OpenAI web, uploaded to ms-comm POST /voice-direct/recording. Both land at backend PUT /internal/voice-engine/recordings/:company_id/:session_id (VOICE_ENGINE_RECORDING_API_KEY, same value in backend, ms-comm and engine secrets; ms-comm uses BACKEND_INTERNAL_URL), stored under S3 voice-engine/recordings/, max RECORDING_UPLOAD_MAX_BYTES (64 MiB).
  • Grok over SIP is the exception: Twilio records the bridged leg (record-from-answer-dual) and the recording stays at Twilio (twilio_recording_sid / twilio_account_sid on InboundOutboundCallModel, ms-comm + backend 5c164b0a6). Backend playback finds twilio_recording_sid and proxies through ms-comm GET /internal/recordings/phone/:sid (API-key gated), which streams it from Twilio with Range forwarded — main-account creds first, the company's own number token on 401/403.
  • The session id is a UUID stored in the same conversation_id / elabs_conversation_id field ElevenLabs uses; playback branches on it (isVoiceEngineSessionId in backend app/lib/voiceEngineRecording.js, frontend utils/callRecordingAudio.js) — UUID → backend streams from S3, conv_… → ms-ai proxies ElevenLabs.

Billing​

Direct calls bill the vendor's own usage, per token, the way the vendor prices the model (VoiceDirectService.bill, since 2026-09-29):

  • One usage_event vd:<session_id> with one usage_charge per unit, agent bucket voice_web / voice_inbound / voice_outbound.
  • Caller transcription is a separate event vd:<session_id>:transcription: OpenAI Realtime's in-session transcription (now gpt-4o-mini-transcribe, per token — da6c9d3cd; sessions minted earlier on whisper-1 bill per minute) and, since 57a081f18, ms-comm's external caller transcriber on Gemini calls (web relay and phone), reported with the session usage and priced under callerTranscriptionModelFor(model) — for Gemini gpt-transcribe ($0.0045 per audio minute, billed per minute — PER_MINUTE_TRANSCRIPTION_MODELS, own upsert seeder; 6ed636544), or the per-token gpt-4o-transcribe / gpt-4o-mini-transcribe on rollback; the in-session model for OpenAI.
  • GPT-Live (api: 'live'): the voice session is $0.05/min billed per second (live_seconds, cumulative — set, not summed) under the gpt-live-1 row; the delegated backend model's tokens are their own event vd:<session_id>:live_backend under that model's catalog row (voice-live-backend-models.seeder.ts). A choice outside LIVE_BACKEND_MODELS is never sent, so it can't bill at zero.
  • request_meta.usage_source: sideband (OpenAI web, server-counted — wins), ms_comm (phone), client (web without a sideband, marked usage_trust=client_reported). A dropped sideband can be beaten by a larger browser report (sideband_partial).
  • Never billed as zero: with no usage report, the model's provisional per-minute rate (voice-direct-per-minute-prices.seeder.ts) is charged, every row is_estimate=true, logged at ERROR. The boot pre-check errors on an S2S model without a minute row.
  • cost-tracking/voice-model-billing.ts says per model which ai_provider row prices it (bills under the model's own provider — a hardcoded google once billed OpenAI calls at zero), whether cached input is priced separately (OpenAI yes, Gemini no), and token vs minute mode.
  • Grok Voice (grok-voice-3.0): minute mode at $0.08/min of session, tokensUnpriced — xAI prices no tokens on it, so token counts stay in the event meta and bill nothing (no token price row exists; the old grok-voice-think-fast-2.0 rows were renamed). On Grok SIP calls the carrier side is billed separately to the telephony wallet by ms-ai's Twilio call-cost sweep: the SIP leg (to sip.voice.x.ai) joins its parent call's wallet activity line, and the call's Twilio recording is debited as telephony_recording ("Call recording" in the wallet, twilio-recording:<sid> idempotency key). Both keep sub-cent precision (subCent; backend TelephonyWalletController now rounds credits to 4 dp instead of whole cents — a SIP leg costs under ½¢). Every other call's recording stays free.
  • Credits flow into the normal token-usage sum and are deducted by AITokensCreditJob, like any AI usage. Vapi mode instead bills at Vapi's cost.

Operational limits​

  • Live listening and takeover work on direct S2S calls (since e2cb782 / a111cdf). ms-comm stamps s2s: true + voice_provider: 'voice-direct' (+ voice_vendor) on the call record, service room and user_session.call_status.
    • Phone (every vendor on the media bridge, incl. GPT-Live): the bridge fans caller/agent audio and final transcripts out to ai-listen:<service_room_id>; takeover ends the vendor session (agent_takeover) and keeps the carrier leg for the human.
    • Web (OpenAI Realtime, GPT-Live, Gemini, and xAI since b158b29 — both relayed through ms-comm): on request_audio_stream the widget taps the session's own audio graph (mic + agent) and streams 16 kHz PCM to ms-comm; takeover ends the direct session through the normal endAICall path and reuses the Twilio Video takeover flow.
    • Still refused (s2s_not_supported): Vapi calls. Grok over SIP phone calls can't be listened to or taken over at all (the audio never passes through ms-comm), though the frontend's isUnmonitorableCall doesn't know that and only excludes Vapi. Frontend isUnmonitorableCall(source, channel) mirrors ms-comm and shows "Live listening and takeover aren't available for this speech-to-speech call." (monitorable: false from the server also wins).
  • The widget agent's ElevenLabs agent-sync cache, background sound and speech settings don't apply.

Env reference​

WhereVars
ms-ai configmapVOICE_S2S_MODE (PROD direct), VOICE_S2S_WEB_MODE, VOICE_ENGINE_ROLES, VOICE_DIRECT_WEB_VENDORS (PROD openai,gemini,xai), VOICE_DIRECT_GEMINI_MANUAL_TURNS (default on), VOICE_S2S_PLATFORM_PROMPT (legacy = old assembly), VOICE_DIRECT_XAI_IDLE_TIMEOUT_MS / _SILENCE_DURATION_MS / _REASONING, VOICE_DIRECT_WEB_GEMINI_RELAY, VOICE_DIRECT_GEMINI_API_VERSION (default v1beta), VOICE_DIRECT_OPENAI_NOISE_REDUCTION (PROD near_field), VOICE_DIRECT_OPENAI_EAGERNESS (PROD auto), VOICE_DIRECT_OPENAI_FIRST_TURN_SILENCE_MS, VOICE_DIRECT_GEMINI_START_SENSITIVITY (PROD high), VOICE_DIRECT_GEMINI_CALLER_TRANSCRIBE_MODEL, VOICE_DIRECT_LIVE_* (backend model, reasoning effort, max output tokens, split on/model/timeout), VOICE_DIRECT_TRACE_COMPANIES (PROD *), other VOICE_DIRECT_* tuning, VAPI_ORG_ID, VAPI_DENOISE, VAPI_PHONE_ENABLED, VOICE_ENGINE_URL, VOICE_ENGINE_PUBLIC_URL, VOICE_ENGINE_AI_BASE_URL, VOICE_ENGINE_RECORDING_ENABLED, VOICE_ENGINE_RECORDING_DISCLOSURE_CHECK, VOICE_ENGINE_RECORDING_S3_BUCKET / _REGION (recording HEAD checks + traces), VOICE_ENGINE_TELEPHONY_SOFT_BUDGET_MS
ms-ai secretOPENAI_API_KEY, VOICE_DIRECT_GEMINI_API_KEY (PROD since flux acd25916; wins over GEMINI_API_KEY), XAI_API_KEY (PROD since flux d88deed1; VOICE_DIRECT_XAI_API_KEY wins), VOICE_ENGINE_RECORDING_S3_ACCESS_KEY_ID / _SECRET_ACCESS_KEY, VAPI_PRIVATE_KEY, VAPI_WEBHOOK_TOKEN
ms-commBACKEND_INTERNAL_URL; secret VOICE_ENGINE_RECORDING_API_KEY, OPENAI_API_KEY (or VOICE_DIRECT_OPENAI_API_KEY) — required for GPT-Live phone (LiveAdapter), the Gemini caller transcriber and the end-intent classifier; optional GEMINI_API_KEY (Gemini phone API-key path; PROD has none → token path); VOICE_DIRECT_CALL_GUARD, VOICE_DIRECT_END_INTENT*, VOICE_DIRECT_IDLE_CHECKIN_S / _HANGUP_S, VOICE_DIRECT_CALLER_TRANSCRIBER*, VOICE_DIRECT_OPENER_MIC_HOLD, VOICE_DIRECT_ECHO_GATE_RMS (PROD 600), VOICE_DIRECT_GEMINI_COMMIT_MS, VOICE_DIRECT_STALL_NUDGE_MS, VOICE_DIRECT_FAST_RECOVERY_MS, VOICE_DIRECT_GEMINI_PROMISE_MS, VOICE_DIRECT_GEMINI_TOOL_SCHEDULING; turn-taking VOICE_DIRECT_TURN_DETECTOR (PROD soniox), VOICE_DIRECT_SONIOX_*, VOICE_DIRECT_NEAR_MIC_GATE (PROD true) / _ADAPTIVE / _RMS / _HOLD_MS, VOICE_DIRECT_TURN_OPEN_RMS, VOICE_DIRECT_WAIT_FILLERS (default off), VOICE_DIRECT_FILLER_DIR; LiveKit VOICE_DIRECT_LIVEKIT (PROD true), LIVEKIT_URL, LIVEKIT_PUBLIC_URL; Grok XAI_SIP_ENABLED (PROD true), XAI_SIP_LOG_TRANSCRIPT; secrets SONIOX_API_KEY, LIVEKIT_API_KEY / _SECRET, XAI_API_KEY; VOICE_ENGINE_TELEPHONY_ENABLED / _COMPANIES (engine pilot, default off), VOICE_S2S_PHONE_MODE
backendsecret VOICE_ENGINE_RECORDING_API_KEY; MS_COMMUNICATION_URL + MS_COMM_API_KEY (recording proxy)
livekitApps/PROD/livekit/ (PROD only) — server config in its ConfigMap (port 7880, RTC TCP 7881 / UDP 7882), API key/secret in its secret (same pair as ms-comm's LIVEKIT_API_*)
ms-voice-engineconfigmap ms-voice-engine-cm (allowlists, Redis, capacity, drain, turn detection); secret GOOGLE_API_KEY, OPENAI_API_KEY, XAI_API_KEY, ALLOWED_XAI_MODELS, INTERNAL_SERVICE_TOKEN, VE_CALLBACK_AUTH_TOKEN, VOICE_ENGINE_RECORDING_API_KEY, REDIS_PASSWORD

Call Agent · Inbound Call Agent · Outbound Call Agent · Telephony · Cost-Tracking · ms-ai · ms-communication · widget · Deployment