Voice Engine (Speech-to-Speech)
Every AI voice call (widget, inbound phone, outbound phone) runs on one of two stacks:
- ElevenLabs Convai — the default. ElevenLabs does STT → inner LLM → TTS; the "AI Model" pick is the inner LLM. See Call Agent and Telephony.
- Native speech-to-speech (S2S) — one realtime model hears the caller and speaks back (offered today: OpenAI GPT Live 1, Google Gemini 3.8 Live and Gemini 3.8 Live Extended Thinking, open to every company; xAI Grok Voice, opt-in per company). Chosen by picking an S2S model in the same AI Model picker; everything below is about this path.
Eleven Realtime (eleven-realtime-v1-max, picker label "Eleven Realtime Max",
ms-ai abf56f790, 2026-10-23) is not this path: it stays an ordinary
ElevenLabs agent with conversation_config.realtime_model set
(ElevenlabsService.applyRealtimeModel, cleared on the next sync when the model
changes), keeps ElevenLabs voices/tools/speech settings and bills through the
ElevenLabs post-call credits. ms-ai appends explicit END CALL rules to its prompt
(elevenRealtimePrompt). eleven-realtime-v1-mini was removed 2026-10-26
(62d88daa2) — on real calls it announced tool actions without calling them; a
stale id falls back to the native default LLM.
The routing decision, the model catalog, and billing all live in ms-ai
(src/voice-engine/). The audio path depends on VOICE_S2S_MODE.
Model catalog
Source of truth: ms-ai/src/voice-engine/voice-engine.models.ts
(VOICE_ENGINE_MODELS, OPEN_TO_ALL_COMPANIES_MODELS, RETIRED_VOICE_MODELS).
Mirrored — add ids in the same change — in backend app/lib/voiceEngineModels.js,
ms-comm app/helpers/VoiceProviderHelper.js, frontend src/utils/voiceEngineModels.js
(an ms-ai drift spec parses the frontend file), and the picker catalog
ms-ai/src/elevenlabs/call-models.ts.
Offered (since 2026-10-13, 0da8c0b6f / backend e0c575e86 / frontend 105bfaac; Extended Thinking back 2026-11-02, Grok 2026-11-06):
Selector id (model_name) | Permission key | Vendor model / API | Picker label | Voices |
|---|---|---|---|---|
gpt-live-1 | openai_live | gpt-live-1 on OpenAI's Live API (api: 'live', /v1/live/sessions), full duplex | GPT Live 1 (S2S, full-duplex) | 17: marin (default), cedar, 12 locale voices (quartz, ripple, vesper, willow, stone, gleam, meridian, delta, cinder, beacon, bossa, tempo), alloy, echo, shimmer |
gemini-3.8-live | google | models/gemini-3.8-live (Live API, v1beta) | Gemini 3.8 Live (S2S) | 12 Gemini Live voices: Puck (default), Charon, Kore, Fenrir, Aoede, Zephyr, Erinome, Achernar, Laomedeia, Rasalgethi, Alnilam, Schedar |
gemini-3.8-live-extended-thinking | google | models/gemini-3.8-live-extended-thinking | Gemini 3.8 Live Extended Thinking (S2S, slower replies) | same 12 Gemini voices |
grok-voice | xai | grok-voice-3.0 (xAI realtime, OpenAI-Realtime wire format) | Grok Voice (S2S) | 4: Eve (default), Ara, Rex, Leo |
The three OpenAI/Gemini rows are in OPEN_TO_ALL_COMPANIES_MODELS: every
company may use them — no voice_engine_enabled, no provider key (routing,
save check, S2S prompt and greeting selection all short-circuit on it). Grok
stays on the per-company permission (opt-in xai, below).
- Gemini 3.8 Live now refuses any thinking setting (Google 1007 "Thinking
budget is not supported", since 2026-11-04) — the minter sends none
(
d6b2d11d2,24408bebe). - Extended Thinking (
isGeminiThinkingModel) rejects a setup without a thinking level, so it is sentthinkingLevel: LOW(accepts LOW/MEDIUM/HIGH, not MINIMAL;6ed636544). Google runs it differently: onlyNON_BLOCKINGtool declarations, it speaks its own fillers while reasoning, andturnCompleteno longer means done (interactionStatusdoes). Tool results go back withWHEN_IDLEscheduling (ms-commVOICE_DIRECT_GEMINI_TOOL_SCHEDULING=offrolls back). Onend_callit is allowed one goodbye before the hang-up, and lookup results still in flight are dropped (975d963,c87f959). Same per-token prices as 3.8 Live (Google's output price includes thinking tokens).
Retired (RETIRED_VOICE_MODELS): gpt-realtime, gpt-realtime-2,
gpt-realtime-2.1, gpt-realtime-2.1-mini, gemini-2.5-flash-native-audio,
gemini-3.1-flash-live. Still catalogued so
an in-flight or historical session resolves for billing/recordings, and still
recognised as S2S (an old saved id keeps its S2S instructions), but removed from
the picker catalog and refused at save for every company — backend 403
VOICE_ENGINE_MODEL_RETIRED ("This speech-to-speech model is no longer available.
Choose GPT Live 1 or Gemini 3.8 Live."). Re-sending an unchanged stored value is
still allowed so the rest of the form saves; the picker shows a saved retired id
as "(Not available — runs on ElevenLabs)". (The backend message still names only
"GPT Live 1 or Gemini 3.8 Live".) Note ms-ai routing does not check
RETIRED_VOICE_MODELS — a company still holding a retired id with
voice_engine_enabled on would keep routing S2S.
Voice ids are provider-native and never shared across providers — a voice from one vendor sent to another is refused at the handshake (a dead call); GPT-Live voices must never be defaulted onto a Realtime model or vice versa. Changing the model in the UI clears a voice the new model can't play. Gemini 3.8 tool calling is not in Google's Live tool matrix — it works on the direct path but smoke-test new tools on stage.
Which models a tenant can actually pick is the intersection of:
- the frontend allowlist
VAPI_SERVED_MODELS(gpt-live-1,gemini-3.8-live,gemini-3.8-live-extended-thinking,grok-voice); - ms-ai
GET /ai-model-pricing/estimates/:company_id, which drops any S2S model the current mode can't serve (isS2SModelServed: indirectmode, only vendors whose API key is present in ms-ai); - for anything not open-to-all, the company permission (below).
Production (2026-11-06): ms-ai PROD has the OpenAI key,
VOICE_DIRECT_GEMINI_API_KEY (flux acd25916) and XAI_API_KEY, with
VOICE_DIRECT_WEB_VENDORS: 'openai,gemini,xai' (flux d88deed1); ms-comm PROD has
XAI_API_KEY + XAI_SIP_ENABLED: "true" (flux 43e353a7). So the three open
models are offered to every tenant on web, inbound and outbound, and Grok to
companies with xai on.
Picker behaviour (buildCallModelOptions): in direct mode S2S rows carry no
credits/minute price (only vapi mode shows a trailing-30-day estimate). The
"No live listening or takeover on web calls" hint is shown for providers outside
the picker's MONITORABLE_PROVIDERS (openai, openai_live, google) — i.e. on
the Grok row, although xAI web calls are in fact monitorable now
(MONITORABLE_WEB_VENDORS includes xai); see Operational limits. A saved S2S
model the company can't use stays visible, suffixed "(Requires speech-to-speech
access)", "(Provider not enabled)" or "(Not available — runs on ElevenLabs)". For
an S2S model the AI Engine tab hides background sound and speech settings
(ElevenLabs-only; values kept). On gpt-live-1 the AI Engine tab adds a
Reasoning Model (GPT-Live) select (see GPT-Live). Voice previews (since 2026-11-06): pickers play the clip
saved in the super-admin Voice Previews page first (frontend
views/admin/voice-previews/List.vue, 01cba755), then a static file. Backend
AdminVoicePreviewController (GET /voice-previews for pickers;
GET /admin/voice-previews, POST /admin/voice-previews/:provider/:voice_id/{generate,upload},
POST /admin/voice-previews/generate-missing → background job polled at
GET /admin/voice-previews/jobs/:job_id, one retry per failure for Gemini's
rate limit) stores clips as files rows on S3 linked per voice in
voice_previews (provider, voice_id, file, source
generated|uploaded, unique on provider+voice). Generation goes through ms-ai
GET /voice-engine/previews/catalog + POST /voice-engine/previews/generate
(a4f24bef8) with the provider keys ms-ai holds. GPT-Live voices have no TTS
twin, so they are not generatable — only their clips stay in the repo
(frontend/public/voice-previews/openai/); all other static clips were removed.
Company permission
The general provider panel is gone (frontend 105bfaac deleted
VoiceEngineProviders.vue and useVoiceEngineGate.js). Grok is the one offered
model still gated, so the super-admin company page (admin/companies/Detail.vue)
has a single Grok Voice switch (GrokVoiceToggle.vue, 34997d6c, "Web,
inbound and outbound calls"): on sends PUT /company/:_id/voice-engine
{ enabled: true, providers: { xai: true } } (sets the main flag too), off sends
{ providers: { xai: false } }. If the off-save fails with a 5xx, the toggle
offers Retry revoke — stored phone agents may still reach Grok until the
revoke (below) succeeds.
company.voice_engine_enabled— main flag (defaulttruefor companies created since 2026-09-30); writable only throughtoggleVoiceEngine(route strips + the CompanyModelstripVoiceEngineFlagFieldsFromUpdatehook, incl.$set/$renameand dotted paths).company.voice_engine_providers—{ google?, openai?, xai?, openai_live? }. Vendor keyopenai: absent = allowed, only explicitfalsedenies. Opt-in keysgoogle(since1f0bafbbb, 2026-10-12),openai_liveandxai(since303d70e3b/ backend5c164b0a6, 2026-11-06): only explicittruepermits;openai_liveadditionally needsopenainotfalse. One rule in four places: backendvoiceEngineModelAllowed, ms-aiisVoiceProviderPermitted, ms-commVoiceProviderHelper, frontendproviderEnabledFor. Open-to-all models skip all of it.- Backend refuses newly saving a non-permitted S2S model — 403
VOICE_ENGINE_NOT_ENABLED/VOICE_ENGINE_PROVIDER_NOT_ENABLED/VOICE_ENGINE_MODEL_RETIRED(voiceEngineRejectionFor). UX only; ms-ai routing re-checks per session. - Turning the gate off calls ms-ai
POST /voice-engine/company-gate/:company_id/revoke-inboundso inbound agents are re-synced onto ElevenLabs.
Routing decision
ms-ai/src/voice-engine/voice-engine.routing.ts → voiceRoutingReason() is the one
place the provider is chosen. A session goes S2S only when all hold:
- the channel's model is an S2S id (
not_s2s_modelotherwise); - the role (
web/inbound/outbound) is enabled —VOICE_ENGINE_ROLES(defaultweb,inbound,outbound;none/off= everything on ElevenLabs,web= pull telephony back) andVOICE_S2S_MODE != off(role_disabled); voice_engine_enabled === true(company_gate) — skipped for an open-to-all model;- the model's provider isn't switched off (
company_gate_provider) — also skipped for open-to-all.
Model selection is per channel and must stay that way: web =
ai_config.call_model; inbound = ai_inbound_call.model / ai_inbound_call_agents[].model;
outbound = the dialled agent's model in ai_outbound_call_agents (resolved by
agent_key, not index 0); SDK voice agents = their own model (blank = the web
agent's). New inbound/outbound agents default to claude-sonnet-4-5 (must stay in
CALL_MODEL_CATALOG, or sync silently swaps in ELEVENLABS_INNER_LLM).
Fallback is always ElevenLabs. Both telephony roles keep a live ElevenLabs
standby agent (inbound re-synced on every save with allowS2SFallback; outbound
provisioned lazily on the first S2S dial), so a declined or failed S2S session
degrades instead of dropping. Deliberate web fallbacks log a
DirectWebFallbackReason: direct_mint_failed, direct_web_unbound,
direct_web_disabled, direct_concurrency_limited, vapi_unsupported_model,
vapi_call_failed, vapi_not_configured. Gotcha (2026-09-16 incident): narrowing
VOICE_ENGINE_ROLES sends a company whose voice is also provider-native (e.g.
Gemini "Puck") to an ElevenLabs agent with no such voice → sync 400
voice_not_found, widget stuck on "Connecting to AI...".
Serving modes (VOICE_S2S_MODE)
| Mode | Audio path | Status |
|---|---|---|
| direct | ms-ai mints a short-lived vendor credential; the peer (widget, or ms-comm's Twilio/Telnyx bridge) talks straight to the vendor | PROD + DEV |
| vapi | Vapi runs the call with a transient assistant built per call — OpenAI realtime models only | built, not active |
| engine | self-hosted ms-voice-engine (Pipecat) sits in the media path; doesn't serve GPT-Live | deployed, dormant |
| off | no S2S — every call on ElevenLabs. Unset/blank/unknown resolves to off | kill switch |
VOICE_S2S_WEB_MODE (direct|vapi) can move the widget alone; off still wins.
All read per call — change the configmap and restart ms-ai (no Reloader in the
cluster). A Pipecat Cloud backend was added (024ff0c26) and removed again the
next day (7825ce5de) — there is no Pipecat Cloud / Daily path; Gemini stays on
direct.
Direct mode
ms-ai/src/voice-engine/direct/ (voice-direct.service.ts, one minter per vendor:
openai-realtime.minter.ts, gemini-live.minter.ts, xai-realtime.minter.ts;
GPT-Live session building in openai-live.session.ts).
The session response (VoiceDirectSessionResponse, provider: 'voice-direct')
carries connect_protocol:
| Peer | Vendor | Protocol |
|---|---|---|
| widget | OpenAI (Realtime + GPT-Live) | webrtc-relay — the browser sends its SDP offer via ms-comm to ms-ai POST /voice-direct/webrtc-offer; ms-ai holds the key, so no vendor credential reaches the browser. ms-ai also joins the call on a server sideband socket (voice-direct-sideband.ts, VOICE_DIRECT_SIDEBAND default on; GPT-Live attaches via liveAttachUrl) and runs server tools itself; the browser relay is the fallback. |
| widget | Gemini | ws-relay — browser streams PCM16 to ms-comm VoiceDirectWebMediaRelay, which holds the Gemini socket and is billed from its exact token count. On by default since 7825ce5de (VOICE_DIRECT_WEB_GEMINI_RELAY=false is the kill switch). Widget plays the agent through a local RTCPeerConnection loopback into the <audio> element (keeps the echo-canceller reference; ?vd_out=element or ?vd_out=ctx selects the old routes). |
| widget | Gemini over WebRTC | Same relay, but the media leg runs over our self-hosted LiveKit (ms-comm LiveKitMediaLeg.js, 137f2b9; widget bd354a2, livekit-client 2.16.1) when ms-comm advertises media_transport: 'livekit': the widget asks voice-direct-livekit-join and joins the session's room (mic track in, agent track out at 24 kHz, relay events on the vd data channel); ms-comm joins as agent and hands VoiceDirectWebMediaRelay a socket-shaped shim, so the relay is unchanged. Opus/UDP with jitter buffer + loss recovery instead of PCM over TCP. On when VOICE_DIRECT_LIVEKIT=true + LIVEKIT_URL / LIVEKIT_API_KEY / LIVEKIT_API_SECRET (LIVEKIT_PUBLIC_URL = what the browser dials); ?vd_transport=ws forces the socket. Needs a glibc image — ms-comm moved to node:22-bookworm-slim (8f3fc7d) because @livekit/rtc-node has no musl binding. |
| widget | xAI | ws-relay through VoiceDirectWebMediaRelay, always (303d70e3b): xAI's token binds no prompt or tools, so they must travel with whoever holds the socket — never a browser. ms-comm runs no caller transcriber for xAI (Grok's own transcription). VOICE_DIRECT_WEB_ALLOW_UNBOUND is gone; Grok web needs xai in VOICE_DIRECT_WEB_VENDORS. |
| ms-comm (phone) | xAI, XAI_SIP_ENABLED=true | Grok over SIP (ms-comm b158b29, XaiSipCall.js): the voice webhook answers with grokSipTwiml — <Dial answerOnBridge record="record-from-answer-dual"> to sip.voice.x.ai with signed x-kk-* SIP headers (company, agent, agent key, direction, call, exp/sig). xAI posts a Standard-Webhooks-signed realtime.call.incoming to POST /xai-sip/incoming/:number; ms-comm joins the call (wss://api.x.ai/v1/realtime?call_id=), sends the session (S2S prompt, tools, disclosure, CRM keyterms) and runs the tools. Caller audio never passes through ms-comm — no live listen/takeover on this leg. Numbers register at xAI on first use (paged lookup, Redis per-number lock); the per-number webhook secret xAI returns only at registration is kept in Mongo xai_sip_numbers (number, phone_number_id, webhook_secret). Twilio callbacks: POST /xai-sip/leg-ended (statusCallback → end the session) and POST /xai-sip/recording (stores twilio_recording_sid on the call row), both HMAC-signed ?call=&sig=. XAI_SIP_ENABLED=false keeps Grok agents on the media bridge (RealtimeAdapter). |
| ms-comm (phone) | Realtime / Gemini / xAI | websocket; ms-comm adapters RealtimeAdapter.js (OpenAI/xAI) and GeminiLiveAdapter.js (streaming upsampler, opener retry; with GEMINI_API_KEY in ms-comm it opens with its own key + POST /voice-direct/setup, else the single-use token path) in app/services/voiceDirect/. |
| ms-comm (phone) | GPT-Live | LiveAdapter.js — the Live API mints no ephemeral credential, so ms-comm opens wss://api.openai.com/v1/live/sessions with its own OpenAI key (VOICE_DIRECT_OPENAI_API_KEY → OPENAI_API_KEY) and sends session.start from POST /voice-direct/setup (answer carries client_secret: 'api-key'). ms-comm advertises live_capable: true; a peer without it is declined live_phone_unsupported → ElevenLabs. |
- Widget:
widget/src/services/directS2SCall.ts(mic with echo-cancel/NS/AGC, agent audio only through an<audio>element, playout jitter buffer 150 ms).s2sSession.tsholds the shared session interface;vapiCall.tsis the Vapi path. - Phone pre-warm: the vendor session is opened while the phone rings; the
direct answer (a credential) is parked in Redis
vd:warm:<callSid>and taken withGETDEL, so whichever ms-comm replica Twilio delivers the media stream to adopts it exactly once. While minting, the pre-warm publishesvd:warming:<callSid>(ms-comm5a0323a): a media stream that lands on another replica ~300 ms after the webhook and finds the marker polls for the credential (bounded by the start budget, stopped by caller hang-up) instead of starting cold — which was refusedcall_sid_in_useand fell back to ElevenLabs. No marker, no wait. - ms-ai endpoints (secret auth):
POST /voice-direct/{tool,heartbeat,webrtc-offer,refresh,setup,usage,session-end,web-analysis,trace}. A sweeper closes sessions whose heartbeats stop (VOICE_DIRECT_SWEEP_GRACE_S). - Limits: credential TTL ≤ 120 s (hard clamp),
VOICE_DIRECT_MAX_CONCURRENT_WEB_PER_COMPANY(10), per-session tool caps (VOICE_DIRECT_TOOL_MAX_PER_SESSION60,_MAX_PER_MIN30,_TIMEOUT_MS30000; ms-comm waits 35 s for a tool). - Tools on direct calls =
ENGINE_HOSTED_TOOLS(voice-engine.tools.ts): KB, products, orders, calendar (check/book/find/reschedule/cancel), the phone tools fromelevenlabs/phone-tool-configs.ts(create_simpro_job,check_simpro_availability,validate_address,send_sms) and — inbound only — caller verification (company_user_authentication,027255494): run in-process byVoiceDirectToolRunnerwith this session's company + agent_key (no HTTP, no secret); the model supplies only username/password, whichVoiceDirectTracescrubs. Prompts advertise exactly these tools (engineHostedTools). - Turn-taking (read per session): Gemini's own detection is off on every
Gemini Live session ms-comm can drive — see Gemini turn-taking;
the Gemini VAD knobs below (
VOICE_DIRECT_GEMINI_START_SENSITIVITY, default low, PRODhigh;_END_SENSITIVITYdefault high;_PREFIX_PADDING_MS300;_SILENCE_MS400;_TEMPERATURE) only apply when manual turns are rolled back or the phone peer isn'tmanual_turns_capable. Grok:VOICE_DIRECT_XAI_SILENCE_DURATION_MS(unset = xAI's tuning),VOICE_DIRECT_XAI_IDLE_TIMEOUT_MS(10000, server check-in on a silent caller;0= off),VOICE_DIRECT_XAI_REASONING(highdefault |none); OpenAI RealtimeVOICE_DIRECT_OPENAI_EAGERNESS(semantic VAD, defaultauto; PRODauto—lowmade callers wait up to ~8 s),_NOISE_REDUCTION(near_field/far_field, default off; PRODnear_fieldsince fluxd7c5e8b9, coughs were cutting the agent off). First-turn VAD (ea0afac4f): the caller's first turn ends on plain server VAD afterVOICE_DIRECT_OPENAI_FIRST_TURN_SILENCE_MS(600,0= off), thensession_update_after_first_turnswitches to semantic VAD — web via the sideband, phone only when ms-comm saysfirst_turn_vad. Realtime only (GPT-Live is full duplex, no turn detection), so it's dormant now the Realtime models are retired. - Greeting guard — stops a "Hello?" over the first words from making the
vendor VAD cancel the greeting. Phone:
ElevenLabsBridgeService._startGreetingGuardholds caller audio from the opener request until the greeting has played out (DIRECT_GREETING_GUARD_LIMITSmaxMs 8000, noAudioMs 3000, tailMs 300). Web OpenAI Realtime: the widget does the same (not armed on GPT-Live — Live never sends the event it waits for). Gemini web relay: mic not forwarded until the opener turn ends (VOICE_DIRECT_OPENER_MIC_HOLD=falsedisables). - Call guard (phone) — ms-comm
DirectCallGuard: end-intent classifier (VOICE_DIRECT_END_INTENT, modelVOICE_DIRECT_END_INTENT_MODEL) plus an idle guard measured from the last activity of either side or the last tool run (so a long lookup isn't a dead line): one check-in afterVOICE_DIRECT_IDLE_CHECKIN_S(40) of silence, hang-upVOICE_DIRECT_IDLE_HANGUP_S(35) after it.VOICE_DIRECT_CALL_GUARD=falseturns it off. - Gemini guards (ms-comm
GeminiLiveAdapter,7463f08→22a93d0; prod 2026-10-16/17: Gemini 3.8 Live web calls went silent after the caller finished while the socket stayed healthy):- Turn commit — 1.2 s after our transcriber ended the caller's turn with
no Gemini commit, send
audioStreamEnd(Google's hybrid-VAD finalize prompt; doesn't provoke a reply to an answered turn).VOICE_DIRECT_GEMINI_COMMIT_MS(1200,0= off). The phone direct leg feeds the transcriber final into the same guard. - Stall nudge — no model output
VOICE_DIRECT_STALL_NUDGE_MS(4000,0= nudge off, log stays) after the turn → one text nudge per turn, max 3 per call. Neither fires while the caller is audibly talking (frame RMS > 200, 800 ms hang). - Backpressure — caller frames are dropped while our socket backlog to Google exceeds ~96 KB (~2 s of audio); a backlog that persists 4 s resumes on a fresh socket.
- Echo gate (web relay) — while the agent is playing (+250 ms tail), mic
frames below
VOICE_DIRECT_ECHO_GATE_RMSare silenced so its own voice leaking into the mic doesn't barge in; real speech still interrupts. Default0(off); PROD600. - Fast recovery —
VOICE_DIRECT_FAST_RECOVERY_MS(default off) nudges a turn our energy detector heard end but Gemini never registered. - Logs Gemini
voiceActivity;STALL/ commit lines carryvad_state,ws_buffered_bytes, frame flow, event-loop lag,gemini_heard,commit_tried.
- Turn commit — 1.2 s after our transcriber ended the caller's turn with
no Gemini commit, send
- External caller transcriber (Gemini) — Gemini's own input transcription
invents words, so ms-comm
CallerTranscriber.jsfeeds the caller audio to an OpenAI realtime transcription session (server VAD) and uses its finals. The model is the one ms-ai names astranscription_modelin the start/refresh answer (since6ed636544, 2026-11-02:gpt-transcribe, $0.0045 per audio minute, billed per minute; rollbackVOICE_DIRECT_GEMINI_CALLER_TRANSCRIBE_MODEL=gpt-4o-transcribeorgpt-4o-mini-transcribe, both per token) and bills under, so they can't drift. Agent and company names go in as transcription hints (51e74c8). On a manual-turn session withVOICE_DIRECT_TURN_DETECTOR=soniox, the Soniox detector supplies the caller transcript instead (see below); without it ms-comm keeps its default (deploy ms-comm first). Lines are stamped with when they were spoken (caller: transcriber VAD speech start; agent: first fragment), emitted in that order withtson the widget message, and held only while an earlier-started caller utterance is still transcribing (max 3 s). A caller transcript with no audio above the silence floor just before it is dropped (invented from silence); Gemini's text is the fallback if it never connects. Runs on the Gemini web relay (12a4423) and Gemini phone calls (8634235, mu-law in); agent echo is filtered and agent lines ordered after the caller line. ms-ai hands the agent language over astranscription_languageon the session/refresh answer. Env:VOICE_DIRECT_CALLER_TRANSCRIBER=false(off),VOICE_DIRECT_CALLER_TRANSCRIBE_MODEL; needs the OpenAI key in ms-comm. Billed — see Billing. - Web call summary (
a1c980b) — at the end of a widget S2S call ms-comm asks ms-aiPOST /voice-direct/web-analysisfor a summary + outcome and fillscall_summary/call_outcomeon the service room (only empty fields); the transcript email carries it. ElevenLabs web calls are untouched. - Keys:
VOICE_DIRECT_<VENDOR>_API_KEYwins over the sharedOPENAI_API_KEY/GEMINI_API_KEY/XAI_API_KEY, so voice spend can sit on its own vendor project.VOICE_DIRECT_WEB_VENDORS(unset =openai,gemini, empty = none) is the no-deploy kill switch for the widget; PROD =openai,gemini,xai. - Debug: ms-comm
VOICE_DIRECT_DEBUG_CAPTURE/ widgetDirectAudioDebugtrace audio quality (DEV).VOICE_DIRECT_TRACE_COMPANIES(*in PROD) writes one JSON trace per direct OpenAI web call (exact session body + every server event, caller-auth args scrubbed) to the recordings bucket undervoice-engine/traces/<company>/<session>.json; read it with secret-authPOST /voice-direct/trace. Holds the full prompt and the caller's words.
Gemini turn-taking (manual turns)
Since ms-ai c8beccb2e (2026-11-05) every Gemini Live model — 3.8 Live and
Extended Thinking, web and phone — runs with Gemini's
automaticActivityDetection disabled; ms-comm GeminiLiveAdapter sends
activityStart / activityEnd itself. Reason: speaker echo at ~2 % made
Gemini's detector cut the agent off mid-sentence, and it cut callers off in
pauses (harness: 3.8 Live 6/10 → 8/10). ms-ai stores manual_turns on the
session snapshot and returns it on mint/refresh; web is always manual through the
relay, phone only when ms-comm's bridge sends manual_turns_capable (an older
ms-comm keeps Gemini's detection). VOICE_DIRECT_GEMINI_MANUAL_TURNS=false
(ms-ai) rolls back to Gemini's detector. ms-comm pieces (51e74c8, 137f2b9,
a335746, 975d963, c87f959):
- Turn controller — opens a turn from our own level detector with the
agent's playback masked (echo-aware), replays ~1.5 s of the caller's masked
first words on barge-in (pre-roll, in order), recovers false barge-ins; barge-in
bar at 40 % of the caller's learned level. Phone tracks agent playback end from
mu-law bytes. Silence check-ins at 15 s then 10 s; promise-stall guard
(
VOICE_DIRECT_GEMINI_PROMISE_MS) when the agent says it will check and goes quiet. - Soniox end-of-turn (
SonioxTurnDetector.js,VOICE_DIRECT_TURN_DETECTOR=sonioxSONIOX_API_KEY; without it, the loudness detector) — streaming transcript +<end>endpoints; a turn opens only on words with near-mic speech; endpoint wait depends on utterance shape (question / statement / trailing word) and grows when the caller resumes. Off-mic filter: a final word counts as the caller's only if the audio around it was ≥25 % of their learned level, so a TV or the room stays out of the transcript and the endpoint text. Runs at 8 kHz on carrier mu-law for phone. Diarization is log-only (VOICE_DIRECT_SONIOX_DIARIZE);VOICE_DIRECT_SONIOX_MODEL/_URL/_MAX_ENDPOINT_DELAY_MS. Soniox usage is a platform cost — nothing bills it to the tenant.
- Near-mic gate (
NearMicGate.js,VOICE_DIRECT_NEAR_MIC_GATE=true) — mutes audio far quieter than the caller before it reaches Gemini / Soniox; adaptive bar (VOICE_DIRECT_NEAR_MIC_GATE_ADAPTIVE=false= fixedVOICE_DIRECT_NEAR_MIC_GATE_RMS,_HOLD_MS). The recording keeps the raw mic. - Wait fillers (
WaitFiller.js) — soft hums in the session voice while a lookup runs (clips underapp/assets/voice-fillers/<voice>, 8 kHz mu-law on phone; the real answer clears them). Off by default —VOICE_DIRECT_WAIT_FILLERS=true, and only for English sessions. ms-ai's/voice-direct/refreshreturns the session voice for them. - Prompt side (ms-ai
d6b2d11d2,c8beccb2e): both Gemini models share one S2S platform prompt — answer the caller mid-lookup, one lookup per question, always the latest request, and always say what you are checking before a lookup, even where tenant instructions ask for very short lookup lines.
PROD (flux): ms-comm VOICE_DIRECT_TURN_DETECTOR: soniox,
VOICE_DIRECT_NEAR_MIC_GATE: "true", VOICE_DIRECT_LIVEKIT: "true"
(LIVEKIT_URL in-cluster livekit-signal-service:7880, LIVEKIT_PUBLIC_URL
wss://livekit.knockknockapp.ai), secrets SONIOX_API_KEY, LIVEKIT_API_KEY,
LIVEKIT_API_SECRET. LiveKit itself is Apps/PROD/livekit/ (f6587fc2):
livekit/livekit-server:v1.9.11, one replica (rooms in memory, no Redis,
fixed UDP port), signal 7880 behind an IngressRoute, RTC TCP 7881 / UDP 7882.
GPT-Live
gpt-live-1 is a different OpenAI product from Realtime: a full-duplex speech
model that delegates every fact, lookup, tool call and booking to a backend
text model (delegation.responses).
- Reasoning model — per agent:
ai_config.call_gpt_live_reasoning_model(widget) /gpt_live_reasoning_modelon inbound/outbound agents;''= env defaultVOICE_DIRECT_LIVE_BACKEND_MODEL(gpt-6-luna). Clamped toLIVE_BACKEND_MODELS(gpt-6-luna,gpt-6-sol,gpt-6.1-sol,gpt-5.6-terra) — a name outside it would bill at zero.gpt-6-astrawas dropped (0da8c0b6f, ~100× Luna's price).VOICE_DIRECT_LIVE_REASONING_EFFORT(defaultlow;nonelifted tolowfor models that refuse it),VOICE_DIRECT_LIVE_MAX_OUTPUT_TOKENS(400). - Two instruction sets —
agent_instructions_gpt_live_speech+agent_instructions_gpt_live_reasoningonai_callingand on each inbound/outbound agent (strict subdocs — declared). BackendgptLiveInstructionsFieldsendsgpt_live_instructions: { speech, reasoning }gpt_live_reasoning_modelonly for a permitted GPT-Live agent; ms-comm does the same for the widget. ms-aiElevenlabsService.getLiveAgentInstructionsassembles both prompts in code (elevenlabs/live-agent-prompts.ts: voice delivery, delegation policy, guardrails, end-call rules; backend capabilities, tool rules, booking/transfer/timezone/company sections). An empty speech half falls back to the agent's S2S/standard prompt; an empty reasoning half = platform blocks only. No LLM on the call path.
- LivePromptSplitService (
direct/live-prompt-split.service.ts, tablevoice_live_prompt_split) — LLM migration of one authored prompt into the two halves (cleanup of ElevenLabs artefacts, restructure, split, audit report), cached by sha256 of the text,SPLITTER_VERSION2, modelVOICE_DIRECT_LIVE_SPLIT_MODEL(gpt-6-sol), timeoutVOICE_DIRECT_LIVE_SPLIT_TIMEOUT_MS(240 s, runs in the background),VOICE_DIRECT_LIVE_SPLIT=falseoff. Only exposed as secret-auth helpers:POST /voice-engine/live-prompt-split(convert a custom prompt) andPOST /voice-engine/live-prompt-preview(the two prompts a call would run on,{{vars}}unrendered). The dashboard's Use default speech/reasoning prompt buttons usefrontend/src/utils/gptLiveDefaultPrompts.jsinstead; switching an agent togpt-live-1with nothing stored seeds both defaults on save (defaultGptLiveOnModelSave). - Widget specifics (
aa00735) — no Realtime greeting request (Live rejects it asunknown_parameter), no greeting guard, "agent speaking" read from Live transcript chunks; the transcript is assembled from Live's deltas (widget/src/services/liveTranscript.ts).
Vapi mode
ms-ai/src/voice-engine/vapi/: vapi-web-call.starter.ts (widget joins over
Daily — widget vapiCall.ts), vapi-phone-call.starter.ts (numbers imported into
Vapi via VapiTwilioNumberApi, gated by VAPI_PHONE_ENABLED; ms-comm
VOICE_S2S_PHONE_MODE=vapi returns Vapi's TwiML from the voice webhook, and
ms-comm POST /twilio/vapi-outbound-started/:company_id records an S2S outbound dial), vapi-assistant.builder.ts
(transient assistant: first message, language, endpointing VAPI_ENDPOINTING,
denoise VAPI_DENOISE, logging VAPI_LOGGING_ENABLED). Webhook POST /voice-vapi/webhook (VAPI_WEBHOOK_TOKEN, ≥ 24 chars). Billing is at cost —
what Vapi charged — polled and reconciled every 5 min (voice-vapi:poll,
voice-vapi:reconcile-costs); calls whose cost can't be trusted are held for
review (GET /api/cost-tracking/admin/vapi/held, POST .../held/:session_id/release).
Admin screen: Cost-Tracking › Vapi. PROD config
(VAPI_ORG_ID, VAPI_PRIVATE_KEY, VAPI_WEBHOOK_TOKEN) is present but PROD runs
direct with VAPI_PHONE_ENABLED=false.
Engine mode (ms-voice-engine)
A separate Python/Pipecat service — no repo in this workspace; deployment only in
flux-deployments/Apps/{DEV,PROD}/ms-voice-engine/. StatefulSet (image
kk-ms-voice-engine-repo-prod, port 7860, HPA 2–6, PDB minAvailable: 1,
terminationGracePeriodSeconds: 900), ingress voice-engine.knock-knockapp.com.
ms-ai creates sessions (VOICE_ENGINE_URL, public VOICE_ENGINE_PUBLIC_URL), the
engine calls back POST /voice-engine/session-callback (host must be in
VE_CALLBACK_ALLOWED_HOSTS), ms-comm's bridge asks POST /voice-engine/telephony/session. Engine config: model allowlists
ALLOWED_GEMINI_MODELS / ALLOWED_OPENAI_MODELS / ALLOWED_XAI_MODELS (an
empty list makes the provider unreachable — OpenAI is held empty since
2026-09-18), Redis session registry on db 3 (REDIS_PASSWORD must be in its
secret or /ready stays 503), MAX_CONCURRENT_SESSIONS 20 / MAX_SESSIONS_PER_POD
12 / WEB_RESERVED_SLOTS 4, DRAIN_TIMEOUT_S 600 (a deploy cuts calls longer than
this), TURN_DETECTION=provider, SMART_TURN_STOP_SECS, VE_PARTIAL_REPORTS
(keep false until ms-ai bills partial reports). Nothing routes here while
VOICE_S2S_MODE=direct.
Instructions
Every voice agent stores two prompts: agent_instructions (ElevenLabs, with
[warmly]-style expression tags) and agent_instructions_s2s (same content,
no tags — S2S models would read tags aloud). The S2S copy is used when the agent's
model is S2S and the company is permitted it (agentInstructionsFor in
backend for inbound/outbound; ms-comm CallsController._webCallSystemPrompt for
the widget); an empty S2S copy falls back to the standard one.
- Behavior tabs show one box at a time: "Agent Instructions for speech-to-speech
(S2S) models" with a Use default S2S prompt button (
frontend/src/utils/s2sDefaultPrompt.js— generic, company name baked in, booking workflow included only if a calendar is connected at the time).s2sInstructions.jsderives a tag-free copy. Onboarding saves the generic S2S prompt for inbound/outbound agents. - Greeting: the same split for the opener —
ai_calling.greeting_message_s2s(widget, read by ms-comm) andopening_script_s2son inbound/outbound agents (backendopeningScriptFor), used under the same rule; empty falls back togreeting_message/opening_script. Behavior tabs show "Greeting Message for speech-to-speech (S2S) models" / "Opening Script for speech-to-speech (S2S) models" (hint: leave out expression tags) in place of the standard field. - GPT-Live agents show two boxes instead of the S2S one — see GPT-Live; the S2S greeting still applies to them.
- Default S2S prompt rules (frontend
105bfaac): every reply is two or three short sentences and must end with a question or a clear next step; a reaction opens a reply, never is the whole reply; emotion cues for impressed / empathetic / clear moments with worked examples. Since9d79f6a1(2026-11-05) a While you look something up section replaces the old "under five words, never explain what you're checking" lookup line (it made Gemini say "let me check" and go silent; 16 of 22 PROD Gemini companies carried it): say in one or two sentences what you're checking and why, answer the caller mid-lookup, never state a result early. Existing saved prompts are not rewritten — the platform prompt overrides the old line instead (next bullet). - S2S platform prompt (ms-ai
elevenlabs/s2s-agent-prompts.ts,a40cc5cb6): Gemini, Grok and legacy Realtime sessions get their own platform layer built from the same workflow sections, every rule once (nolanguage_detectionline, no "filler before the tool" rule, one tool rule with human lookup lines; end-call, thinking and language folded in). Both Gemini models share it, differences in code; Extended Thinking adds "answer first, never promise a lookup" (57b511969); Grok is pinned on it plus Grok delivery rules.VOICE_S2S_PLATFORM_PROMPT=legacyrestores the old assembly; ElevenLabs and GPT-Live prompts are untouched. - Honest AI disclosure (backend
a5ebca159, frontend9d2e906e, 2026-11-12): the default inbound/outbound/playground prompts say "If asked whether you are an AI, say yes honestly" and no longer call the agent a "human representative". Only defaults changed — saved prompts keep their text. The separate per-agent spoken disclosure line (call_disclosure) is in Telephony / Inbound Call Agent. - ms-comm prefixes exactly one
You are <agent>(a prompt that already names[agent_name]is stored as written). ms-ai adds an explicit language-response mandate (voice-prompt-rules.ts) — native-audio models take no language code, so only the prompt can hold the agent language. Since ms-comm7463f08S2S web calls no longer get ms-comm's "switch to whatever you hear" directive (it contradicted ms-ai and let the model drift); ms-ai'slanguageRulesown it, built fromagent_language: the default language for the whole call, a switch only on an explicit request (next sentence already in the new language), otherwise offer once. The ElevenLabs pipeline keeps its directive. - Agent language inherits (backend
de172c75a, ms-comm6fa0900, frontend3ad468dd, 2026-10-27): a blankagent_language/ai_config.call_agent_languagemeans "same as organization". Resolution: own value →company.language→en(backend: outbound dial, inbound sync, Platform API; ms-commcallLanguage(company, sdkEngine)for both web paths: SDK agent → widget voice agent → organization →en, used for the language hint, directive and default greeting). Changingcompany.languagere-deploys inheriting inbound agents. Schema defaults and new agents start blank. The sharedAgentLanguagePicker.vueshowsSame as organization (<language>). One-shot admin migration Call agents follow the organization language (lib/inheritAgentLanguages, in Integrations Maintenance) clears stored languages that repeat what the call already speaks; it never changes a spoken language. Deploy order: backend → migration → ms-comm (or non-English widget voices fall back to the storeden).
Recording
- Policy (
voice-engine-recording-policy.ts), every input fails closed:VOICE_ENGINE_RECORDING_ENABLED=trueAND package allows recording AND agent hasn't opted out AND visitor hasn't refused consent AND the disclosure check passes (VOICE_ENGINE_RECORDING_DISCLOSURE_CHECK=offmust be set explicitly, else the policy isinvalidand records nothing). PROD: enabled, checkoff(product decision 2026-09-16 — disclosure handled like ElevenLabs calls). - Direct calls are recorded by the peer (there is no media hop to tap): ms-comm
DirectCallRecorderon phone and Gemini web relay; the widget (src/services/recording/callRecorder.ts, MP3 via lamejs in a worker) on OpenAI web, uploaded to ms-commPOST /voice-direct/recording. Both land at backendPUT /internal/voice-engine/recordings/:company_id/:session_id(VOICE_ENGINE_RECORDING_API_KEY, same value in backend, ms-comm and engine secrets; ms-comm usesBACKEND_INTERNAL_URL), stored under S3voice-engine/recordings/, maxRECORDING_UPLOAD_MAX_BYTES(64 MiB). - Grok over SIP is the exception: Twilio records the bridged leg
(
record-from-answer-dual) and the recording stays at Twilio (twilio_recording_sid/twilio_account_sidonInboundOutboundCallModel, ms-comm + backend5c164b0a6). Backend playback findstwilio_recording_sidand proxies through ms-commGET /internal/recordings/phone/:sid(API-key gated), which streams it from Twilio with Range forwarded — main-account creds first, the company's own number token on 401/403. - The session id is a UUID stored in the same
conversation_id/elabs_conversation_idfield ElevenLabs uses; playback branches on it (isVoiceEngineSessionIdin backendapp/lib/voiceEngineRecording.js, frontendutils/callRecordingAudio.js) — UUID → backend streams from S3,conv_…→ ms-ai proxies ElevenLabs.
Billing
Direct calls bill the vendor's own usage, per token, the way the vendor prices the
model (VoiceDirectService.bill, since 2026-09-29):
- One
usage_eventvd:<session_id>with oneusage_chargeper unit, agent bucketvoice_web/voice_inbound/voice_outbound. - Caller transcription is a separate event
vd:<session_id>:transcription: OpenAI Realtime's in-session transcription (nowgpt-4o-mini-transcribe, per token —da6c9d3cd; sessions minted earlier onwhisper-1bill per minute) and, since57a081f18, ms-comm's external caller transcriber on Gemini calls (web relay and phone), reported with the session usage and priced undercallerTranscriptionModelFor(model)— for Geminigpt-transcribe($0.0045 per audio minute, billed per minute —PER_MINUTE_TRANSCRIPTION_MODELS, own upsert seeder;6ed636544), or the per-tokengpt-4o-transcribe/gpt-4o-mini-transcribeon rollback; the in-session model for OpenAI. - GPT-Live (
api: 'live'): the voice session is $0.05/min billed per second (live_seconds, cumulative — set, not summed) under thegpt-live-1row; the delegated backend model's tokens are their own eventvd:<session_id>:live_backendunder that model's catalog row (voice-live-backend-models.seeder.ts). A choice outsideLIVE_BACKEND_MODELSis never sent, so it can't bill at zero. request_meta.usage_source:sideband(OpenAI web, server-counted — wins),ms_comm(phone),client(web without a sideband, markedusage_trust=client_reported). A dropped sideband can be beaten by a larger browser report (sideband_partial).- Never billed as zero: with no usage report, the model's provisional
per-minute rate (
voice-direct-per-minute-prices.seeder.ts) is charged, every rowis_estimate=true, logged at ERROR. The boot pre-check errors on an S2S model without aminuterow. cost-tracking/voice-model-billing.tssays per model whichai_providerrow prices it (bills under the model's own provider — a hardcodedgoogleonce billed OpenAI calls at zero), whether cached input is priced separately (OpenAI yes, Gemini no), andtokenvsminutemode.- Grok Voice (
grok-voice-3.0):minutemode at $0.08/min of session,tokensUnpriced— xAI prices no tokens on it, so token counts stay in the event meta and bill nothing (no token price row exists; the oldgrok-voice-think-fast-2.0rows were renamed). On Grok SIP calls the carrier side is billed separately to the telephony wallet by ms-ai's Twilio call-cost sweep: the SIP leg (tosip.voice.x.ai) joins its parent call's wallet activity line, and the call's Twilio recording is debited astelephony_recording("Call recording" in the wallet,twilio-recording:<sid>idempotency key). Both keep sub-cent precision (subCent; backendTelephonyWalletControllernow rounds credits to 4 dp instead of whole cents — a SIP leg costs under ½¢). Every other call's recording stays free. - Credits flow into the normal
token-usagesum and are deducted byAITokensCreditJob, like any AI usage. Vapi mode instead bills at Vapi's cost.
Operational limits
- Live listening and takeover work on direct S2S calls (since
e2cb782/a111cdf). ms-comm stampss2s: true+voice_provider: 'voice-direct'(+voice_vendor) on the call record, service room anduser_session.call_status.- Phone (every vendor on the media bridge, incl. GPT-Live): the bridge fans caller/agent audio and
final transcripts out to
ai-listen:<service_room_id>; takeover ends the vendor session (agent_takeover) and keeps the carrier leg for the human. - Web (OpenAI Realtime, GPT-Live, Gemini, and xAI since
b158b29— both relayed through ms-comm): onrequest_audio_streamthe widget taps the session's own audio graph (mic + agent) and streams 16 kHz PCM to ms-comm; takeover ends the direct session through the normalendAICallpath and reuses the Twilio Video takeover flow. - Still refused (
s2s_not_supported): Vapi calls. Grok over SIP phone calls can't be listened to or taken over at all (the audio never passes through ms-comm), though the frontend'sisUnmonitorableCalldoesn't know that and only excludes Vapi. FrontendisUnmonitorableCall(source, channel)mirrors ms-comm and shows "Live listening and takeover aren't available for this speech-to-speech call." (monitorable: falsefrom the server also wins).
- Phone (every vendor on the media bridge, incl. GPT-Live): the bridge fans caller/agent audio and
final transcripts out to
- The widget agent's ElevenLabs agent-sync cache, background sound and speech settings don't apply.
Env reference
| Where | Vars |
|---|---|
| ms-ai configmap | VOICE_S2S_MODE (PROD direct), VOICE_S2S_WEB_MODE, VOICE_ENGINE_ROLES, VOICE_DIRECT_WEB_VENDORS (PROD openai,gemini,xai), VOICE_DIRECT_GEMINI_MANUAL_TURNS (default on), VOICE_S2S_PLATFORM_PROMPT (legacy = old assembly), VOICE_DIRECT_XAI_IDLE_TIMEOUT_MS / _SILENCE_DURATION_MS / _REASONING, VOICE_DIRECT_WEB_GEMINI_RELAY, VOICE_DIRECT_GEMINI_API_VERSION (default v1beta), VOICE_DIRECT_OPENAI_NOISE_REDUCTION (PROD near_field), VOICE_DIRECT_OPENAI_EAGERNESS (PROD auto), VOICE_DIRECT_OPENAI_FIRST_TURN_SILENCE_MS, VOICE_DIRECT_GEMINI_START_SENSITIVITY (PROD high), VOICE_DIRECT_GEMINI_CALLER_TRANSCRIBE_MODEL, VOICE_DIRECT_LIVE_* (backend model, reasoning effort, max output tokens, split on/model/timeout), VOICE_DIRECT_TRACE_COMPANIES (PROD *), other VOICE_DIRECT_* tuning, VAPI_ORG_ID, VAPI_DENOISE, VAPI_PHONE_ENABLED, VOICE_ENGINE_URL, VOICE_ENGINE_PUBLIC_URL, VOICE_ENGINE_AI_BASE_URL, VOICE_ENGINE_RECORDING_ENABLED, VOICE_ENGINE_RECORDING_DISCLOSURE_CHECK, VOICE_ENGINE_RECORDING_S3_BUCKET / _REGION (recording HEAD checks + traces), VOICE_ENGINE_TELEPHONY_SOFT_BUDGET_MS |
| ms-ai secret | OPENAI_API_KEY, VOICE_DIRECT_GEMINI_API_KEY (PROD since flux acd25916; wins over GEMINI_API_KEY), XAI_API_KEY (PROD since flux d88deed1; VOICE_DIRECT_XAI_API_KEY wins), VOICE_ENGINE_RECORDING_S3_ACCESS_KEY_ID / _SECRET_ACCESS_KEY, VAPI_PRIVATE_KEY, VAPI_WEBHOOK_TOKEN |
| ms-comm | BACKEND_INTERNAL_URL; secret VOICE_ENGINE_RECORDING_API_KEY, OPENAI_API_KEY (or VOICE_DIRECT_OPENAI_API_KEY) — required for GPT-Live phone (LiveAdapter), the Gemini caller transcriber and the end-intent classifier; optional GEMINI_API_KEY (Gemini phone API-key path; PROD has none → token path); VOICE_DIRECT_CALL_GUARD, VOICE_DIRECT_END_INTENT*, VOICE_DIRECT_IDLE_CHECKIN_S / _HANGUP_S, VOICE_DIRECT_CALLER_TRANSCRIBER*, VOICE_DIRECT_OPENER_MIC_HOLD, VOICE_DIRECT_ECHO_GATE_RMS (PROD 600), VOICE_DIRECT_GEMINI_COMMIT_MS, VOICE_DIRECT_STALL_NUDGE_MS, VOICE_DIRECT_FAST_RECOVERY_MS, VOICE_DIRECT_GEMINI_PROMISE_MS, VOICE_DIRECT_GEMINI_TOOL_SCHEDULING; turn-taking VOICE_DIRECT_TURN_DETECTOR (PROD soniox), VOICE_DIRECT_SONIOX_*, VOICE_DIRECT_NEAR_MIC_GATE (PROD true) / _ADAPTIVE / _RMS / _HOLD_MS, VOICE_DIRECT_TURN_OPEN_RMS, VOICE_DIRECT_WAIT_FILLERS (default off), VOICE_DIRECT_FILLER_DIR; LiveKit VOICE_DIRECT_LIVEKIT (PROD true), LIVEKIT_URL, LIVEKIT_PUBLIC_URL; Grok XAI_SIP_ENABLED (PROD true), XAI_SIP_LOG_TRANSCRIPT; secrets SONIOX_API_KEY, LIVEKIT_API_KEY / _SECRET, XAI_API_KEY; VOICE_ENGINE_TELEPHONY_ENABLED / _COMPANIES (engine pilot, default off), VOICE_S2S_PHONE_MODE |
| backend | secret VOICE_ENGINE_RECORDING_API_KEY; MS_COMMUNICATION_URL + MS_COMM_API_KEY (recording proxy) |
| livekit | Apps/PROD/livekit/ (PROD only) — server config in its ConfigMap (port 7880, RTC TCP 7881 / UDP 7882), API key/secret in its secret (same pair as ms-comm's LIVEKIT_API_*) |
| ms-voice-engine | configmap ms-voice-engine-cm (allowlists, Redis, capacity, drain, turn detection); secret GOOGLE_API_KEY, OPENAI_API_KEY, XAI_API_KEY, ALLOWED_XAI_MODELS, INTERNAL_SERVICE_TOKEN, VE_CALLBACK_AUTH_TOKEN, VOICE_ENGINE_RECORDING_API_KEY, REDIS_PASSWORD |
Related
Call Agent · Inbound Call Agent · Outbound Call Agent · Telephony · Cost-Tracking · ms-ai · ms-communication · widget · Deployment