Telephony
Phone numbers and voice transport for the AI call agents, split across three
services: ms-ai owns numbers, purchase, and rental billing
(src/elevenlabs/); ms-communication owns the
live audio bridges and post-call persistence; backend
owns the wallet ledger and the webhook fallback cron.
Carriers
| Carrier | Numbers | Audio path | Credentials |
|---|---|---|---|
| Twilio | in-app purchase (per-company subaccounts) + BYO import | TwiML <Connect><Stream> → /twilio/media-stream WS → ElevenLabs | ms-ai master TWILIO_MASTER_ACCOUNT_SID/_AUTH_TOKEN creates subaccounts; BYO = per-number SID + token |
| Telnyx | BYO import only | TeXML <Connect><Stream> → /telnyx/media-stream WS → same bridge | per-number telnyx_api_key + telnyx_connection_id (TeXML app), stored on the phone_number owner row |
- Carrier is per number:
elabs_phone_number.provider(twilio/telnyx) in ms-ai selects the dial path;inbound_outbound_calls.providerin ms-comm records it. ElevenLabs has no native Telnyx support — ms-ai synthesizesnumber_id: telnyx_<uuid>and the ms-comm bridge carries the audio. - ms-comm Telnyx surface:
TelnyxInboundCallController/TelnyxOutboundCallController+TelnyxHelper(TeXML builders). Routes:POST /telnyx/inbound-voice,/telnyx/inbound-status(neutral, current), legacy/telnyx/inbound-voice/:company_id+/telnyx/inbound-status/:company_id,/telnyx/outbound-voice/:company_id,/telnyx/outbound-status/:company_id,/telnyx/preheat-outbound/:company_id/:agent_id. - Telnyx inbound routes by the dialled number (ms-comm
f72871b, ms-aif0dd8ba94). A TeXML Application has onevoice_urlfor all its numbers and customers reuse one Application across companies, so importing used to rewrite it with the importer's company and every number rang the last importer's agent. Import now writes the neutral URLs; on a call ms-comm asks ms-aiGET /elevenlabs/telnyx-inbound-route?number=<To>(secret auth) →{ company_id, agent_id, agent_key, webhook_url, company_phone }. An outbound-only number routes to the company's latest inbound Telnyx binding on the same Application, else its default inbound agent. The legacy URL's company is a fallback only for numbers ms-ai doesn't know (or a missing agent on the same company) — never another tenant's per-URL values. The owner is cached per call in Redistelnyx-inbound-company:<CallSid>(6h) for the neutral status callback; the prewarm budget shrinks by time already spent so the webhook stays under Telnyx's ~15s. Deploy ms-comm before ms-ai. Env:TELNYX_MEDIA_STREAM_PATH(default/telnyx/media-stream),TELNYX_PUBLIC_WS_URL(else derived fromELEVENLABS_PUBLIC_WS_URL). - Codecs: Twilio is always PCMU (mu-law) 8k. Telnyx inbound legs are PCMU, outbound/international PCMA (A-law) — TeXML declares
bidirectionalCodecper direction;AudioTranscodeHelper(A-law support) normalizes to mu-law on ingest so the rest of the bridge is unchanged. - Live monitor/takeover is carrier-agnostic, keyed by
service_room_id: both bridges fan audio + transcripts intoai-listen:<service_room_id>, served by the existing socket handlers — no Telnyx-specific sockets.
Phone number ownership (ms-ai phone_number)
Since 8cee58357 (migration 1794197775714-central-phone-number) every number a company or agency holds has one phone_number row — owner, carrier credentials, billing and suspension — and each agent's use of it is an elabs_phone_number binding pointing at it (phone_number_id, eager ManyToOne; a QueryBuilder must join it itself).
- Columns:
number(E.164),provider, exactly one ofcompany_id/agency_id(check constraint),source∈app_purchase | byo | agency_purchase | agency_import,twilio_sid,twilio_auth_token+telnyx_api_key(encrypted at rest viaencryptedText),twilio_number_sid,telnyx_connection_id,monthly_price_usd,country,number_type,purchased_at,next_rental_at,suspended_at,release_at,warnings_sent,released_at(history, never matched again). Unique per company+number and agency+number. - Why: a number attached to a second agent got a binding with no source/price/rental date, so it read as BYO — outbound calls and texts skipped the wallet gate, removing the purchased binding stopped the rent, and a re-import wiped billing. Rent, suspension and cost sweeps, the wallet gates, removal, trust checks and the list endpoints now all read the owner. Helpers in
src/elevenlabs/owned-numbers.ts(ensureOwneronly ever upgrades a source toapp_purchase,credsOf,ownershipFields,forgetIfUnused). - Disconnect ≠ release. Dropping the last binding of a bought number keeps the owner (still billed); a BYO owner is forgotten once nothing uses it.
POST /elevenlabs/remove-phone-number/from-accountdrops every binding;release_from_carrier: truereleases a bought number at Twilio,falseis a plain disconnect, omitted → 409requires_release_confirmation. A customer's own number is never touched at the carrier. GET /elevenlabs/phone-numbers/:company_idreturns one row per binding with ownership read from the owner, plusagency_id,a2p_applies(isUs10dlc),suspended/release_at.include_unused=trueadds owned numbers with no binding (connected: false, used by billing Add-ons);include_twilio=truealso adds numbers found in the subaccount with no record (the central Phone Numbers screen).- Credentials never leave ms-ai (
a841c02d1): all 11 buy/import/attach/assign endpoints returnwithoutPhoneCredentials(row)(phone-number-public.ts) — before, the response carried the subaccount token, letting a tenant drive the subaccount outside the wallet. Stripped at the controller, never in the service (the buy path re-saves the row the import returns). - Reconcile (
internal/reconcile-phone-numbers, one page of subaccounts per call, dry run unlessapply): finds numbers rented in a subaccount but never billed and records them as bought, billing from now (no back-charges); page one also links bindings an older pod wrote unlinked. Super-admin wrapper: backendPOST /admin/maintenance/reconcile-phone-numbers(background job loopingnext_page), card Reconcile rented phone numbers in Admin → Integrations maintenance. - Agency-pool rows are not migrated yet (
agent_key: 'agency-pool'; rent idempotencyagency-twilio-rental:<poolRowId>:<period>).
Number lifecycle (ms-ai)
Buy (Twilio only). GET /elevenlabs/available-phone-numbers/twilio → POST /elevenlabs/buy-phone-number/twilio/{inbound|outbound|sms} (+ cancel-phone-number/twilio). The UI offers US/CA/GB/AU × local/toll-free/mobile (the server accepts any ISO country, default US). The number is purchased into the company's Twilio subaccount (findOrCreateTwilioSubaccount, FriendlyName knock-sub:<company_id>; subaccount SID + token live on the number's meta). The buy is gated on the wallet covering the marked-up price (markup_percent from the wallet response; 402 when short) and the first month is charged at purchase (meta.next_rental_at starts the clock).
Billing crons — NumberRentalScheduler (src/elevenlabs/number-rental.scheduler.ts; kill switch NUMBER_RENTAL_CRON_ENABLED=false; app-purchased numbers only, BYO is never touched):
| Cron name | Schedule | Does |
|---|---|---|
number-rental:meter-due | daily 06:00 UTC | monthly rental debit, idempotent on twilio-rental:<number>:<period>; catches up missed months; frozen while suspended |
twilio-call-cost:meter | every 10 min | sweeps each subaccount's completed calls (48h lookback), debits Twilio's settled per-call price, idempotent on twilio-call:<CallSid> |
twilio-sms-cost:meter | every 10 min | twin of the call sweep for sent SMS (meterTwilioSmsCosts()), debits Twilio's settled per-message price, idempotent per Message SID |
number-rental:suspension-sweep | every 15 min | suspend / warn / release lifecycle below |
Suspension (state on the phone_number owner: suspended_at, release_at, warnings_sent): wallet < 0 → suspend, release_at = +7d, day-0 warning; day-3/day-6 warning emails; past release_at → release from Twilio; wallet back > 0 → auto-unsuspend. Emails route through backend POST /internal/telephony/notify (suspend_warning / released → subscription/number-suspended.html / number-released.html).
Agency pause. Before anything else, ms-comm's inbound handler checks company.paused_at: a paused client's calls go to a human (first routing-priority number → transfer numbers → company.phone_number) or get the "temporarily unavailable" message — the AI never answers or bills. Fails open.
Inbound block. ms-comm calls GET /elevenlabs/telephony-inbound-allowed/:company_id?number= (4s timeout, fail-open) before answering; blocked callers hear "This number is temporarily unavailable. Goodbye." Only meta.source === 'app_purchase' numbers are gated.
Ops: POST /elevenlabs/internal/meter-rentals, POST /elevenlabs/internal/sweep-suspensions, GET /elevenlabs/internal/number-detail.
Verification & registration (Twilio Trust Hub, Regulatory, A2P)
Three carrier review flows, all in ms-ai src/elevenlabs/, all advanced by TwilioTrustScheduler (leader-locked; kill switch TWILIO_TRUST_CRON_ENABLED=false): twilio-trust:sync every 30 min (Trust Hub + A2P), twilio-regulatory:sync at :15/:45. Each sync is idempotent and also runs on submit, on purchase (onNumberPurchased, not awaited) and on a stale status read.
1. Business verification — Trust Hub (twilio-trust.service.ts, table twilio_trust_registration). Fixes "Spam Likely" on outbound caller ID: KnockKnock's Primary Customer Profile (master account; TWILIO_PRIMARY_CUSTOMER_PROFILE_SID or auto-discovered) → a Secondary Customer Profile per company in its knock-sub: subaccount → trust products SHAKEN/STIR, Voice Integrity, CNAM (optional; not toll-free), created only once the profile is approved. Sync assigns every +1 subaccount number to the profile and each product, so numbers bought before or after are covered.
- Routes:
GET /elevenlabs/trust-registration/:company_id,POST /elevenlabs/trust-registration(submit / resubmit; an approved profile is never resubmitted — only rejected products are rebuilt). - Outbound buy gate: buying a US outbound number throws 428
business_registration_requiredunless a non-rejected profile exists (TWILIO_TRUST_REQUIRED_FOR_OUTBOUND, default true); the buy modal shows the form, andskip_trust_registration("Skip for now, do it later") buys anyway. - Rejection reasons (
464c4d9a7,twilio-rejection-reasons.ts): Twilio review rejections carry only an error code; codes 186xx etc. map to{ title, fix, fields[] }, stored asissueson the registration and returned to the form, which marks the fields. Migration1793900000000-twilio-trust-rejection-issues. One-shot backfill: ms-aiPOST /elevenlabs/internal/trust-registrations/refresh-rejections(reads Twilio only) wrapped by backendPOST /admin/maintenance/trust-refresh-rejections(admin card Refresh business verification rejections).
2. Number verification — Regulatory Compliance (twilio-regulatory.service.ts / twilio-regulatory.controller.ts, table twilio_regulatory_bundle). GB, AU and NZ won't sell a number without an approved bundle per country + number type (US/CA need none; SUPPORTED_COUNTRIES = US CA GB AU NZ). The form is generated from Twilio's live Regulation, not hard-coded; uploads stream to Twilio, never stored. A type chosen for several requirements is sent as one document with all their fields (Twilio 70002 refuses two documents of one type). Buying without approval → regulatory_bundle_required / regulatory_in_review.
- Routes (
company_idis a path param so the tenant guard runs before multipart parsing):GET /elevenlabs/regulatory/:company_id?country&type,GET /elevenlabs/regulatory/:company_id/statuses(every started verification, for the status cards),POST /elevenlabs/regulatory/:company_id(multipart, ≤10 files), opsPOST /elevenlabs/internal/regulatory/sync.
3. US texting — A2P 10DLC (d75d94198; state in the a2p jsonb column, migration 1793774257012-twilio-trust-a2p). Built on the approved Secondary Profile: A2P Messaging Profile trust product (TWILIO_A2P_POLICY_SID override) → Brand (TCR; low-volume brands skip secondary vetting) → Messaging Service (keeps each number's own inbound webhook) → Usa2p campaign → every US long code in the subaccount added to the service. One step per sync.
- Route:
POST /elevenlabs/trust-registration/sms(submit / fix-and-resubmit; an approved brand is never re-vetted). Public statusnone | in_review | approved | action_requiredon thesmskey of the trust-registration GET, with per-stepprofile/brand/campaignstatuses,error,fees, andsms_required. - Fees debit the company wallet, source
telephony_sms_registration(wallet label "Texting registration"), idempotent pera2p-brand:<sid>/a2p-campaign:<sid>/a2p-campaign-month:<sid>:<YYYY-MM>. Raw defaults (USD) brand 4.5, secondary vetting 41.5, campaign vetting 15, monthly 10 (1.5 forLOW_VOLUME); overrideTWILIO_A2P_FEES_USD(JSON). The backend debit adds markup; submit is refused 402insufficient_balancewhen the wallet can't cover the marked-up total. - Send block (
assertCanSendSms, called bysend-sms): a company-bought US 10DLC number (isUs10dlcinus-10dlc.ts: +1, not toll-free, not a Canadian area code) can't text until the campaign isVERIFIED→ 409sms_registration_requiredwith a human reason; backendAiSmsWebhookControllerlogs it as afailedrow. Agency-pool and BYO numbers are never blocked (the UI warns). Kill switchSMS_A2P_REQUIRED=false.
Notifications. ms-ai _notifyTelephony → backend POST /internal/telephony/notify (TelephonyWalletController) with type sms_a2p (blocked at most once per 24h per company, approved, rejected), business_verification or number_verification (approved / rejected, only on a status change). Each sends an in-app system notification (relations.kind/event/path; push except on approval) and an email to the billing owner via subscription/sms-registration.html. The link goes to the first agent Phone Numbers tab the company has (_phoneNumbersPath), not the central screen.
Frontend. Central screen views/settings/ai/PhoneNumbers.vue (route /settings/ai/phone-numbers, menu Settings → AI Agents → Phone Numbers): one row per number grouped on digits (components/telephony/phoneNumbersOverview.js — any binding naming app_purchase/agency wins over a sourceless extra attachment), used-by links, spam-label and A2P columns, price/next charge, suspension; above it BusinessVerificationCard, SmsRegistrationCard, RegulatoryStatusCard (a rejected bundle is fixed in a modal; an approved one deep-links to an agent's buy popup with ?buy=CC:type). The same cards sit on the inbound/outbound/SMS agents' Phone Numbers tabs; SmsRegistrationCard falls back to the business form while the business is unsubmitted or rejected.
Carrier webhook & token auth (ms-comm 411462b)
- Twilio webhooks (
/twilio/inbound-*,/twilio/outbound-*,/twilio/amd-status,/twilio/inbound-screen,/twilio/inbound-dial-fallback,/twilio/call,/service-room/twilio-callback) sit behindapp.middlewares.twilioSignature:X-Twilio-Signaturemust verify against the env tokens (CALL_TWILIO_ACCOUNT_AUTH,TWILIO_AUTH_TOKEN) or the company's own account token for theTo/Fromnumber (fetched from ms-ai, cached per pod) — 403 otherwise. Emergency off:TWILIO_VALIDATE_SIGNATURE=false. - Telnyx webhooks check the Ed25519
telnyx-signature-ed25519overtimestamp|raw body(±300s) againstTELNYX_PUBLIC_KEY(base64, comma-separated). Unset = accepted unverified (BYO accounts sign with their own keys) — logs a CRITICAL once in production. Prod sets neitherTWILIO_AUTH_TOKENnorTELNYX_PUBLIC_KEYin flux. - Media streams: the
<Stream>carrieskk_exp+kk_sig(HMAC-SHA256 ofcompany_id|agent_id|exp, key derived fromWEBHOOK_API_KEY, 2h TTL) or a valid Twilio signature on the WS upgrade (MediaStreamAuthHelper). A stream'swebhook_urlis honoured only on our own hosts. - Toll fraud — widget "forward to phone":
POST /twilio/tokenmints an identity ofvisitor_<user_session_id>only when that session belongs to the given company (403 otherwise), elsevisitor_anon_<random>; the widget sends itsuser_session_id(widget5711ddd). The TwiML App URL/twilio/calldials only a company's own publishedphone_numberwithallow_twilio_callsnot false (a session-bound identity only its own company's), never an arbitraryTo, and checksApplicationSid.
Telephony wallet (backend)
TelephonyWalletController (routes in app/routes/internalTelephonyRoutes.js, guarded by apiKeyAuth = BACKEND_API_KEY). The wallet is the subscription's remaining_purchased_ai_credits (1 credit = $0.01); plan remaining_ai_credits is AI-only and never pays telephony.
| Route | Purpose |
|---|---|
GET /internal/telephony/wallet/:company_id | wallet_credits, plan_credits, has_telephony (wallet > 0), ai_available (plan > 0 || wallet > 0), markup_percent |
POST /internal/telephony/debit | idempotent on idempotency_key (unique in ai_credit_logs); charges base × (1 + markup) at 4-dp precision; balance may go negative (paid down by the next top-up) |
POST /internal/telephony/notify | suspension/release emails to the resolved billing user |
Markup: global setting telephony_markup_percent (default 0), applied on every debit (margin recorded on the ledger row) and returned so ms-ai gates the buy and the frontend shows final prices. Wallet UI: frontend/src/views/packages/Wallet.vue; buy UI in the agents' Phone Numbers tabs (markup via useTelephonyMarkup.js).
Transports
Widget AI calls — WebRTC (LiveKit). ms-comm hardcodes calling_type: 'webrtc' on POST {ms-ai}/elevenlabs/start-audio-call (CallsController); ms-ai returns a LiveKit conversation token (the websocket path still returns signedUrl) and the widget starts the ElevenLabs SDK with { conversationToken, connectionType: 'webrtc' } (widget/src/services/aiCall.ts). Feature parity runs on SDK callbacks instead of the raw socket: transcripts, typed user messages, product cards (events-gateway), booking confirmation (delivered as a userMsg — the only signal the agent gets on LiveKit), and live-listen taps the agent's LiveKit audio track into the audio_chunk stream. Token wait is 30s (was 15s); a ringback loop (on-call-ringing.mp3, started inside the click gesture in widget/src/stores/widget.ts) covers the connect gap.
PSTN. Dual-WebSocket bridge (ElevenLabsBridgeService): carrier leg (mu-law/A-law 8k) ↔ ElevenLabs Convai (PCM 16k), transcoding both ways, transcript + audio fan-out to ai-listen:<service_room_id>.
Speech-to-speech (S2S). When the agent's model is a native S2S model the company may use (GPT Live 1 and Gemini 3.8 Live are open to every company), ms-ai stamps voice_provider on the TwiML URL and the same bridge connects the carrier leg straight to the vendor — app/services/voiceDirect/ (GeminiLiveAdapter with a credential ms-ai mints; LiveAdapter for GPT-Live, which opens the Live API with ms-comm's own OpenAI key and session.start from ms-ai /voice-direct/setup; RealtimeAdapter for the retired Realtime models / xAI). The vendor session is pre-warmed while the phone rings (Redis vd:warm:<callSid>, adopted by whichever replica gets the media stream). Audio + final transcripts do fan out to ai-listen:<service_room_id>, so monitor and takeover work for every vendor (takeover ends the vendor session, keeps the carrier leg). The bridge also runs the greeting guard (caller audio held until the opener has played), DirectCallGuard (end-intent + idle check-in/hang-up) and, on Gemini, the external OpenAI caller transcriber. Inbound direct calls get the caller-verification tool when the agent requires it. ms-comm records the call itself. A declined or failed S2S session falls back to the agent's ElevenLabs standby on the same call. Widget S2S calls bypass the ElevenLabs LiveKit path (WebRTC to OpenAI; Gemini over ms-comm's own LiveKit room or WS relay; xAI through the relay). Grok phone calls (XAI_SIP_ENABLED=true) skip the bridge entirely: the voice webhook answers with a <Dial> to sip.voice.x.ai (Twilio records the bridged leg), so there is no live listen/takeover, and the SIP leg (joined to its parent call's activity line) plus the Twilio recording (telephony_recording, "Call recording") are debited to the telephony wallet at sub-cent precision — see Voice Engine › Recording. Full detail: Voice Engine.
Number lookups (ms-ai bb71ab594, 3be179c8c). elabs_phone_number.number holds whatever spelling the UI or an import wrote (+1 938 388 5440, (938) 388-5440), so every lookup keyed by a number arriving from outside (carrier webhook, called/dialled number) uses PhoneNumberEquals (src/elevenlabs/phone-number-match.ts, digits-only regexp_replace compare; exact match when the input has no digits). Exact Equal refused S2S inbound with "no phone record" and let the telephony wallet check fall open on an unknown number. One number may carry an inbound and an outbound row (each with its own agent_key): inbound call context prefers meta.purpose: 'inbound', falling back to any row for numbers that predate purpose — an arbitrary pick used to resolve the outbound key and decline no_agent_config.
Agent-sync skip cache (web only). start-audio-call used to re-sync the ElevenLabs agent on every connect (~6.2s). ms-ai now skips the sync when nothing changed: per-replica in-memory cache keyed on a sha256 of the agent config, TTL AGENT_SYNC_CACHE_TTL_MS (default 5 min), invalidated via invalidateAgentSyncCache() on every mutating path (elevenlabs.service.ts). Telephony agents are deliberately not cached — they have preheat.
WebRTC parity fixes: service-room-update-data (stores elabs_conversation_id on the service room — the key for recordings and the transcript email) now fires on both transports, not just websocket; the widget emits ai-call-client-log → ms-comm CallsController persists client-side call diagnostics (event, transport, handshake UA) into the logs collection; transcript-email timestamps render in the company timezone.
Inbound routing priorities
Per inbound agent routing_priorities: [{ number, ring_timeout }] — ring real phones in order before the AI. Sanitized in backend AiInboundAgentController._sanitizeRoutingPriorities (max 10 numbers, timeout clamped 5–120s, default 20s); UI InboundCallAgent/CallForwarding.vue. ms-comm dials the chain with <Dial timeout=… answerOnBridge>, advancing via POST /twilio/inbound-dial-fallback/:company_id. Empty list = straight to AI; applies regardless of working hours; any error fails open to the AI. The wallet gate runs before the chain — forwarding legs cost carrier minutes too.
Inbound call screening (Twilio only)
Optional <Gather> keypress gate that runs before routing priorities, so auto-dialers never reach the AI (or a forwarded human):
- ms-comm
POST /twilio/inbound-screen/:company_idsits atophandleInboundVoice. Stateless — the expected digit and agent ids ride the Gather action URL (?d=…etc.), no session store. Any error fails open to the normal flow. - Config: per-inbound-agent
agent.call_screening(sanitized in backendAiInboundAgentController._sanitizeCallScreening, normalized again in ms-comm_normalizeCallScreening). - Failed screens land in
ScreenedCallModel(screened_calls: uniquecall_sidupsert so webhook retries don't double-log;reason: no_input | wrong_digit; TTL index 90d). Deliberately NOTinbound_outbound_calls— a screened call has no transcript/duration and must not pollute call lists, Knox stats, briefs, or GHL sync. Screening never blocks a number; each call gets a fresh prompt. - Latency: preheat starts during the Gather (fire-and-forget); on screened re-entry (
?screened=1) the prewarm budget is 3s (vs 11s) and nothing runs before<Connect>. - No Telnyx equivalent. Read route for the dashboard log: backend
GET /ai-inbound-agents/:agent_key/screened-calls.
Voicemail detection (async AMD, outbound)
Every outbound dial now carries Twilio/Telnyx async answering-machine detection:
- ms-ai
_appendAmdParams(elevenlabs.service.ts) addsMachineDetection=Enable+AsyncAmd=true+ status callback — the call connects immediately, the AMD verdict arrives async. - Callbacks: ms-comm
POST /twilio/amd-status/:company_id(OutboundCallController.handleAmdStatus) andPOST /telnyx/amd-status/:company_id(TelnyxOutboundCallController) — both always answer 204. Telnyx verdicts match the bridge registry by bothCallSidandCallSidLegacy(Telnyx doesn't document which equals the TeXML sid). ElevenLabsBridgeService.handleMachineDetectedacts only onmachine*/faxverdicts;human/unknownare no-ops. The record's voicemail flag is written with$max— sticky true, a latehumancan't un-flag it.- Env (ms-ai):
AMD_ENABLED(default true),AMD_TIMEOUT_SECONDS(20),AMD_SPEECH_THRESHOLD_MS(3000). If the carrier rejects the AMD params the dial retries once without AMD. Log tag[voicemailAMD].
Cross-call memory (continuation first message)
Returning callers get a personalized opener instead of the stock greeting:
- ms-ai
POST /agents/generate-first-message— public, throttled (5/s, 60/min), backed byFirstMessageAgent(modules/agents/definitions/first-message.agent.ts). Always returns 200 with the default greeting on any failure; never emits{{…}}template tokens. - ms-comm
helpers/FirstMessageHelper.jswraps the call (20s timeout — the LLM round-trip is the work, not overhead; callers bound total wait themselves). - PSTN:
ElevenLabsBridgeService._loadPriorCallTranscriptkeys oncaller_number(index{ caller_number: 1, started_at: -1 }), requires at least one caller turn, takes the last 30 turns._getContinuationFirstMessageCacheddedupes so preheat + call start share one Mongo read/LLM call. Total budget on the connect path:FIRST_MESSAGE_INIT_CAP_MS(2000) — over budget falls back to the configured greeting. Gate: per-inbound-agentcontinue_previous_conversation. - Web: gate
ai_config.call_continue_previous_conversation(widget opt-in). The generated opener rides thejoining-ai-callpayload as a per-sessionfirst_messageoverride — never baked into the shared ElevenLabs agent. An accepted auto-trigger greeting wins over the continuation opener.
Caller authentication (inbound, 2026-09)
Inbound phone agents can verify a caller by username + password before saying
anything account-specific. Three AND gates
(backend/app/lib/aiAgentAuthentication.js isAuthenticationEffective):
company.ai_agent_authentication_allowed (super-admin, default false) &&
company.ai_agent_authentication.enabled (tenant) &&
agent.authentication_required (per inbound agent, declared on
ai_inbound_call — keep it declared or the strict subdoc strips it).
- Users:
ai_agent_authentication.users(Mixed,select:false, max 500),max_attempts(1–5, default 3),failure_action(end_call | continue_without_access). Passwords are scrypt (scrypt$<salt>$<hash>, N=16384/r=8/p=1, keylen 64, salt per user) over the normalized spoken form (lowercase, whitespace and punctuation stripped — "Hunter 2" stores as "hunter2");timingSafeEqual; an unknown username still verifies against a decoy hash so timing can't enumerate users. Rate limits inAiAgentAuthenticationService: per-conversation =max_attempts, per-company 120/60s, per-caller-number. - Routes: tenant
GET|PUT /ai-agent-authentication,POST …/users,PUT|DELETE …/users/:id(JWT); super-adminPUT /company/:_id/ai-agent-authentication-allowed; internal (apiKeyAuth)POST /internal/ai-agent/authenticateandGET /internal/ai-agent/authentication-status/:company_id— both always 200. Bothai_agent_authentication*keys are innoxConfigFieldMap.jsFORBIDDEN_PREFIXES, so Nox can never write them. - ms-ai tool
company_user_authentication(hosted namecompanyUserAuthentication, webhookelevenlabs/tools/company-user-authentication, header secret from workspace secretCOMPANY_AUTH_TOOL_AUTH— ElevenLabs drops plain-string headers).company-auth-tool.service.tsresolves the tenant fromsystem__agent_idviaelabs_agent; thecompany_iddynamic var is a cross-check only (company_mismatchrefuses); non-inbound-telephonyrefused. Forwards{company_id, agent_key, username, password, conversation_id, call_sid, caller_number}and returns the backend body verbatim:{authenticated: true, instruction}or{authenticated: false, message, attempts_remaining, locked, instruction}/verification_unavailable/not_enabled. Success reveals nothing about the user. - Direct S2S inbound (
027255494): the same tool is registered on direct (GPT-Live / Gemini) inbound calls when the hosted-tool list includescompanyUserAuthentication, butVoiceDirectToolRunneranswers it in-process with the session's own company +agent_key(no webhook, nosystem__vars, no secret); the model supplies only username/password, andVoiceDirectTracescrubs them from debug traces. - Prompt placement (
elevenlabs.service.ts~1494–1524, hardened ind891d7471/bb2c9098c/d92567856): the full# Caller Verificationblock goes first (right after the company prompt) and a short# Before you say anything elsereminder goes last, after the caller context — primacy and recency; middle-only and end-only both failed in testing. Both sit after the company prompt so the per-call refresh strip regenerates rather than accumulates. Attached only forinbound-telephonywhenportalToolsincludescompanyUserAuthentication. - ms-comm:
ElevenLabsBridgeService._isAuthRequired()re-derives the three-gate rule from a projected lean read (30s cache, fails closed) and passes?auth_required=1on the tools refresh. Live redaction:_notePasswordPromptarms on an agent turn matching/\bpassword\b/i, and_redactSpokenPasswordreplaces the next caller turn with[password redacted]before logging, socket fan-out, and transcript persistence. Post-call, ms-airedact-auth-transcript.tsruns at the top ofpostCallWebhook— beforewebhook_inboxstorage and before forwarding — scrubbing the tool'sparams_as_json/tool_detailsand the nearest preceding user turn (andoriginal_message). - Never persisted: the plaintext password anywhere;
usersisselect:falseand never returned to the dashboard; audit lines carry masked usernames (j***); failure messages never say which field was wrong. preserveInboundAgentFields(aiAgentAuthentication.js, used inCompanyControllerupdate): the default agent saves as one wholeai_inbound_callobject and Mongo$setreplaces the subdoc, so a browser tab holding a pre-toggle copy silently erasedauthentication_required(andwebhook_secret) by saving a new greeting. Server-owned keys are now carried forward when omitted, thenauthentication_requiredis forced false unlessai_agent_authentication_allowed. This is a different failure mode from the strict-subdoc drop — declaring the field doesn't fix it.
Agent memory gating
Per-agent use_memory (absent = true) and continue_previous_conversation
(absent = false) resolve through ms-comm
AgentMemoryFlagHelper.resolveAgentFlags(companyId, {direction, agentKey}) —
one cached (30s) lean read; use_memory fails open, continuation fails
closed, and continuation is ANDed with use_memory. Consumed in
ElevenLabsBridgeService (456970e): gates _getVisitorContextByPhoneCached
and the continuation opener.
Widget voice uses ai_config.call_use_memory, resolved by
CallsController._isWidgetVoiceMemoryEnabled as a deliberate .lean() raw
read — ms-comm's CompanyModel is a mirror, and a hydrated read would drop
the path and flip memory back on. Off also stamps _stampRoomMemoryOptOut
(room excluded from fact extraction and identity-graph past_conversations)
and skips _loadPriorTranscript. In ms-ai, use_memory is part of the
agent-config hash so a toggle re-syncs the prompt, and memory-off withholds
upsert_memory / delete_memory_slot and removes them from the
instructions (memory-tool-gating.spec.ts). Outbound (telephony) agents are
inert — null user_session_id, no {{visitor_context}}, useMemory
hardcoded true.
Custom LLM for ElevenLabs agents
ms-ai/src/elevenlabs/call-models.ts is the single source of truth for the AI
Model selector. NATIVE ids write prompt.llm = <id>; CUSTOM ids
(Baseten-hosted: baseten-glm-5.3-flash → zai-org/GLM-5.3-Flash, added 2026-09-25 when Baseten deprecated GLM 4.7; baseten-glm-4.7 still listed; both use the BASETEN_GLM47 workspace secret, which holds the account key) write
prompt.llm = 'custom-llm' plus a prompt.customLlm block built by
custom-llm.service.ts pointing at BASETEN_INFERENCE_URL
(https://inference.baseten.co/v1; ElevenLabs appends /chat/completions).
Persisted as ai_config.call_model / ai_inbound_call_agents[].model /
ai_outbound_call_agents[].model. Env: ELEVENLABS_CUSTOM_LLM_PROVIDER
(default baseten), ELEVENLABS_CUSTOM_LLM_MODEL (default zai-org/GLM-5.3-Flash — cost tracking prices custom-LLM usage by this name, since ElevenLabs reports it only as custom-llm).
The provider API key never enters ms-ai. It exists only as a hand-created
ElevenLabs workspace secret, resolved by name to a secretId and referenced
as apiKey: { secretId }. The name→id map is memoised per process (promise
cached, dropped on failure, cooldown before a miss re-lists) because the web
path syncs on the blocking getCallUrl / getCallToken path. It never
throws — a missing secret or unreachable ElevenLabs returns null and the
call connects on the native server-default LLM. KNOX_VOICE_LLM=custom-llm
for the Nox voice agent is intentional. GET /elevenlabs/agent/:company_id?include_prompt=1
reads system_prompt + tool_ids back off the live agent (multiple KB, so
opt-in); backend liveInboundAgentFor() (app/lib/inboundAgentSync.js) uses
it to show what ElevenLabs is actually holding, which differs from
buildInboundSyncPayload().system_prompt by thousands of characters.
Speech settings & background sound
- None of this applies to S2S models — the dashboard hides both blocks for them (values kept).
company.ai_config.call_speech_settings— responsiveness (mapped to ElevenLabs turn-eagerness buckets),interruption_sensitivity(default 0.5), backchannel, audio normalization — andcompany.ai_config.call_background_sound(ElevenLabs-native ambience: preset track,volume/100, crossfade loop) apply to the widget/web agent.- Telephony agents carry per-agent Mixed
speech_settings/background_soundblocks on the sync DTOs (ms-ai/src/elevenlabs/dto/elevenlab-all.dto.ts), same shape. - Gotcha fixed:
updateAgentre-appliesbuildLanguagePresets— previously a settings update dropped the multilingual presets from the ElevenLabs agent.
SMS
Twilio SMS over the same subaccounts/numbers (purpose sms):
- ms-ai:
POST /elevenlabs/send-sms(send via the company subaccount; refused 409sms_registration_requiredfor an unregistered bought US 10DLC number — see A2P 10DLC),POST /elevenlabs/compose-sms(LLM-drafted message, cost bucketKnownAgent.SMS),POST /elevenlabs/import-phone-number/twilio/sms,POST /elevenlabs/buy-phone-number/twilio/sms. Costs swept bytwilio-sms-cost:meter(table above). - Backend: SMS agents (
company.sms_agents, gatesms_agents_allowed) withAiSmsAgentControllerCRUD, publicPOST /ai-sms/webhook(per-agentsecret_key; actionssend_sms|generate_sms), message logSmsAgentMessageModel, GHL workflow-action routes — see SMS Agent. POST /ai-outbound/send-sms— texts from an outbound call agent's number; signed exactly like the call webhook (webhook_secret), DNC-suppressed, relays to ms-aisend-sms. MeanwhilePOST /ai-outbound/sdk-triggeris disabled — hard 403, SDK error codeoutbound_unavailable.
Post-call pipeline
- ms-comm
InboundOutboundCallController.saveAndEmailpersists transcript/outcome. The post-call webhook is held until the ElevenLabs analysis lands (it carriessummary, sentiment, recording); single-fire via an atomic claim onpost_call_webhook_sent_at. - Backend
PostCallWebhookFallbackJob(every 1 min,SchedulerService): calls still unfired 5 min afterended_atfire with what exists (max age 24h, 5 attempts, batch 50, same claim — the two paths can never both send). Its siblingCallTranscriptEmailFallbackJob(every 1 min) does the same for the transcript email, which now waits for the analysis (call_summary_titlebecomes the subject, the summary renders above the transcript):status: 'completed',email_sent_at: {$in: [null]},email_fallback_attempts: {$not: {$gte: 5}}($not/$gte, not$lt—$ltmisses a missing field),'transcript.0'exists,ended_atbetween now−24h and now−5min, 50 per pass →POST {ms-comm}/inbound-outbound-calls/send-transcript-email {call_sid}. The mark-sent-before-send invariant holds:email_sent_atis claimed atomically before sending and$unseton failure. Extraction is claimed by the first invocation soextracted_contactkeeps the caller's real name out of "Phone caller". 2b. Live call events: on answer the bridge postsPOST /call-events/live(backendCallEventsController.live, apiKeyAuth, fire-and-forget, 5s timeout) — carrier-agnostic, both directions, deduped on Redis SETNXcall-events:live:<service_room_id>for 15 min (Redis blip falls through) — which fires the governedai-call-livepush to every admin device.POST /call-events/endedcarriesduration_seconds, stamped ontoservice_rooms.call_durationwithout overwriting; the super-admin Call Duration Backfill (POST /admin/calls/backfill-durations+ jobs poll, dry-run, cursor-streamed) fills history from ms-aiGET /knox-signals/transcripts/durations. FrontendCallPanel.isEventForThisCallrequires the session to match, ignoresreplaced_by_agent_callevents, and requires a namedservice_room_idto be ours — a visitor's widget call can end while the agent's call to the same visitor stays live. - Payload (
PostCallWebhookService, everything namespaced undernext_level_call):agent_key,transcript[],transcript_text(joinedSpeaker: text),summary,call_sentiment(positive|neutral|negative, classified from the caller's words via ms-aiPOST /agents/classify-call-sentiment, stored assentimenton the record), signedrecording_url. - Recording links: HMAC-SHA256 over
<conversationId>.<exp>, TTL ~365d, secretRECORDING_LINK_SECRET, fail-closed (no secret → no link).RecordingLinkServiceexists as identical copies in ms-comm (mint) and backend (mint + verify) — change together. Served by backendGET /public/recordings/:id?exp&tokenonapi.nextlevelai.site(streams from ElevenLabs, nothing stored; 403 bad signature / 410 expired). CRM inline players use the separate.mp3-suffixedGET {ms-comm}/recordings/ai/<id>.mp3(fromAiCallGhlTranscriptSyncJob). - Outcomes: phone
call_outcome∈booked|qualified|voicemail|no_answer|no_outcome, upgrade-only. Widget AI calls have a separate two-valueservice_rooms.call_outcome∈booked|qualified|''(declared in backend and ms-commServiceRoomModellock-step), written by ms-commMessageControllerfrom the ElevenLabs post-call payload:bookedwhen abookCalendarMeetingtool result succeeded (paired totool_callsby id, positional fallback), elsequalifiedwhen the visitor spoke; no signal → no claim. Also: the ms-comm calls list takes server-sidestart_date/end_date($oroverstarted_atandcreated_atwhenstarted_atis null) so the tab count and the list agree. Phone outcomes are upgrade-only (a late analysis may promotequalified→voicemail/booked, never downgrade). Outbound voicemail = ElevenLabs analysis flag OR AMD verdict (see Voicemail detection) OR transcript heuristic → record fieldvoicemail. GHL tagging runs in the backend (POST /internal/ai-call/contact-extractedapplies the agent'spost_call_tags; inbound built-ins arebooked/qualifiedonly), idempotent via ms-comm'spost_call_tags_appliedflag on the call record. - GHL recordings + notes are cron-driven in the backend: ms-comm stamps
ghl_recording_state='pending';GhlCallRecordingAttachJob(every 5 min) consumes it and attaches the recording to the GHL contact;GhlCallNotePostJob(every 1 min) posts call notes, deferring inbound notes until the summary lands. Shared libsbackend/app/lib/callRecordingUrl.js/callNoteBody.js/ghlErrors.js. ms-commGET /inbound-outbound-calls/:idfalls through to acall_sidlookup when:idisn't a Mongo id.
Human escalation
Bridge tool escalateToHuman (ElevenLabsBridgeService._handleEscalateToHuman; the widget websocket path still registers the legacy esclateToHuman spelling):
- Destination = the agent's
transfer_numberread live from Mongo (default agent onai_inbound_call, others inai_inbound_call_agents[]); blank →company.phone_number. - Mixed accounts: credentials resolve per dialed number via ms-ai
GET /elevenlabs/inbound-company-phone/:company_id?number=— BYO-imported and platform-bought (subaccount) numbers each transfer with their own creds (read from thephone_numberowner). For a number shared between an inbound agent and an outbound/SMS agent the lookup prefers the inbound binding (d1a61931e) — it used to take the newest row, which had no transfer line, so escalation came back empty. - Caller-ID pass-through: inbound transfers present the original caller's number; on carrier reject (Twilio 13223/21212, or Telnyx) retry once with the owned number. Outbound always presents the owned number.
- Twilio:
Calls.updatewith<Dial timeout="30" answerOnBridge="true">; Telnyx: TeXML call-update REST (POST /v2/texml/calls/:sid/update). Both pause ~1.2s so the "transferring you now" TTS finishes.
Related
Call Agent · Voice Engine (S2S) · Inbound Call Agent · Outbound Call Agent · SMS Agent · Calls · AI Credits