Skip to main content

Telephony

Phone numbers and voice transport for the AI call agents, split across three services: ms-ai owns numbers, purchase, and rental billing (src/elevenlabs/); ms-communication owns the live audio bridges and post-call persistence; backend owns the wallet ledger and the webhook fallback cron.

Carriers​

CarrierNumbersAudio pathCredentials
Twilioin-app purchase (per-company subaccounts) + BYO importTwiML <Connect><Stream> → /twilio/media-stream WS → ElevenLabsms-ai master TWILIO_MASTER_ACCOUNT_SID/_AUTH_TOKEN creates subaccounts; BYO = per-number SID + token
TelnyxBYO import onlyTeXML <Connect><Stream> → /telnyx/media-stream WS → same bridgeper-number telnyx_api_key + telnyx_connection_id (TeXML app), stored on the phone_number owner row
  • Carrier is per number: elabs_phone_number.provider (twilio/telnyx) in ms-ai selects the dial path; inbound_outbound_calls.provider in ms-comm records it. ElevenLabs has no native Telnyx support — ms-ai synthesizes number_id: telnyx_<uuid> and the ms-comm bridge carries the audio.
  • ms-comm Telnyx surface: TelnyxInboundCallController / TelnyxOutboundCallController + TelnyxHelper (TeXML builders). Routes: POST /telnyx/inbound-voice, /telnyx/inbound-status (neutral, current), legacy /telnyx/inbound-voice/:company_id + /telnyx/inbound-status/:company_id, /telnyx/outbound-voice/:company_id, /telnyx/outbound-status/:company_id, /telnyx/preheat-outbound/:company_id/:agent_id.
  • Telnyx inbound routes by the dialled number (ms-comm f72871b, ms-ai f0dd8ba94). A TeXML Application has one voice_url for all its numbers and customers reuse one Application across companies, so importing used to rewrite it with the importer's company and every number rang the last importer's agent. Import now writes the neutral URLs; on a call ms-comm asks ms-ai GET /elevenlabs/telnyx-inbound-route?number=<To> (secret auth) → { company_id, agent_id, agent_key, webhook_url, company_phone }. An outbound-only number routes to the company's latest inbound Telnyx binding on the same Application, else its default inbound agent. The legacy URL's company is a fallback only for numbers ms-ai doesn't know (or a missing agent on the same company) — never another tenant's per-URL values. The owner is cached per call in Redis telnyx-inbound-company:<CallSid> (6h) for the neutral status callback; the prewarm budget shrinks by time already spent so the webhook stays under Telnyx's ~15s. Deploy ms-comm before ms-ai. Env: TELNYX_MEDIA_STREAM_PATH (default /telnyx/media-stream), TELNYX_PUBLIC_WS_URL (else derived from ELEVENLABS_PUBLIC_WS_URL).
  • Codecs: Twilio is always PCMU (mu-law) 8k. Telnyx inbound legs are PCMU, outbound/international PCMA (A-law) — TeXML declares bidirectionalCodec per direction; AudioTranscodeHelper (A-law support) normalizes to mu-law on ingest so the rest of the bridge is unchanged.
  • Live monitor/takeover is carrier-agnostic, keyed by service_room_id: both bridges fan audio + transcripts into ai-listen:<service_room_id>, served by the existing socket handlers — no Telnyx-specific sockets.

Phone number ownership (ms-ai phone_number)​

Since 8cee58357 (migration 1794197775714-central-phone-number) every number a company or agency holds has one phone_number row — owner, carrier credentials, billing and suspension — and each agent's use of it is an elabs_phone_number binding pointing at it (phone_number_id, eager ManyToOne; a QueryBuilder must join it itself).

  • Columns: number (E.164), provider, exactly one of company_id / agency_id (check constraint), source ∈ app_purchase | byo | agency_purchase | agency_import, twilio_sid, twilio_auth_token + telnyx_api_key (encrypted at rest via encryptedText), twilio_number_sid, telnyx_connection_id, monthly_price_usd, country, number_type, purchased_at, next_rental_at, suspended_at, release_at, warnings_sent, released_at (history, never matched again). Unique per company+number and agency+number.
  • Why: a number attached to a second agent got a binding with no source/price/rental date, so it read as BYO — outbound calls and texts skipped the wallet gate, removing the purchased binding stopped the rent, and a re-import wiped billing. Rent, suspension and cost sweeps, the wallet gates, removal, trust checks and the list endpoints now all read the owner. Helpers in src/elevenlabs/owned-numbers.ts (ensureOwner only ever upgrades a source to app_purchase, credsOf, ownershipFields, forgetIfUnused).
  • Disconnect ≠ release. Dropping the last binding of a bought number keeps the owner (still billed); a BYO owner is forgotten once nothing uses it. POST /elevenlabs/remove-phone-number/from-account drops every binding; release_from_carrier: true releases a bought number at Twilio, false is a plain disconnect, omitted → 409 requires_release_confirmation. A customer's own number is never touched at the carrier.
  • GET /elevenlabs/phone-numbers/:company_id returns one row per binding with ownership read from the owner, plus agency_id, a2p_applies (isUs10dlc), suspended/release_at. include_unused=true adds owned numbers with no binding (connected: false, used by billing Add-ons); include_twilio=true also adds numbers found in the subaccount with no record (the central Phone Numbers screen).
  • Credentials never leave ms-ai (a841c02d1): all 11 buy/import/attach/assign endpoints return withoutPhoneCredentials(row) (phone-number-public.ts) — before, the response carried the subaccount token, letting a tenant drive the subaccount outside the wallet. Stripped at the controller, never in the service (the buy path re-saves the row the import returns).
  • Reconcile (internal/reconcile-phone-numbers, one page of subaccounts per call, dry run unless apply): finds numbers rented in a subaccount but never billed and records them as bought, billing from now (no back-charges); page one also links bindings an older pod wrote unlinked. Super-admin wrapper: backend POST /admin/maintenance/reconcile-phone-numbers (background job looping next_page), card Reconcile rented phone numbers in Admin → Integrations maintenance.
  • Agency-pool rows are not migrated yet (agent_key: 'agency-pool'; rent idempotency agency-twilio-rental:<poolRowId>:<period>).

Number lifecycle (ms-ai)​

Buy (Twilio only). GET /elevenlabs/available-phone-numbers/twilio → POST /elevenlabs/buy-phone-number/twilio/{inbound|outbound|sms} (+ cancel-phone-number/twilio). The UI offers US/CA/GB/AU × local/toll-free/mobile (the server accepts any ISO country, default US). The number is purchased into the company's Twilio subaccount (findOrCreateTwilioSubaccount, FriendlyName knock-sub:<company_id>; subaccount SID + token live on the number's meta). The buy is gated on the wallet covering the marked-up price (markup_percent from the wallet response; 402 when short) and the first month is charged at purchase (meta.next_rental_at starts the clock).

Billing crons — NumberRentalScheduler (src/elevenlabs/number-rental.scheduler.ts; kill switch NUMBER_RENTAL_CRON_ENABLED=false; app-purchased numbers only, BYO is never touched):

Cron nameScheduleDoes
number-rental:meter-duedaily 06:00 UTCmonthly rental debit, idempotent on twilio-rental:<number>:<period>; catches up missed months; frozen while suspended
twilio-call-cost:meterevery 10 minsweeps each subaccount's completed calls (48h lookback), debits Twilio's settled per-call price, idempotent on twilio-call:<CallSid>
twilio-sms-cost:meterevery 10 mintwin of the call sweep for sent SMS (meterTwilioSmsCosts()), debits Twilio's settled per-message price, idempotent per Message SID
number-rental:suspension-sweepevery 15 minsuspend / warn / release lifecycle below

Suspension (state on the phone_number owner: suspended_at, release_at, warnings_sent): wallet < 0 → suspend, release_at = +7d, day-0 warning; day-3/day-6 warning emails; past release_at → release from Twilio; wallet back > 0 → auto-unsuspend. Emails route through backend POST /internal/telephony/notify (suspend_warning / released → subscription/number-suspended.html / number-released.html).

Agency pause. Before anything else, ms-comm's inbound handler checks company.paused_at: a paused client's calls go to a human (first routing-priority number → transfer numbers → company.phone_number) or get the "temporarily unavailable" message — the AI never answers or bills. Fails open.

Inbound block. ms-comm calls GET /elevenlabs/telephony-inbound-allowed/:company_id?number= (4s timeout, fail-open) before answering; blocked callers hear "This number is temporarily unavailable. Goodbye." Only meta.source === 'app_purchase' numbers are gated.

Ops: POST /elevenlabs/internal/meter-rentals, POST /elevenlabs/internal/sweep-suspensions, GET /elevenlabs/internal/number-detail.

Verification & registration (Twilio Trust Hub, Regulatory, A2P)​

Three carrier review flows, all in ms-ai src/elevenlabs/, all advanced by TwilioTrustScheduler (leader-locked; kill switch TWILIO_TRUST_CRON_ENABLED=false): twilio-trust:sync every 30 min (Trust Hub + A2P), twilio-regulatory:sync at :15/:45. Each sync is idempotent and also runs on submit, on purchase (onNumberPurchased, not awaited) and on a stale status read.

1. Business verification — Trust Hub (twilio-trust.service.ts, table twilio_trust_registration). Fixes "Spam Likely" on outbound caller ID: KnockKnock's Primary Customer Profile (master account; TWILIO_PRIMARY_CUSTOMER_PROFILE_SID or auto-discovered) → a Secondary Customer Profile per company in its knock-sub: subaccount → trust products SHAKEN/STIR, Voice Integrity, CNAM (optional; not toll-free), created only once the profile is approved. Sync assigns every +1 subaccount number to the profile and each product, so numbers bought before or after are covered.

  • Routes: GET /elevenlabs/trust-registration/:company_id, POST /elevenlabs/trust-registration (submit / resubmit; an approved profile is never resubmitted — only rejected products are rebuilt).
  • Outbound buy gate: buying a US outbound number throws 428 business_registration_required unless a non-rejected profile exists (TWILIO_TRUST_REQUIRED_FOR_OUTBOUND, default true); the buy modal shows the form, and skip_trust_registration ("Skip for now, do it later") buys anyway.
  • Rejection reasons (464c4d9a7, twilio-rejection-reasons.ts): Twilio review rejections carry only an error code; codes 186xx etc. map to { title, fix, fields[] }, stored as issues on the registration and returned to the form, which marks the fields. Migration 1793900000000-twilio-trust-rejection-issues. One-shot backfill: ms-ai POST /elevenlabs/internal/trust-registrations/refresh-rejections (reads Twilio only) wrapped by backend POST /admin/maintenance/trust-refresh-rejections (admin card Refresh business verification rejections).

2. Number verification — Regulatory Compliance (twilio-regulatory.service.ts / twilio-regulatory.controller.ts, table twilio_regulatory_bundle). GB, AU and NZ won't sell a number without an approved bundle per country + number type (US/CA need none; SUPPORTED_COUNTRIES = US CA GB AU NZ). The form is generated from Twilio's live Regulation, not hard-coded; uploads stream to Twilio, never stored. A type chosen for several requirements is sent as one document with all their fields (Twilio 70002 refuses two documents of one type). Buying without approval → regulatory_bundle_required / regulatory_in_review.

  • Routes (company_id is a path param so the tenant guard runs before multipart parsing): GET /elevenlabs/regulatory/:company_id?country&type, GET /elevenlabs/regulatory/:company_id/statuses (every started verification, for the status cards), POST /elevenlabs/regulatory/:company_id (multipart, ≤10 files), ops POST /elevenlabs/internal/regulatory/sync.

3. US texting — A2P 10DLC (d75d94198; state in the a2p jsonb column, migration 1793774257012-twilio-trust-a2p). Built on the approved Secondary Profile: A2P Messaging Profile trust product (TWILIO_A2P_POLICY_SID override) → Brand (TCR; low-volume brands skip secondary vetting) → Messaging Service (keeps each number's own inbound webhook) → Usa2p campaign → every US long code in the subaccount added to the service. One step per sync.

  • Route: POST /elevenlabs/trust-registration/sms (submit / fix-and-resubmit; an approved brand is never re-vetted). Public status none | in_review | approved | action_required on the sms key of the trust-registration GET, with per-step profile/brand/campaign statuses, error, fees, and sms_required.
  • Fees debit the company wallet, source telephony_sms_registration (wallet label "Texting registration"), idempotent per a2p-brand:<sid> / a2p-campaign:<sid> / a2p-campaign-month:<sid>:<YYYY-MM>. Raw defaults (USD) brand 4.5, secondary vetting 41.5, campaign vetting 15, monthly 10 (1.5 for LOW_VOLUME); override TWILIO_A2P_FEES_USD (JSON). The backend debit adds markup; submit is refused 402 insufficient_balance when the wallet can't cover the marked-up total.
  • Send block (assertCanSendSms, called by send-sms): a company-bought US 10DLC number (isUs10dlc in us-10dlc.ts: +1, not toll-free, not a Canadian area code) can't text until the campaign is VERIFIED → 409 sms_registration_required with a human reason; backend AiSmsWebhookController logs it as a failed row. Agency-pool and BYO numbers are never blocked (the UI warns). Kill switch SMS_A2P_REQUIRED=false.

Notifications. ms-ai _notifyTelephony → backend POST /internal/telephony/notify (TelephonyWalletController) with type sms_a2p (blocked at most once per 24h per company, approved, rejected), business_verification or number_verification (approved / rejected, only on a status change). Each sends an in-app system notification (relations.kind/event/path; push except on approval) and an email to the billing owner via subscription/sms-registration.html. The link goes to the first agent Phone Numbers tab the company has (_phoneNumbersPath), not the central screen.

Frontend. Central screen views/settings/ai/PhoneNumbers.vue (route /settings/ai/phone-numbers, menu Settings → AI Agents → Phone Numbers): one row per number grouped on digits (components/telephony/phoneNumbersOverview.js — any binding naming app_purchase/agency wins over a sourceless extra attachment), used-by links, spam-label and A2P columns, price/next charge, suspension; above it BusinessVerificationCard, SmsRegistrationCard, RegulatoryStatusCard (a rejected bundle is fixed in a modal; an approved one deep-links to an agent's buy popup with ?buy=CC:type). The same cards sit on the inbound/outbound/SMS agents' Phone Numbers tabs; SmsRegistrationCard falls back to the business form while the business is unsubmitted or rejected.

Carrier webhook & token auth (ms-comm 411462b)​

  • Twilio webhooks (/twilio/inbound-*, /twilio/outbound-*, /twilio/amd-status, /twilio/inbound-screen, /twilio/inbound-dial-fallback, /twilio/call, /service-room/twilio-callback) sit behind app.middlewares.twilioSignature: X-Twilio-Signature must verify against the env tokens (CALL_TWILIO_ACCOUNT_AUTH, TWILIO_AUTH_TOKEN) or the company's own account token for the To/From number (fetched from ms-ai, cached per pod) — 403 otherwise. Emergency off: TWILIO_VALIDATE_SIGNATURE=false.
  • Telnyx webhooks check the Ed25519 telnyx-signature-ed25519 over timestamp|raw body (±300s) against TELNYX_PUBLIC_KEY (base64, comma-separated). Unset = accepted unverified (BYO accounts sign with their own keys) — logs a CRITICAL once in production. Prod sets neither TWILIO_AUTH_TOKEN nor TELNYX_PUBLIC_KEY in flux.
  • Media streams: the <Stream> carries kk_exp + kk_sig (HMAC-SHA256 of company_id|agent_id|exp, key derived from WEBHOOK_API_KEY, 2h TTL) or a valid Twilio signature on the WS upgrade (MediaStreamAuthHelper). A stream's webhook_url is honoured only on our own hosts.
  • Toll fraud — widget "forward to phone": POST /twilio/token mints an identity of visitor_<user_session_id> only when that session belongs to the given company (403 otherwise), else visitor_anon_<random>; the widget sends its user_session_id (widget 5711ddd). The TwiML App URL /twilio/call dials only a company's own published phone_number with allow_twilio_calls not false (a session-bound identity only its own company's), never an arbitrary To, and checks ApplicationSid.

Telephony wallet (backend)​

TelephonyWalletController (routes in app/routes/internalTelephonyRoutes.js, guarded by apiKeyAuth = BACKEND_API_KEY). The wallet is the subscription's remaining_purchased_ai_credits (1 credit = $0.01); plan remaining_ai_credits is AI-only and never pays telephony.

RoutePurpose
GET /internal/telephony/wallet/:company_idwallet_credits, plan_credits, has_telephony (wallet > 0), ai_available (plan > 0 || wallet > 0), markup_percent
POST /internal/telephony/debitidempotent on idempotency_key (unique in ai_credit_logs); charges base × (1 + markup) at 4-dp precision; balance may go negative (paid down by the next top-up)
POST /internal/telephony/notifysuspension/release emails to the resolved billing user

Markup: global setting telephony_markup_percent (default 0), applied on every debit (margin recorded on the ledger row) and returned so ms-ai gates the buy and the frontend shows final prices. Wallet UI: frontend/src/views/packages/Wallet.vue; buy UI in the agents' Phone Numbers tabs (markup via useTelephonyMarkup.js).

Transports​

Widget AI calls — WebRTC (LiveKit). ms-comm hardcodes calling_type: 'webrtc' on POST {ms-ai}/elevenlabs/start-audio-call (CallsController); ms-ai returns a LiveKit conversation token (the websocket path still returns signedUrl) and the widget starts the ElevenLabs SDK with { conversationToken, connectionType: 'webrtc' } (widget/src/services/aiCall.ts). Feature parity runs on SDK callbacks instead of the raw socket: transcripts, typed user messages, product cards (events-gateway), booking confirmation (delivered as a userMsg — the only signal the agent gets on LiveKit), and live-listen taps the agent's LiveKit audio track into the audio_chunk stream. Token wait is 30s (was 15s); a ringback loop (on-call-ringing.mp3, started inside the click gesture in widget/src/stores/widget.ts) covers the connect gap.

PSTN. Dual-WebSocket bridge (ElevenLabsBridgeService): carrier leg (mu-law/A-law 8k) ↔ ElevenLabs Convai (PCM 16k), transcoding both ways, transcript + audio fan-out to ai-listen:<service_room_id>.

Speech-to-speech (S2S). When the agent's model is a native S2S model the company may use (GPT Live 1 and Gemini 3.8 Live are open to every company), ms-ai stamps voice_provider on the TwiML URL and the same bridge connects the carrier leg straight to the vendor — app/services/voiceDirect/ (GeminiLiveAdapter with a credential ms-ai mints; LiveAdapter for GPT-Live, which opens the Live API with ms-comm's own OpenAI key and session.start from ms-ai /voice-direct/setup; RealtimeAdapter for the retired Realtime models / xAI). The vendor session is pre-warmed while the phone rings (Redis vd:warm:<callSid>, adopted by whichever replica gets the media stream). Audio + final transcripts do fan out to ai-listen:<service_room_id>, so monitor and takeover work for every vendor (takeover ends the vendor session, keeps the carrier leg). The bridge also runs the greeting guard (caller audio held until the opener has played), DirectCallGuard (end-intent + idle check-in/hang-up) and, on Gemini, the external OpenAI caller transcriber. Inbound direct calls get the caller-verification tool when the agent requires it. ms-comm records the call itself. A declined or failed S2S session falls back to the agent's ElevenLabs standby on the same call. Widget S2S calls bypass the ElevenLabs LiveKit path (WebRTC to OpenAI; Gemini over ms-comm's own LiveKit room or WS relay; xAI through the relay). Grok phone calls (XAI_SIP_ENABLED=true) skip the bridge entirely: the voice webhook answers with a <Dial> to sip.voice.x.ai (Twilio records the bridged leg), so there is no live listen/takeover, and the SIP leg (joined to its parent call's activity line) plus the Twilio recording (telephony_recording, "Call recording") are debited to the telephony wallet at sub-cent precision — see Voice Engine › Recording. Full detail: Voice Engine.

Number lookups (ms-ai bb71ab594, 3be179c8c). elabs_phone_number.number holds whatever spelling the UI or an import wrote (+1 938 388 5440, (938) 388-5440), so every lookup keyed by a number arriving from outside (carrier webhook, called/dialled number) uses PhoneNumberEquals (src/elevenlabs/phone-number-match.ts, digits-only regexp_replace compare; exact match when the input has no digits). Exact Equal refused S2S inbound with "no phone record" and let the telephony wallet check fall open on an unknown number. One number may carry an inbound and an outbound row (each with its own agent_key): inbound call context prefers meta.purpose: 'inbound', falling back to any row for numbers that predate purpose — an arbitrary pick used to resolve the outbound key and decline no_agent_config.

Agent-sync skip cache (web only). start-audio-call used to re-sync the ElevenLabs agent on every connect (~6.2s). ms-ai now skips the sync when nothing changed: per-replica in-memory cache keyed on a sha256 of the agent config, TTL AGENT_SYNC_CACHE_TTL_MS (default 5 min), invalidated via invalidateAgentSyncCache() on every mutating path (elevenlabs.service.ts). Telephony agents are deliberately not cached — they have preheat.

WebRTC parity fixes: service-room-update-data (stores elabs_conversation_id on the service room — the key for recordings and the transcript email) now fires on both transports, not just websocket; the widget emits ai-call-client-log → ms-comm CallsController persists client-side call diagnostics (event, transport, handshake UA) into the logs collection; transcript-email timestamps render in the company timezone.

Inbound routing priorities​

Per inbound agent routing_priorities: [{ number, ring_timeout }] — ring real phones in order before the AI. Sanitized in backend AiInboundAgentController._sanitizeRoutingPriorities (max 10 numbers, timeout clamped 5–120s, default 20s); UI InboundCallAgent/CallForwarding.vue. ms-comm dials the chain with <Dial timeout=… answerOnBridge>, advancing via POST /twilio/inbound-dial-fallback/:company_id. Empty list = straight to AI; applies regardless of working hours; any error fails open to the AI. The wallet gate runs before the chain — forwarding legs cost carrier minutes too.

Inbound call screening (Twilio only)​

Optional <Gather> keypress gate that runs before routing priorities, so auto-dialers never reach the AI (or a forwarded human):

  • ms-comm POST /twilio/inbound-screen/:company_id sits atop handleInboundVoice. Stateless — the expected digit and agent ids ride the Gather action URL (?d=… etc.), no session store. Any error fails open to the normal flow.
  • Config: per-inbound-agent agent.call_screening (sanitized in backend AiInboundAgentController._sanitizeCallScreening, normalized again in ms-comm _normalizeCallScreening).
  • Failed screens land in ScreenedCallModel (screened_calls: unique call_sid upsert so webhook retries don't double-log; reason: no_input | wrong_digit; TTL index 90d). Deliberately NOT inbound_outbound_calls — a screened call has no transcript/duration and must not pollute call lists, Knox stats, briefs, or GHL sync. Screening never blocks a number; each call gets a fresh prompt.
  • Latency: preheat starts during the Gather (fire-and-forget); on screened re-entry (?screened=1) the prewarm budget is 3s (vs 11s) and nothing runs before <Connect>.
  • No Telnyx equivalent. Read route for the dashboard log: backend GET /ai-inbound-agents/:agent_key/screened-calls.

Voicemail detection (async AMD, outbound)​

Every outbound dial now carries Twilio/Telnyx async answering-machine detection:

  • ms-ai _appendAmdParams (elevenlabs.service.ts) adds MachineDetection=Enable + AsyncAmd=true + status callback — the call connects immediately, the AMD verdict arrives async.
  • Callbacks: ms-comm POST /twilio/amd-status/:company_id (OutboundCallController.handleAmdStatus) and POST /telnyx/amd-status/:company_id (TelnyxOutboundCallController) — both always answer 204. Telnyx verdicts match the bridge registry by both CallSid and CallSidLegacy (Telnyx doesn't document which equals the TeXML sid).
  • ElevenLabsBridgeService.handleMachineDetected acts only on machine* / fax verdicts; human / unknown are no-ops. The record's voicemail flag is written with $max — sticky true, a late human can't un-flag it.
  • Env (ms-ai): AMD_ENABLED (default true), AMD_TIMEOUT_SECONDS (20), AMD_SPEECH_THRESHOLD_MS (3000). If the carrier rejects the AMD params the dial retries once without AMD. Log tag [voicemailAMD].

Cross-call memory (continuation first message)​

Returning callers get a personalized opener instead of the stock greeting:

  • ms-ai POST /agents/generate-first-message — public, throttled (5/s, 60/min), backed by FirstMessageAgent (modules/agents/definitions/first-message.agent.ts). Always returns 200 with the default greeting on any failure; never emits {{…}} template tokens.
  • ms-comm helpers/FirstMessageHelper.js wraps the call (20s timeout — the LLM round-trip is the work, not overhead; callers bound total wait themselves).
  • PSTN: ElevenLabsBridgeService._loadPriorCallTranscript keys on caller_number (index { caller_number: 1, started_at: -1 }), requires at least one caller turn, takes the last 30 turns. _getContinuationFirstMessageCached dedupes so preheat + call start share one Mongo read/LLM call. Total budget on the connect path: FIRST_MESSAGE_INIT_CAP_MS (2000) — over budget falls back to the configured greeting. Gate: per-inbound-agent continue_previous_conversation.
  • Web: gate ai_config.call_continue_previous_conversation (widget opt-in). The generated opener rides the joining-ai-call payload as a per-session first_message override — never baked into the shared ElevenLabs agent. An accepted auto-trigger greeting wins over the continuation opener.

Caller authentication (inbound, 2026-09)​

Inbound phone agents can verify a caller by username + password before saying anything account-specific. Three AND gates (backend/app/lib/aiAgentAuthentication.js isAuthenticationEffective): company.ai_agent_authentication_allowed (super-admin, default false) && company.ai_agent_authentication.enabled (tenant) && agent.authentication_required (per inbound agent, declared on ai_inbound_call — keep it declared or the strict subdoc strips it).

  • Users: ai_agent_authentication.users (Mixed, select:false, max 500), max_attempts (1–5, default 3), failure_action (end_call | continue_without_access). Passwords are scrypt (scrypt$<salt>$<hash>, N=16384/r=8/p=1, keylen 64, salt per user) over the normalized spoken form (lowercase, whitespace and punctuation stripped — "Hunter 2" stores as "hunter2"); timingSafeEqual; an unknown username still verifies against a decoy hash so timing can't enumerate users. Rate limits in AiAgentAuthenticationService: per-conversation = max_attempts, per-company 120/60s, per-caller-number.
  • Routes: tenant GET|PUT /ai-agent-authentication, POST …/users, PUT|DELETE …/users/:id (JWT); super-admin PUT /company/:_id/ai-agent-authentication-allowed; internal (apiKeyAuth) POST /internal/ai-agent/authenticate and GET /internal/ai-agent/authentication-status/:company_id — both always 200. Both ai_agent_authentication* keys are in noxConfigFieldMap.js FORBIDDEN_PREFIXES, so Nox can never write them.
  • ms-ai tool company_user_authentication (hosted name companyUserAuthentication, webhook elevenlabs/tools/company-user-authentication, header secret from workspace secret COMPANY_AUTH_TOOL_AUTH — ElevenLabs drops plain-string headers). company-auth-tool.service.ts resolves the tenant from system__agent_id via elabs_agent; the company_id dynamic var is a cross-check only (company_mismatch refuses); non-inbound-telephony refused. Forwards {company_id, agent_key, username, password, conversation_id, call_sid, caller_number} and returns the backend body verbatim: {authenticated: true, instruction} or {authenticated: false, message, attempts_remaining, locked, instruction} / verification_unavailable / not_enabled. Success reveals nothing about the user.
  • Direct S2S inbound (027255494): the same tool is registered on direct (GPT-Live / Gemini) inbound calls when the hosted-tool list includes companyUserAuthentication, but VoiceDirectToolRunner answers it in-process with the session's own company + agent_key (no webhook, no system__ vars, no secret); the model supplies only username/password, and VoiceDirectTrace scrubs them from debug traces.
  • Prompt placement (elevenlabs.service.ts ~1494–1524, hardened in d891d7471 / bb2c9098c / d92567856): the full # Caller Verification block goes first (right after the company prompt) and a short # Before you say anything else reminder goes last, after the caller context — primacy and recency; middle-only and end-only both failed in testing. Both sit after the company prompt so the per-call refresh strip regenerates rather than accumulates. Attached only for inbound-telephony when portalTools includes companyUserAuthentication.
  • ms-comm: ElevenLabsBridgeService._isAuthRequired() re-derives the three-gate rule from a projected lean read (30s cache, fails closed) and passes ?auth_required=1 on the tools refresh. Live redaction: _notePasswordPrompt arms on an agent turn matching /\bpassword\b/i, and _redactSpokenPassword replaces the next caller turn with [password redacted] before logging, socket fan-out, and transcript persistence. Post-call, ms-ai redact-auth-transcript.ts runs at the top of postCallWebhook — before webhook_inbox storage and before forwarding — scrubbing the tool's params_as_json / tool_details and the nearest preceding user turn (and original_message).
  • Never persisted: the plaintext password anywhere; users is select:false and never returned to the dashboard; audit lines carry masked usernames (j***); failure messages never say which field was wrong.
  • preserveInboundAgentFields (aiAgentAuthentication.js, used in CompanyController update): the default agent saves as one whole ai_inbound_call object and Mongo $set replaces the subdoc, so a browser tab holding a pre-toggle copy silently erased authentication_required (and webhook_secret) by saving a new greeting. Server-owned keys are now carried forward when omitted, then authentication_required is forced false unless ai_agent_authentication_allowed. This is a different failure mode from the strict-subdoc drop — declaring the field doesn't fix it.

Agent memory gating​

Per-agent use_memory (absent = true) and continue_previous_conversation (absent = false) resolve through ms-comm AgentMemoryFlagHelper.resolveAgentFlags(companyId, {direction, agentKey}) — one cached (30s) lean read; use_memory fails open, continuation fails closed, and continuation is ANDed with use_memory. Consumed in ElevenLabsBridgeService (456970e): gates _getVisitorContextByPhoneCached and the continuation opener.

Widget voice uses ai_config.call_use_memory, resolved by CallsController._isWidgetVoiceMemoryEnabled as a deliberate .lean() raw read — ms-comm's CompanyModel is a mirror, and a hydrated read would drop the path and flip memory back on. Off also stamps _stampRoomMemoryOptOut (room excluded from fact extraction and identity-graph past_conversations) and skips _loadPriorTranscript. In ms-ai, use_memory is part of the agent-config hash so a toggle re-syncs the prompt, and memory-off withholds upsert_memory / delete_memory_slot and removes them from the instructions (memory-tool-gating.spec.ts). Outbound (telephony) agents are inert — null user_session_id, no {{visitor_context}}, useMemory hardcoded true.

Custom LLM for ElevenLabs agents​

ms-ai/src/elevenlabs/call-models.ts is the single source of truth for the AI Model selector. NATIVE ids write prompt.llm = <id>; CUSTOM ids (Baseten-hosted: baseten-glm-5.3-flash → zai-org/GLM-5.3-Flash, added 2026-09-25 when Baseten deprecated GLM 4.7; baseten-glm-4.7 still listed; both use the BASETEN_GLM47 workspace secret, which holds the account key) write prompt.llm = 'custom-llm' plus a prompt.customLlm block built by custom-llm.service.ts pointing at BASETEN_INFERENCE_URL (https://inference.baseten.co/v1; ElevenLabs appends /chat/completions). Persisted as ai_config.call_model / ai_inbound_call_agents[].model / ai_outbound_call_agents[].model. Env: ELEVENLABS_CUSTOM_LLM_PROVIDER (default baseten), ELEVENLABS_CUSTOM_LLM_MODEL (default zai-org/GLM-5.3-Flash — cost tracking prices custom-LLM usage by this name, since ElevenLabs reports it only as custom-llm).

The provider API key never enters ms-ai. It exists only as a hand-created ElevenLabs workspace secret, resolved by name to a secretId and referenced as apiKey: { secretId }. The name→id map is memoised per process (promise cached, dropped on failure, cooldown before a miss re-lists) because the web path syncs on the blocking getCallUrl / getCallToken path. It never throws — a missing secret or unreachable ElevenLabs returns null and the call connects on the native server-default LLM. KNOX_VOICE_LLM=custom-llm for the Nox voice agent is intentional. GET /elevenlabs/agent/:company_id?include_prompt=1 reads system_prompt + tool_ids back off the live agent (multiple KB, so opt-in); backend liveInboundAgentFor() (app/lib/inboundAgentSync.js) uses it to show what ElevenLabs is actually holding, which differs from buildInboundSyncPayload().system_prompt by thousands of characters.

Speech settings & background sound​

  • None of this applies to S2S models — the dashboard hides both blocks for them (values kept).
  • company.ai_config.call_speech_settings — responsiveness (mapped to ElevenLabs turn-eagerness buckets), interruption_sensitivity (default 0.5), backchannel, audio normalization — and company.ai_config.call_background_sound (ElevenLabs-native ambience: preset track, volume/100, crossfade loop) apply to the widget/web agent.
  • Telephony agents carry per-agent Mixed speech_settings / background_sound blocks on the sync DTOs (ms-ai/src/elevenlabs/dto/elevenlab-all.dto.ts), same shape.
  • Gotcha fixed: updateAgent re-applies buildLanguagePresets — previously a settings update dropped the multilingual presets from the ElevenLabs agent.

SMS​

Twilio SMS over the same subaccounts/numbers (purpose sms):

  • ms-ai: POST /elevenlabs/send-sms (send via the company subaccount; refused 409 sms_registration_required for an unregistered bought US 10DLC number — see A2P 10DLC), POST /elevenlabs/compose-sms (LLM-drafted message, cost bucket KnownAgent.SMS), POST /elevenlabs/import-phone-number/twilio/sms, POST /elevenlabs/buy-phone-number/twilio/sms. Costs swept by twilio-sms-cost:meter (table above).
  • Backend: SMS agents (company.sms_agents, gate sms_agents_allowed) with AiSmsAgentController CRUD, public POST /ai-sms/webhook (per-agent secret_key; actions send_sms | generate_sms), message log SmsAgentMessageModel, GHL workflow-action routes — see SMS Agent.
  • POST /ai-outbound/send-sms — texts from an outbound call agent's number; signed exactly like the call webhook (webhook_secret), DNC-suppressed, relays to ms-ai send-sms. Meanwhile POST /ai-outbound/sdk-trigger is disabled — hard 403, SDK error code outbound_unavailable.

Post-call pipeline​

  1. ms-comm InboundOutboundCallController.saveAndEmail persists transcript/outcome. The post-call webhook is held until the ElevenLabs analysis lands (it carries summary, sentiment, recording); single-fire via an atomic claim on post_call_webhook_sent_at.
  2. Backend PostCallWebhookFallbackJob (every 1 min, SchedulerService): calls still unfired 5 min after ended_at fire with what exists (max age 24h, 5 attempts, batch 50, same claim — the two paths can never both send). Its sibling CallTranscriptEmailFallbackJob (every 1 min) does the same for the transcript email, which now waits for the analysis (call_summary_title becomes the subject, the summary renders above the transcript): status: 'completed', email_sent_at: {$in: [null]}, email_fallback_attempts: {$not: {$gte: 5}} ($not/$gte, not $lt — $lt misses a missing field), 'transcript.0' exists, ended_at between now−24h and now−5min, 50 per pass → POST {ms-comm}/inbound-outbound-calls/send-transcript-email {call_sid}. The mark-sent-before-send invariant holds: email_sent_at is claimed atomically before sending and $unset on failure. Extraction is claimed by the first invocation so extracted_contact keeps the caller's real name out of "Phone caller". 2b. Live call events: on answer the bridge posts POST /call-events/live (backend CallEventsController.live, apiKeyAuth, fire-and-forget, 5s timeout) — carrier-agnostic, both directions, deduped on Redis SETNX call-events:live:<service_room_id> for 15 min (Redis blip falls through) — which fires the governed ai-call-live push to every admin device. POST /call-events/ended carries duration_seconds, stamped onto service_rooms.call_duration without overwriting; the super-admin Call Duration Backfill (POST /admin/calls/backfill-durations + jobs poll, dry-run, cursor-streamed) fills history from ms-ai GET /knox-signals/transcripts/durations. Frontend CallPanel.isEventForThisCall requires the session to match, ignores replaced_by_agent_call events, and requires a named service_room_id to be ours — a visitor's widget call can end while the agent's call to the same visitor stays live.
  3. Payload (PostCallWebhookService, everything namespaced under next_level_call): agent_key, transcript[], transcript_text (joined Speaker: text), summary, call_sentiment (positive|neutral|negative, classified from the caller's words via ms-ai POST /agents/classify-call-sentiment, stored as sentiment on the record), signed recording_url.
  4. Recording links: HMAC-SHA256 over <conversationId>.<exp>, TTL ~365d, secret RECORDING_LINK_SECRET, fail-closed (no secret → no link). RecordingLinkService exists as identical copies in ms-comm (mint) and backend (mint + verify) — change together. Served by backend GET /public/recordings/:id?exp&token on api.nextlevelai.site (streams from ElevenLabs, nothing stored; 403 bad signature / 410 expired). CRM inline players use the separate .mp3-suffixed GET {ms-comm}/recordings/ai/<id>.mp3 (from AiCallGhlTranscriptSyncJob).
  5. Outcomes: phone call_outcome ∈ booked|qualified|voicemail|no_answer|no_outcome, upgrade-only. Widget AI calls have a separate two-value service_rooms.call_outcome ∈ booked|qualified|'' (declared in backend and ms-comm ServiceRoomModel lock-step), written by ms-comm MessageController from the ElevenLabs post-call payload: booked when a bookCalendarMeeting tool result succeeded (paired to tool_calls by id, positional fallback), else qualified when the visitor spoke; no signal → no claim. Also: the ms-comm calls list takes server-side start_date / end_date ($or over started_at and created_at when started_at is null) so the tab count and the list agree. Phone outcomes are upgrade-only (a late analysis may promote qualified → voicemail/booked, never downgrade). Outbound voicemail = ElevenLabs analysis flag OR AMD verdict (see Voicemail detection) OR transcript heuristic → record field voicemail. GHL tagging runs in the backend (POST /internal/ai-call/contact-extracted applies the agent's post_call_tags; inbound built-ins are booked/qualified only), idempotent via ms-comm's post_call_tags_applied flag on the call record.
  6. GHL recordings + notes are cron-driven in the backend: ms-comm stamps ghl_recording_state='pending'; GhlCallRecordingAttachJob (every 5 min) consumes it and attaches the recording to the GHL contact; GhlCallNotePostJob (every 1 min) posts call notes, deferring inbound notes until the summary lands. Shared libs backend/app/lib/callRecordingUrl.js / callNoteBody.js / ghlErrors.js. ms-comm GET /inbound-outbound-calls/:id falls through to a call_sid lookup when :id isn't a Mongo id.

Human escalation​

Bridge tool escalateToHuman (ElevenLabsBridgeService._handleEscalateToHuman; the widget websocket path still registers the legacy esclateToHuman spelling):

  • Destination = the agent's transfer_number read live from Mongo (default agent on ai_inbound_call, others in ai_inbound_call_agents[]); blank → company.phone_number.
  • Mixed accounts: credentials resolve per dialed number via ms-ai GET /elevenlabs/inbound-company-phone/:company_id?number= — BYO-imported and platform-bought (subaccount) numbers each transfer with their own creds (read from the phone_number owner). For a number shared between an inbound agent and an outbound/SMS agent the lookup prefers the inbound binding (d1a61931e) — it used to take the newest row, which had no transfer line, so escalation came back empty.
  • Caller-ID pass-through: inbound transfers present the original caller's number; on carrier reject (Twilio 13223/21212, or Telnyx) retry once with the owned number. Outbound always presents the owned number.
  • Twilio: Calls.update with <Dial timeout="30" answerOnBridge="true">; Telnyx: TeXML call-update REST (POST /v2/texml/calls/:sid/update). Both pause ~1.2s so the "transferring you now" TTS finishes.

Call Agent · Voice Engine (S2S) · Inbound Call Agent · Outbound Call Agent · SMS Agent · Calls · AI Credits