Skip to main content

Audio & Video Calls

Audience: Sales · Support · QA · Management · Engineering · Where in app: Calls (recordings, AI calls, phone transcripts) · configured under Settings → Calls/Preferences and Settings → AI → Agents · Plan availability: All plans (AI voice, inbound, and outbound phone may be plan- or add-on-gated — verify)

Knock Knock supports several flavors of voice (and video) calling that share infrastructure but use different providers under the hood. A visitor can talk to a human over Twilio WebRTC in the browser, talk to an AI voice agent (ElevenLabs over LiveKit WebRTC, or Ultravox), or call/be-called over the regular phone network (PSTN — carriers Twilio and Telnyx) where an AI agent answers via an ElevenLabs media-stream bridge. Humans can take over an AI call mid-conversation or silently monitor it for QA. Everything is logged, most things are recorded or transcribed. Carrier/number/wallet details: Telephony subsystem.

What it does​

  • Browser calls (WebRTC): audio or video between a visitor and a human agent, via Twilio Video. Always recorded.
  • AI voice calls: the AI answers the visitor by voice (ElevenLabs signed-URL or Ultravox), can book meetings and look up products mid-call.
  • Inbound phone calls: a visitor dials the company's number; an AI agent answers over PSTN through the ElevenLabs bridge.
  • Outbound phone calls: the company (via ms-ai) dials a lead; the same AI bridge runs the conversation.
  • Takeover & monitor: a human can jump into an in-progress AI call (takeover) or listen silently (monitor) for QA.
  • Captures recordings (WebRTC) and turn-by-turn transcripts (AI calls) and surfaces them in Calls.

How it works​

The three flavors at a glance​

FlavorProviderTriggered when
Human ↔ Visitor (audio/video)Twilio Video (WebRTC)call_routing = human and an agent is online
AI ↔ Visitor (audio)ElevenLabs Conversational AI over LiveKit WebRTC (calling_type: 'webrtc') or Ultravoxcall_routing = ai, or fallback when no human
Inbound / Outbound phoneTwilio or Telnyx PSTN + ElevenLabs media-stream bridgeVisitor dials the company number (inbound) / ms-ai dials a lead (outbound)

Twilio WebRTC lifecycle (human ↔ visitor)​

  1. Visitor clicks call. If take_phone_number_before_call is set, a phone number is captured first (stored in the widget's user_profile); contact-info questions can also run.
  2. The widget creates a service room (POST /service-room/create) and emits twilio-webrtc-call. The server (handleTwilio) mints Twilio tokens and creates a Twilio Video room named after the service_room_id, with recordParticipantsOnConnect: true.
  3. The widget retries new-call-request every ~5s; available agents get an "Incoming call". The desktop app pops a dedicated call window and rings (VoIP push).
  4. First agent to accept (call-accepted-by-agent) joins the Twilio room; the room is marked is_started: true and Firestore status flips to accepted. Call begins.
  5. On hangup, the call event is saved and the live session updated (update-live-session).
  6. Recordings are participant recordings; the merged video is produced asynchronously (see ms-sessions, RoomVideoMerging job).

Agent → visitor calls (dashboard-initiated, 2026-10-06). The dashboard CallPanel.vue shows a derived status — Calling... → Calling again (n)... → Ringing... (visitor's widget joined the room, remote_joined) → Connected (call-accepted-by-user), or Reconnecting... / Visitor not reachable (red). If the widget hasn't joined the Twilio room after 8 / 16 / 24 s (socket reconnecting, tab asleep) it emits twilio-webrtc-call-ring-again; ms-comm (89cff1d) re-sends twilio-webrtc-call-response (is_ring_again: true) to the visitor only, for the same room, with a visitor token minted server-side — ignored once the room is_started, the call is no longer active, or it's an AI call. At 34 s with no join the panel shows Visitor not reachable. Other team members see the session row as "agent calling · Ringing..." (sessions store ringingAgentCalls) until it connects ("On call with agent"). Widget side, Accept now awaits the in-flight listener-mode auto-connect (waitForIncomingAutoConnect, widget 9da09b1) — a second join as user used to make Twilio evict the connection carrying the visitor's camera/mic.

Per-call step log (2026-10-06). Every service room (AI or human, web or phone) can carry call_logs (backend ServiceRoomModel, select: false, excluded from the Calls aggregates). Written only by ms-comm app/helpers/CallLogHelper.js: server LokiService.log(service_room_id, …) lifecycle steps, call socket events naming a room (SocketService middleware), and client steps sent with the call-log socket event by the widget (f596455) and dashboard (frontend/src/utils/callLog.js). Entries { event, message, level (debug/info/warn/error), source (server/socket/widget/dashboard), data, at }; secrets (token|secret|password|authorization|api_key keys) stripped, data capped at 4000 chars, client clocks trusted within ±5 min; batched one $push per room per second, capped at the last 1000. Read via the super-admin Call logs screen — see Media & Logs (Call Logs screen).

Ringtones in the widget: before-phone.mp3 (connecting, loops) → on-call-ringing.mp3 (agent available, loops) → drop-call.mp3 / end-call.mp3 (one-shot on end/decline). Total ring window is ~35s before falling back.

AI voice lifecycle (ElevenLabs / Ultravox, in-browser)​

  1. Visitor clicks call; contact info captured as available. A ringback loop (on-call-ringing.mp3, started inside the click gesture in widget/src/stores/widget.ts) plays until the agent picks up.
  2. ElevenLabs path (WebRTC / LiveKit): widget emits get-singed-url; ms-communication (CallsController, calling_type: 'webrtc' hardcoded) calls ms-ai start-audio-call and returns elab-signed-url carrying a LiveKit conversation token + connection_type and previous_conversation memory (widget waits 30s, was 15s). The widget starts the ElevenLabs SDK with { conversationToken, connectionType: 'webrtc' }; the legacy websocket transport (signedUrl) still works when connection_type: 'websocket'. Dynamic variables (company_id, user_session_id, webhook_url, timezones, calendar type) and client tools (askUserDetails, saveUserProfile, esclateToHuman — legacy spelling on this path) pass through unchanged; transcripts/typed messages/tool responses arrive via SDK callbacks instead of the raw socket.
  3. Ultravox path: widget emits connect-call-with-ai-agent / get-audio-call-token; receives ai-agent-joining-data (20s timeout) and joins via the Ultravox client SDK.
  4. Tool calls (book a meeting, look up a product) route through ms-ai's voice tools. Product results come back over events-gateway (ElevenLabs) / voice-ai-tools-gateway (Ultravox) and the widget displays matched products. Booking confirmation is delivered to the agent as a userMsg (on LiveKit there is no raw socket to inject context).
  5. On end, the widget emits ai-all-messages (full transcript, collected name/email/phone, booking detail). The transcript is captured for the call record.

Max AI call duration is 900s (15 min). Inactivity prompts fire at ~10s / 15s / 20s, after which the AI ends the call gracefully.

Inbound phone call lifecycle (PSTN → AI)​

  1. The number's carrier webhook hits POST /twilio/inbound-voice/:company_id (Twilio) or POST /telnyx/inbound-voice/:company_id (Telnyx TeXML, TelnyxInboundCallController) with agent_id / agent_key.
  2. Wallet gate: for app-bought numbers, ms-comm asks ms-ai GET /elevenlabs/telephony-inbound-allowed/:company_id?number= (4s timeout, fail-open); wallet-empty callers hear "This number is temporarily unavailable. Goodbye." BYO numbers are never gated.
  3. Call screening (Twilio only, opt-in): if agent.call_screening.enabled, ms-comm answers with a <Gather> and asks for a fresh random digit (Twilio TTS, free leg) before any forwarding leg or ElevenLabs stream. Correct keypress re-enters via POST /twilio/inbound-screen/:company_id (?screened=1, digit rides ?d= — stateless); wrong/no digit logs to screened_calls (TTL 90d, deliberately not inbound_outbound_calls) and hangs up. Preheat runs during the Gather; fail-open. See Telephony › Inbound call screening.
  4. Routing priorities: if the agent has routing_priorities, ms-comm rings those numbers in order (each with its own ring_timeout, chain advanced via POST /twilio/inbound-dial-fallback/:company_id) regardless of working hours; the first to answer takes the call, exhausted/error falls through to the AI.
  5. If inbound working hours are configured and it's within hours, the call is transferred to the human transfer_number instead of the AI.
  6. Otherwise the bridge "preheats" (signed URL, agent config, company context, visitor context) and returns TwiML/TeXML (<Connect><Stream>) pointing at the bridge WebSocket (/twilio/media-stream or /telnyx/media-stream — the path selects the carrier codec handling). Agents with continue_previous_conversation get a per-call first message generated from the caller's last transcript (ms-ai POST /agents/generate-first-message via FirstMessageHelper, budget FIRST_MESSAGE_INIT_CAP_MS=2000ms).
  7. ElevenLabsBridgeService opens two WebSockets — carrier (mu-law 8kHz; Telnyx A-law legs are normalized by AudioTranscodeHelper) and ElevenLabs (PCM 16kHz) — transcodes audio both ways, and fans the audio out to the ai-listen:{service_room_id} room for supervisors (carrier-agnostic).
  8. Turn-by-turn transcript is appended to InboundOutboundCallModel.transcript on every user_transcript / agent_response. On completion the record is finalized; the transcript email and post-call webhook follow (see Post-call below).

Inbound config lives on companies.ai_inbound_call (single agent) or companies.ai_inbound_call_agents[] (multi-agent, keyed by agent_key): agent_id, agent_key, name, transfer_number, working_hours, post_call_webhook_url / _header. Caller-ID identity verification fires an identity-graph lookup (/identity-graph/markdown-by-phone, /identity-graph/inbound-call-verify) so the AI greets a known caller with context.

Outbound phone call lifecycle (AI → lead)​

  1. ms-ai places the call (ms-communication does not dial directly); the number's provider picks the carrier (Twilio REST or Telnyx TeXML create-call). Every dial carries async AMD params (_appendAmdParams: MachineDetection=Enable, AsyncAmd=true; env AMD_ENABLED default true). Before dialing, ms-ai hits POST /twilio/preheat-outbound/:company_id/:agent_id (or /telnyx/preheat-outbound/...) to warm caches.
  2. When the customer answers, the carrier hits POST /twilio/outbound-voice/:company_id / POST /telnyx/outbound-voice/:company_id, returning the same media-stream bridge TwiML/TeXML (purpose: 'outbound'). On answer the bridge fire-and-forgets POST /call-events/live to backend (CallEventsController.live, apiKeyAuth) — carrier-agnostic, both directions — which dedups on Redis call-events:live:<service_room_id> for 15 min and fires the governed ai-call-live push to every admin device.
  3. The AMD verdict lands async on POST /twilio/amd-status/:company_id / POST /telnyx/amd-status/:company_id (always 204); ElevenLabsBridgeService.handleMachineDetected flags voicemail on machine*/fax only ($max-sticky) — see Telephony › Voicemail detection.
  4. Same transcoding/transcript/fanout as inbound; on completion the agent's post_call_webhook_url receives the full call metadata + transcript (see below).

Post-call pipeline​

The post-call webhook is held until the ElevenLabs analysis arrives (it carries summary, sentiment, recording), single-fired via an atomic claim on post_call_webhook_sent_at. Backend PostCallWebhookFallbackJob (every 1 min, 5-min grace, 24h max age, 5 attempts) fires calls whose analysis never lands — the claim guarantees exactly-once. Payload (PostCallWebhookService, namespaced next_level_call): agent_key, transcript[], transcript_text, summary, call_sentiment (positive|neutral|negative, classified from the caller's words via ms-ai POST /agents/classify-call-sentiment), signed recording_url (HMAC, ~365d, RECORDING_LINK_SECRET, fail-closed, served by backend GET /public/recordings/:id on api.nextlevelai.site). Outbound voicemail detection (analysis flag OR transcript heuristic → record voicemail) and call_outcome (booked|qualified|voicemail|no_answer|no_outcome, upgrade-only) drive GHL outcome tagging via backend /internal/ai-call/contact-extracted. Full detail: Telephony subsystem.

Outbound config: companies.ai_outbound_call_enabled (feature on), ai_outbound_call_allowed (admin gate / allowed initiators), ai_outbound_call_agents[] (per-agent: agent_key, name, webhook). Knox can place calls via its knox-place-call tool when granted.

Recording playback​

Since 2026-10-27 (ms-comm ed9b0ae, backend c83d4cde3 / df983134f, frontend dba706f8 → 6f0bdc04) ms-comm no longer streams widget call recordings to anyone holding the SID. Its routes are internal only — GET /internal/recordings/call/:sid (Twilio composition), /internal/recordings/agent/:sid (Ultravox), /internal/recordings/phone/:sid (Grok-over-SIP Twilio recording) — behind the api middleware (x-api-key), with Twilio media URLs cached (3acd9bc). The backend fronts them:

Route (backend)AuthPurpose
GET /recordings/{call,agent}/:sid/play-urllogged in; room must belong to a company the user may accessReturns a /recordings/stream/...?exp&token URL carrying a short-lived (2 h, WIDGET_PLAY_TTL_SECONDS) HMAC token for that recording only (RecordingLink.signToken('widget-recording:<kind>:<sid>')) — never the user's login token — so the <video>/<audio> element streams with Range and starts at once
GET /recordings/stream/{call,agent}/:sidthe token above (410 expired / 403 invalid)Proxies the bytes, Range forwarded; 404 if the room was deleted after issue
GET /recordings/{call,agent}/:sidbearerSame proxy for API callers

The proxy (ServiceRoomController._proxyWidgetRecording) calls ms-comm with MS_COMMUNICATION_URL + MS_COMM_API_KEY (backend config; 503 "Recording playback is not configured" without them) and sends Cache-Control: no-store. SIDs are format-checked (composition CJ… SID, Ultravox UUID). ElevenLabs and voice-engine recordings keep their own authed paths (GET /elab-calls/:id, _authorizeRecording); a Grok SIP call (row with twilio_recording_sid) is routed to the phone kind. Webhook/CRM links stay on the public signed route (GET /public/recordings/:id, RecordingLinkService).

In the Calls thread, Download (MP3/video + transcript) is shown (ALLOW_RECORDING_DOWNLOAD = true) but Copy link to this call is hidden (ALLOW_RECORDING_SHARE = false, dfaafc1a), and the More → Delete call menu renders only for users who may delete (canDeleteConversations: admin, web_owner, agency, or agency_staff with agency.edit_clients / agency.add_client_subscription; fd4cfc77).

Visitor phone-call token (toll-fraud fix)​

The widget's "forward to phone" call is a browser Twilio Voice call. Since ms-comm 411462b / widget 5711ddd (2026-10-27): POST /twilio/token mints a 1 h token whose identity is ours (visitor_<user_session_id> when the widget sends a company_id + user_session_id pair that exists — 403 "Unknown session" otherwise — else a random visitor_…), never a caller-supplied identity; and the TwiML App voice URL /twilio/call is Twilio-signature-checked and dials only a company's own published phone_number with phone calls allowed (_dialableCompany), never an arbitrary To. 9af1c8f lets anonymous tokens still dial.

Configuration & options​

Routing modes (companies.ai_config.call_routing)​

ModeBehavior
aiAI voice handles every call
humanCalls go to humans; if none online, fall back to missed-call / offline message
ai_to_humanAI starts; takeover transfers to a human when triggered. The escalation tool is only offered when AGENTS_STATUS == 'AVAILABLE' — named escalateToHuman on the PSTN bridge, legacy esclateToHuman in the widget path. See Telephony › Human escalation for creds/caller-ID behavior

Provider & call settings​

SettingPurpose
ai_config.call_model_providerSelects the AI tech (elevenlabs, fixie-ai/ultravox, openai-realtime) — verify exact field/values per tenant; some live in ms-ai/company schema
take_phone_number_before_callCapture the visitor's phone before starting a call
allow_twilio_callsEnable phone-call fallback
always_twilio_callsForce the Twilio path even when AI voice is available (compliance/recording)
company_phone_numberNumber used for phone fallback
ai_inbound_call.* / ai_inbound_call_agents[]Inbound AI agent(s), transfer number, working hours, post-call webhook
ai_outbound_call_enabled / ai_outbound_call_allowed / ai_outbound_call_agents[]Outbound calling gates and agents
communication_options.take_over_permissionsWhether agents may take over AI calls

Calls dashboard tabs​

The Calls screen was rewritten (2026-08-22 → 09-09) as a three-pane rail / thread / contact layout (CallsView.vue + components/calls/{CallsRail,CallThread,CallContact,CallSheet,CallIcon,ScheduleCallDialog}.vue). useCallsFeed.js stitches one feed from getAICallRecordings, getAIPhoneTranscripts, getCallRecordings, and getForwardedCalls; there is no client-side paging — "load more" re-fetches every source with a larger limit so one sort over the union stays correct, and every query stays enabled regardless of tab because each tab shows a live count from its endpoint's total. CALL_TABS = ['widget', 'in', 'out', 'scheduled'] — all is deliberately not navigable (still valid for feedFor() on caller history and deep links). The Scheduled tab is served by the backend action queue, not the feed (ScheduledCallsPanel.vue was deleted); its rows reuse the out source so chips and agent grouping read correctly. Normalizers live in utils/callsFeed.js. Thread extras: download menu (MP3/video + transcript text), a single More information entry into the call info card, copy-link (currently hidden — see Recording playback), duration falling back to the player's length, player reset on tab change. The phone transcripts query now sends start_date/end_date to ms-comm and range joins the query key — the tab badge used the server's all-time total above a client-filtered list. ScheduleCallDialog maps action kinds to scheduleNoxCall({kind, payload, ghlContactId}): ai_call (sent as kind: undefined, the backend default), sms, email, note, reminder, stage_move, enrich, one Zod schema per kind; the allKinds prop gates everything beyond calls (the Nox Scheduled screen).

Agent names and lead names (2026-09). Scheduled calls carry only payload.agent_key; CallsView.vue resolves names from the outbound-agents list (listOutboundAgents, query key ['ai-outbound-agents'], shared with ScheduleCallDialog) into normalizeScheduledCall(r, agentNames) / buildAgentRoster — the key is never shown ("Outbound agent" while loading; the detail row is Agent, formerly Agent key). Open agent settings in the agent CallSheet deep-links to that agent (?agent=outbound-call-agent&agent_key=…, inbound likewise) instead of the agent-type grid (d09f2e16). ms-communication b08c7fc attaches lead { name, company, lead_action_id } to outbound rows on GET list/show via InboundOutboundCallController._leadsFor (read-only ScheduledNoxActionModel lookup by call_sid; payload.full_name, or person_name for standup tasks); normalizePhoneCall titles the row by the extracted caller name, then lead.name/lead.company, then the number, and CallThread.vue renders an "Agent K lead →" chip when lead_action_id is set. The dialog's sender label reads "Company LeadConnector number (exact text only)". Scheduled tasks now fire at their exact time — see Nox scheduled-action queue.

Behaviors & edge cases​

  • Takeover (human joins an AI call). Socket ai-call-takeover-request ends the AI service room (ended_by: 'agent_takeover'), clears is_ai_call, creates a new service room (takeover_from_ai: true) with fresh Twilio tokens, and tells the widget (ai-call-takeover, auto-accepts) and the dashboard (twilio-webrtc-call-response). Gated by take_over_permissions. Inbound PSTN has its own takeover_inbound_call path where the agent's mic audio is injected into the live Twilio call via the bridge's pub/sub. Speech-to-speech calls can be taken over too (phone: every vendor — the vendor session ends with agent_takeover, the carrier leg stays; web: OpenAI/GPT-Live and Gemini — the widget ends the direct session via endAICall); xAI web calls work like Gemini (relayed through ms-comm); Vapi calls are refused s2s_not_supported, and Grok-over-SIP phone calls can't be monitored or taken over (the audio never reaches ms-comm). See Voice Engine › Operational limits.
  • Monitor (silent listen for QA). Socket monitor_call (browser AI) / monitor_inbound_call (PSTN) joins the ai-listen:{service_room_id} room — carrier-agnostic (Twilio and Telnyx bridges feed the same room). Access is verified: the session's call must be active and AI, and the agent must belong to the company. The widget streams PCM audio_chunks (base64) only while a listener is present (request_audio_stream / stop_audio_stream); on WebRTC calls the widget taps the agent's LiveKit audio track to build that stream; on direct S2S web calls it taps the session's own audio graph (mic + agent) and streams 16 kHz PCM. Listener counts and streaming flags are tracked in Redis (~15-min TTL). The monitor confirms streaming within ~5s.
  • Permission fallbacks (WebRTC). Mic granted + camera denied → audio-only with cameraDenied. Mic denied → listener mode (micDenied): the visitor can see/hear the agent but cannot broadcast; they can publish tracks later. iOS Safari background suspension is detected (track ended) and the widget attempts to restart the video track.
  • Missed call. If a user-initiated call isn't answered before the ring window, it's logged (loki.logTimeout) and a missed-call email is sent (sendMissedCallEmail). A separate AI voice missed-call email fires when an inbound AI call ends with no action (e.g. no booking).
  • Recording. WebRTC: always on (recordParticipantsOnConnect: true), merged async by ms-sessions. AI in-browser: transcript captured (audio recording optional via ElevenLabs). PSTN: the bridge does not record; audio streams on demand from ElevenLabs via signed links (RecordingLinkService, HMAC ~365d, fail-closed) — never stored. Exceptions: direct S2S calls are recorded by the peer into S3, and Grok-over-SIP calls are recorded by Twilio — see Voice Engine › Recording. Dashboard playback goes through the backend proxy (Recording playback). always_twilio_calls forces the recorded Twilio path.
  • Stale-call sweep. In-progress phone calls with no update past a ~5-min cutoff are swept to failed so the dashboard doesn't show phantom live calls.
  • Timeouts (engineering reference). Max session duration is config-driven; signed-URL / WebRTC-token cache ~45s (widget waits 30s for elab-signed-url); agent-config cache ~10min; company-context cache ~2min; visitor-context init cap ~1.2s (inbound prewarm cap ~11s); transfer dial timeout 30s; routing-priority ring timeouts 5–120s (default 20s); AI max duration 900s.
  • Call rate limit: 3 points / 60s per connection (much stricter than chat) to prevent call spam.

Plan & limits​

  • Browser WebRTC human calls and basic AI voice are documented as available on all plans.
  • Inbound/outbound AI phone calling, provisioned phone numbers, and some recording/compliance features are likely add-on or higher-tier — verify against the current plan matrix before quoting.
  • Hard limits in code: AI calls cap at 900s; ring window ~35s; transfer dial 30s. There is no documented per-tenant monthly call cap here (billing/credits govern AI minutes) — verify.

Technical implementation​

  • Owning service: ms-communication.
    • app/controllers/CallsController.js — Twilio WebRTC, AI-call setup (WebRTC token / signed URL / Ultravox), takeover, monitor, missed-call email.
    • app/controllers/InboundCallController.js, OutboundCallController.js, InboundOutboundCallController.js — Twilio PSTN webhooks, routing priorities, wallet gate, call screening (POST /twilio/inbound-screen/:company_id, ScreenedCallModel), AMD callbacks (POST /twilio/amd-status/:company_id; Telnyx twin on TelnyxOutboundCallController), preheat, transcript persistence (show falls through to call_sid lookup), sentiment/voicemail/outcome classification.
    • app/helpers/FirstMessageHelper.js — continuation first-message call to ms-ai POST /agents/generate-first-message (20s timeout).
    • app/controllers/TelnyxInboundCallController.js, TelnyxOutboundCallController.js + app/helpers/TelnyxHelper.js, AudioTranscodeHelper.js — Telnyx TeXML surface and A-law transcoding.
    • app/services/ElevenLabsBridgeService.js — the dual-WebSocket media-stream bridge (transcode, tool calls incl. escalateToHuman, transcript, takeover/monitor fanout).
    • app/services/PostCallWebhookService.js, RecordingLinkService.js — post-call webhook payload + signed recording links (backend holds the twin copy + fallback job PostCallWebhookFallbackJob).
  • Widget: widget — src/services/call.ts, twilio.ts (Twilio Video/Client SDK, listener mode, screen share, camera switch), aiCall.ts (ElevenLabs/Ultravox sessions, live-listen audio capture), sound.ts, socket.ts.
  • Agent UI: frontend — src/views/CallsView.vue (rail tabs Widget / Inbound / Outbound / Scheduled; booked meetings moved to their own /meetings page, MeetingsView.vue, 0814ce65 — a stale ?tab=meetings redirects), ChatsView.vue (one-click call from a chat), components/CallPanel.vue (agent-call status + ring-again).
  • Recording merge: ms-sessions — RoomVideoMerging job; see Session Recording.
  • AI voice tools / transcripts: ms-ai; Identity Graph (caller-ID verification, visitor_context).