Skip to main content

Audio & Video Calls

Audience: Sales · Support · QA · Management · Engineering · Where in app: Calls (recordings, AI calls, phone transcripts) · configured under Settings → Calls/Preferences and Settings → AI → Agents · Plan availability: All plans (AI voice, inbound, and outbound phone may be plan- or add-on-gated — verify)

Knock Knock supports several flavors of voice (and video) calling that share infrastructure but use different providers under the hood. A visitor can talk to a human over Twilio WebRTC in the browser, talk to an AI voice agent (ElevenLabs over LiveKit WebRTC, or Ultravox), or call/be-called over the regular phone network (PSTN — carriers Twilio and Telnyx) where an AI agent answers via an ElevenLabs media-stream bridge. Humans can take over an AI call mid-conversation or silently monitor it for QA. Everything is logged, most things are recorded or transcribed. Carrier/number/wallet details: Telephony subsystem.

What it does​

  • Browser calls (WebRTC): audio or video between a visitor and a human agent, via Twilio Video. Always recorded.
  • AI voice calls: the AI answers the visitor by voice (ElevenLabs signed-URL or Ultravox), can book meetings and look up products mid-call.
  • Inbound phone calls: a visitor dials the company's number; an AI agent answers over PSTN through the ElevenLabs bridge.
  • Outbound phone calls: the company (via ms-ai) dials a lead; the same AI bridge runs the conversation.
  • Takeover & monitor: a human can jump into an in-progress AI call (takeover) or listen silently (monitor) for QA.
  • Captures recordings (WebRTC) and turn-by-turn transcripts (AI calls) and surfaces them in Calls.

How it works​

The three flavors at a glance​

FlavorProviderTriggered when
Human ↔ Visitor (audio/video)Twilio Video (WebRTC)call_routing = human and an agent is online
AI ↔ Visitor (audio)ElevenLabs Conversational AI over LiveKit WebRTC (calling_type: 'webrtc') or Ultravoxcall_routing = ai, or fallback when no human
Inbound / Outbound phoneTwilio or Telnyx PSTN + ElevenLabs media-stream bridgeVisitor dials the company number (inbound) / ms-ai dials a lead (outbound)

Twilio WebRTC lifecycle (human ↔ visitor)​

  1. Visitor clicks call. If take_phone_number_before_call is set, a phone number is captured first (stored in the widget's user_profile); contact-info questions can also run.
  2. The widget creates a service room (POST /service-room/create) and emits twilio-webrtc-call. The server (handleTwilio) mints Twilio tokens and creates a Twilio Video room named after the service_room_id, with recordParticipantsOnConnect: true.
  3. The widget retries new-call-request every ~5s; available agents get an "Incoming call". The desktop app pops a dedicated call window and rings (VoIP push).
  4. First agent to accept (call-accepted-by-agent) joins the Twilio room; the room is marked is_started: true and Firestore status flips to accepted. Call begins.
  5. On hangup, the call event is saved and the live session updated (update-live-session).
  6. Recordings are participant recordings; the merged video is produced asynchronously (see ms-sessions, RoomVideoMerging job).

Ringtones in the widget: before-phone.mp3 (connecting, loops) → on-call-ringing.mp3 (agent available, loops) → drop-call.mp3 / end-call.mp3 (one-shot on end/decline). Total ring window is ~35s before falling back.

AI voice lifecycle (ElevenLabs / Ultravox, in-browser)​

  1. Visitor clicks call; contact info captured as available. A ringback loop (on-call-ringing.mp3, started inside the click gesture in widget/src/stores/widget.ts) plays until the agent picks up.
  2. ElevenLabs path (WebRTC / LiveKit): widget emits get-singed-url; ms-communication (CallsController, calling_type: 'webrtc' hardcoded) calls ms-ai start-audio-call and returns elab-signed-url carrying a LiveKit conversation token + connection_type and previous_conversation memory (widget waits 30s, was 15s). The widget starts the ElevenLabs SDK with { conversationToken, connectionType: 'webrtc' }; the legacy websocket transport (signedUrl) still works when connection_type: 'websocket'. Dynamic variables (company_id, user_session_id, webhook_url, timezones, calendar type) and client tools (askUserDetails, saveUserProfile, esclateToHuman — legacy spelling on this path) pass through unchanged; transcripts/typed messages/tool responses arrive via SDK callbacks instead of the raw socket.
  3. Ultravox path: widget emits connect-call-with-ai-agent / get-audio-call-token; receives ai-agent-joining-data (20s timeout) and joins via the Ultravox client SDK.
  4. Tool calls (book a meeting, look up a product) route through ms-ai's voice tools. Product results come back over events-gateway (ElevenLabs) / voice-ai-tools-gateway (Ultravox) and the widget displays matched products. Booking confirmation is delivered to the agent as a userMsg (on LiveKit there is no raw socket to inject context).
  5. On end, the widget emits ai-all-messages (full transcript, collected name/email/phone, booking detail). The transcript is captured for the call record.

Max AI call duration is 900s (15 min). Inactivity prompts fire at ~10s / 15s / 20s, after which the AI ends the call gracefully.

Inbound phone call lifecycle (PSTN → AI)​

  1. The number's carrier webhook hits POST /twilio/inbound-voice/:company_id (Twilio) or POST /telnyx/inbound-voice/:company_id (Telnyx TeXML, TelnyxInboundCallController) with agent_id / agent_key.
  2. Wallet gate: for app-bought numbers, ms-comm asks ms-ai GET /elevenlabs/telephony-inbound-allowed/:company_id?number= (4s timeout, fail-open); wallet-empty callers hear "This number is temporarily unavailable. Goodbye." BYO numbers are never gated.
  3. Call screening (Twilio only, opt-in): if agent.call_screening.enabled, ms-comm answers with a <Gather> and asks for a fresh random digit (Twilio TTS, free leg) before any forwarding leg or ElevenLabs stream. Correct keypress re-enters via POST /twilio/inbound-screen/:company_id (?screened=1, digit rides ?d= — stateless); wrong/no digit logs to screened_calls (TTL 90d, deliberately not inbound_outbound_calls) and hangs up. Preheat runs during the Gather; fail-open. See Telephony › Inbound call screening.
  4. Routing priorities: if the agent has routing_priorities, ms-comm rings those numbers in order (each with its own ring_timeout, chain advanced via POST /twilio/inbound-dial-fallback/:company_id) regardless of working hours; the first to answer takes the call, exhausted/error falls through to the AI.
  5. If inbound working hours are configured and it's within hours, the call is transferred to the human transfer_number instead of the AI.
  6. Otherwise the bridge "preheats" (signed URL, agent config, company context, visitor context) and returns TwiML/TeXML (<Connect><Stream>) pointing at the bridge WebSocket (/twilio/media-stream or /telnyx/media-stream — the path selects the carrier codec handling). Agents with continue_previous_conversation get a per-call first message generated from the caller's last transcript (ms-ai POST /agents/generate-first-message via FirstMessageHelper, budget FIRST_MESSAGE_INIT_CAP_MS=2000ms).
  7. ElevenLabsBridgeService opens two WebSockets — carrier (mu-law 8kHz; Telnyx A-law legs are normalized by AudioTranscodeHelper) and ElevenLabs (PCM 16kHz) — transcodes audio both ways, and fans the audio out to the ai-listen:{service_room_id} room for supervisors (carrier-agnostic).
  8. Turn-by-turn transcript is appended to InboundOutboundCallModel.transcript on every user_transcript / agent_response. On completion the record is finalized; the transcript email and post-call webhook follow (see Post-call below).

Inbound config lives on companies.ai_inbound_call (single agent) or companies.ai_inbound_call_agents[] (multi-agent, keyed by agent_key): agent_id, agent_key, name, transfer_number, working_hours, post_call_webhook_url / _header. Caller-ID identity verification fires an identity-graph lookup (/identity-graph/markdown-by-phone, /identity-graph/inbound-call-verify) so the AI greets a known caller with context.

Outbound phone call lifecycle (AI → lead)​

  1. ms-ai places the call (ms-communication does not dial directly); the number's provider picks the carrier (Twilio REST or Telnyx TeXML create-call). Every dial carries async AMD params (_appendAmdParams: MachineDetection=Enable, AsyncAmd=true; env AMD_ENABLED default true). Before dialing, ms-ai hits POST /twilio/preheat-outbound/:company_id/:agent_id (or /telnyx/preheat-outbound/...) to warm caches.
  2. When the customer answers, the carrier hits POST /twilio/outbound-voice/:company_id / POST /telnyx/outbound-voice/:company_id, returning the same media-stream bridge TwiML/TeXML (purpose: 'outbound'). On answer the bridge fire-and-forgets POST /call-events/live to backend (CallEventsController.live, apiKeyAuth) — carrier-agnostic, both directions — which dedups on Redis call-events:live:<service_room_id> for 15 min and fires the governed ai-call-live push to every admin device.
  3. The AMD verdict lands async on POST /twilio/amd-status/:company_id / POST /telnyx/amd-status/:company_id (always 204); ElevenLabsBridgeService.handleMachineDetected flags voicemail on machine*/fax only ($max-sticky) — see Telephony › Voicemail detection.
  4. Same transcoding/transcript/fanout as inbound; on completion the agent's post_call_webhook_url receives the full call metadata + transcript (see below).

Post-call pipeline​

The post-call webhook is held until the ElevenLabs analysis arrives (it carries summary, sentiment, recording), single-fired via an atomic claim on post_call_webhook_sent_at. Backend PostCallWebhookFallbackJob (every 1 min, 5-min grace, 24h max age, 5 attempts) fires calls whose analysis never lands — the claim guarantees exactly-once. Payload (PostCallWebhookService, namespaced next_level_call): agent_key, transcript[], transcript_text, summary, call_sentiment (positive|neutral|negative, classified from the caller's words via ms-ai POST /agents/classify-call-sentiment), signed recording_url (HMAC, ~365d, RECORDING_LINK_SECRET, fail-closed, served by backend GET /public/recordings/:id on api.nextlevelai.site). Outbound voicemail detection (analysis flag OR transcript heuristic → record voicemail) and call_outcome (booked|qualified|voicemail|no_answer|no_outcome, upgrade-only) drive GHL outcome tagging via backend /internal/ai-call/contact-extracted. Full detail: Telephony subsystem.

Outbound config: companies.ai_outbound_call_enabled (feature on), ai_outbound_call_allowed (admin gate / allowed initiators), ai_outbound_call_agents[] (per-agent: agent_key, name, webhook). Knox can place calls via its knox-place-call tool when granted.

Configuration & options​

Routing modes (companies.ai_config.call_routing)​

ModeBehavior
aiAI voice handles every call
humanCalls go to humans; if none online, fall back to missed-call / offline message
ai_to_humanAI starts; takeover transfers to a human when triggered. The escalation tool is only offered when AGENTS_STATUS == 'AVAILABLE' — named escalateToHuman on the PSTN bridge, legacy esclateToHuman in the widget path. See Telephony › Human escalation for creds/caller-ID behavior

Provider & call settings​

SettingPurpose
ai_config.call_model_providerSelects the AI tech (elevenlabs, fixie-ai/ultravox, openai-realtime) — verify exact field/values per tenant; some live in ms-ai/company schema
take_phone_number_before_callCapture the visitor's phone before starting a call
allow_twilio_callsEnable phone-call fallback
always_twilio_callsForce the Twilio path even when AI voice is available (compliance/recording)
company_phone_numberNumber used for phone fallback
ai_inbound_call.* / ai_inbound_call_agents[]Inbound AI agent(s), transfer number, working hours, post-call webhook
ai_outbound_call_enabled / ai_outbound_call_allowed / ai_outbound_call_agents[]Outbound calling gates and agents
communication_options.take_over_permissionsWhether agents may take over AI calls

Calls dashboard tabs​

The Calls screen was rewritten (2026-08-22 → 09-09) as a three-pane rail / thread / contact layout (CallsView.vue + components/calls/{CallsRail,CallThread,CallContact,CallSheet,CallIcon,ScheduleCallDialog}.vue). useCallsFeed.js stitches one feed from getAICallRecordings, getAIPhoneTranscripts, getCallRecordings, and getForwardedCalls; there is no client-side paging — "load more" re-fetches every source with a larger limit so one sort over the union stays correct, and every query stays enabled regardless of tab because each tab shows a live count from its endpoint's total. CALL_TABS = ['widget', 'in', 'out', 'scheduled'] — all is deliberately not navigable (still valid for feedFor() on caller history and deep links). The Scheduled tab is served by the backend action queue, not the feed (ScheduledCallsPanel.vue was deleted); its rows reuse the out source so chips and agent grouping read correctly. Normalizers live in utils/callsFeed.js. Thread extras: download menu (MP3/video + transcript text), a single More information entry into the call info card, copy-link, duration falling back to the player's length, player reset on tab change. The phone transcripts query now sends start_date/end_date to ms-comm and range joins the query key — the tab badge used the server's all-time total above a client-filtered list. ScheduleCallDialog maps action kinds to scheduleNoxCall({kind, payload, ghlContactId}): ai_call (sent as kind: undefined, the backend default), sms, email, note, reminder, stage_move, enrich, one Zod schema per kind; the allKinds prop gates everything beyond calls (the Nox Scheduled screen).

Behaviors & edge cases​

  • Takeover (human joins an AI call). Socket ai-call-takeover-request ends the AI service room (ended_by: 'agent_takeover'), clears is_ai_call, creates a new service room (takeover_from_ai: true) with fresh Twilio tokens, and tells the widget (ai-call-takeover, auto-accepts) and the dashboard (twilio-webrtc-call-response). Gated by take_over_permissions. Inbound PSTN has its own takeover_inbound_call path where the agent's mic audio is injected into the live Twilio call via the bridge's pub/sub.
  • Monitor (silent listen for QA). Socket monitor_call (browser AI) / monitor_inbound_call (PSTN) joins the ai-listen:{service_room_id} room — carrier-agnostic (Twilio and Telnyx bridges feed the same room). Access is verified: the session's call must be active and AI, and the agent must belong to the company. The widget streams PCM audio_chunks (base64) only while a listener is present (request_audio_stream / stop_audio_stream); on WebRTC calls the widget taps the agent's LiveKit audio track to build that stream. Listener counts and streaming flags are tracked in Redis (~15-min TTL). The monitor confirms streaming within ~5s.
  • Permission fallbacks (WebRTC). Mic granted + camera denied → audio-only with cameraDenied. Mic denied → listener mode (micDenied): the visitor can see/hear the agent but cannot broadcast; they can publish tracks later. iOS Safari background suspension is detected (track ended) and the widget attempts to restart the video track.
  • Missed call. If a user-initiated call isn't answered before the ring window, it's logged (loki.logTimeout) and a missed-call email is sent (sendMissedCallEmail). A separate AI voice missed-call email fires when an inbound AI call ends with no action (e.g. no booking).
  • Recording. WebRTC: always on (recordParticipantsOnConnect: true), merged async by ms-sessions. AI in-browser: transcript captured (audio recording optional via ElevenLabs). PSTN: the bridge does not record; audio streams on demand from ElevenLabs via signed links (RecordingLinkService, HMAC ~365d, fail-closed) — never stored. always_twilio_calls forces the recorded Twilio path.
  • Stale-call sweep. In-progress phone calls with no update past a ~5-min cutoff are swept to failed so the dashboard doesn't show phantom live calls.
  • Timeouts (engineering reference). Max session duration is config-driven; signed-URL / WebRTC-token cache ~45s (widget waits 30s for elab-signed-url); agent-config cache ~10min; company-context cache ~2min; visitor-context init cap ~1.2s (inbound prewarm cap ~11s); transfer dial timeout 30s; routing-priority ring timeouts 5–120s (default 20s); AI max duration 900s.
  • Call rate limit: 3 points / 60s per connection (much stricter than chat) to prevent call spam.

Plan & limits​

  • Browser WebRTC human calls and basic AI voice are documented as available on all plans.
  • Inbound/outbound AI phone calling, provisioned phone numbers, and some recording/compliance features are likely add-on or higher-tier — verify against the current plan matrix before quoting.
  • Hard limits in code: AI calls cap at 900s; ring window ~35s; transfer dial 30s. There is no documented per-tenant monthly call cap here (billing/credits govern AI minutes) — verify.

Technical implementation​

  • Owning service: ms-communication.
    • app/controllers/CallsController.js — Twilio WebRTC, AI-call setup (WebRTC token / signed URL / Ultravox), takeover, monitor, missed-call email.
    • app/controllers/InboundCallController.js, OutboundCallController.js, InboundOutboundCallController.js — Twilio PSTN webhooks, routing priorities, wallet gate, call screening (POST /twilio/inbound-screen/:company_id, ScreenedCallModel), AMD callbacks (POST /twilio/amd-status/:company_id; Telnyx twin on TelnyxOutboundCallController), preheat, transcript persistence (show falls through to call_sid lookup), sentiment/voicemail/outcome classification.
    • app/helpers/FirstMessageHelper.js — continuation first-message call to ms-ai POST /agents/generate-first-message (20s timeout).
    • app/controllers/TelnyxInboundCallController.js, TelnyxOutboundCallController.js + app/helpers/TelnyxHelper.js, AudioTranscodeHelper.js — Telnyx TeXML surface and A-law transcoding.
    • app/services/ElevenLabsBridgeService.js — the dual-WebSocket media-stream bridge (transcode, tool calls incl. escalateToHuman, transcript, takeover/monitor fanout).
    • app/services/PostCallWebhookService.js, RecordingLinkService.js — post-call webhook payload + signed recording links (backend holds the twin copy + fallback job PostCallWebhookFallbackJob).
  • Widget: widget — src/services/call.ts, twilio.ts (Twilio Video/Client SDK, listener mode, screen share, camera switch), aiCall.ts (ElevenLabs/Ultravox sessions, live-listen audio capture), sound.ts, socket.ts.
  • Agent UI: frontend — src/views/CallsView.vue, ChatsView.vue (one-click call from a chat).
  • Recording merge: ms-sessions — RoomVideoMerging job; see Session Recording.
  • AI voice tools / transcripts: ms-ai; Identity Graph (caller-ID verification, visitor_context).