Skip to main content

Audio & Video Calls

Audience: Sales · Support · QA · Management · Engineering · Where in app: Calls (recordings, AI calls, phone transcripts) · configured under Settings → Calls/Preferences and Settings → AI → Agents · Plan availability: All plans (AI voice, inbound, and outbound phone may be plan- or add-on-gated — verify)

Knock Knock supports several flavors of voice (and video) calling that share infrastructure but use different providers under the hood. A visitor can talk to a human over Twilio WebRTC in the browser, talk to an AI voice agent (ElevenLabs over LiveKit WebRTC, or Ultravox), or call/be-called over the regular phone network (PSTN — carriers Twilio and Telnyx) where an AI agent answers via an ElevenLabs media-stream bridge. Humans can take over an AI call mid-conversation or silently monitor it for QA. Everything is logged, most things are recorded or transcribed. Carrier/number/wallet details: Telephony subsystem.

What it does​

  • Browser calls (WebRTC): audio or video between a visitor and a human agent, via Twilio Video. Always recorded.
  • AI voice calls: the AI answers the visitor by voice (ElevenLabs signed-URL or Ultravox), can book meetings and look up products mid-call.
  • Inbound phone calls: a visitor dials the company's number; an AI agent answers over PSTN through the ElevenLabs bridge.
  • Outbound phone calls: the company (via ms-ai) dials a lead; the same AI bridge runs the conversation.
  • Takeover & monitor: a human can jump into an in-progress AI call (takeover) or listen silently (monitor) for QA.
  • Captures recordings (WebRTC) and turn-by-turn transcripts (AI calls) and surfaces them in Calls.

How it works​

The three flavors at a glance​

FlavorProviderTriggered when
Human ↔ Visitor (audio/video)Twilio Video (WebRTC)call_routing = human and an agent is online
AI ↔ Visitor (audio)ElevenLabs Conversational AI over LiveKit WebRTC (calling_type: 'webrtc') or Ultravoxcall_routing = ai, or fallback when no human
Inbound / Outbound phoneTwilio or Telnyx PSTN + ElevenLabs media-stream bridgeVisitor dials the company number (inbound) / ms-ai dials a lead (outbound)

Twilio WebRTC lifecycle (human ↔ visitor)​

  1. Visitor clicks call. If take_phone_number_before_call is set, a phone number is captured first (stored in the widget's user_profile); contact-info questions can also run.
  2. The widget creates a service room (POST /service-room/create) and emits twilio-webrtc-call. The server (handleTwilio) mints Twilio tokens and creates a Twilio Video room named after the service_room_id, with recordParticipantsOnConnect: true.
  3. The widget retries new-call-request every ~5s; available agents get an "Incoming call". The desktop app pops a dedicated call window and rings (VoIP push).
  4. First agent to accept (call-accepted-by-agent) joins the Twilio room; the room is marked is_started: true and Firestore status flips to accepted. Call begins.
  5. On hangup, the call event is saved and the live session updated (update-live-session).
  6. Recordings are participant recordings; the merged video is produced asynchronously (see ms-sessions, RoomVideoMerging job).

Ringtones in the widget: before-phone.mp3 (connecting, loops) → on-call-ringing.mp3 (agent available, loops) → drop-call.mp3 / end-call.mp3 (one-shot on end/decline). Total ring window is ~35s before falling back.

AI voice lifecycle (ElevenLabs / Ultravox, in-browser)​

  1. Visitor clicks call; contact info captured as available. A ringback loop (on-call-ringing.mp3, started inside the click gesture in widget/src/stores/widget.ts) plays until the agent picks up.
  2. ElevenLabs path (WebRTC / LiveKit): widget emits get-singed-url; ms-communication (CallsController, calling_type: 'webrtc' hardcoded) calls ms-ai start-audio-call and returns elab-signed-url carrying a LiveKit conversation token + connection_type and previous_conversation memory (widget waits 30s, was 15s). The widget starts the ElevenLabs SDK with { conversationToken, connectionType: 'webrtc' }; the legacy websocket transport (signedUrl) still works when connection_type: 'websocket'. Dynamic variables (company_id, user_session_id, webhook_url, timezones, calendar type) and client tools (askUserDetails, saveUserProfile, esclateToHuman — legacy spelling on this path) pass through unchanged; transcripts/typed messages/tool responses arrive via SDK callbacks instead of the raw socket.
  3. Ultravox path: widget emits connect-call-with-ai-agent / get-audio-call-token; receives ai-agent-joining-data (20s timeout) and joins via the Ultravox client SDK.
  4. Tool calls (book a meeting, look up a product) route through ms-ai's voice tools. Product results come back over events-gateway (ElevenLabs) / voice-ai-tools-gateway (Ultravox) and the widget displays matched products. Booking confirmation is delivered to the agent as a userMsg (on LiveKit there is no raw socket to inject context).
  5. On end, the widget emits ai-all-messages (full transcript, collected name/email/phone, booking detail). The transcript is captured for the call record.

Max AI call duration is 900s (15 min). Inactivity prompts fire at ~10s / 15s / 20s, after which the AI ends the call gracefully.

Inbound phone call lifecycle (PSTN → AI)​

  1. The number's carrier webhook hits POST /twilio/inbound-voice/:company_id (Twilio) or POST /telnyx/inbound-voice/:company_id (Telnyx TeXML, TelnyxInboundCallController) with agent_id / agent_key.
  2. Wallet gate: for app-bought numbers, ms-comm asks ms-ai GET /elevenlabs/telephony-inbound-allowed/:company_id?number= (4s timeout, fail-open); wallet-empty callers hear "This number is temporarily unavailable. Goodbye." BYO numbers are never gated.
  3. Routing priorities: if the agent has routing_priorities, ms-comm rings those numbers in order (each with its own ring_timeout, chain advanced via POST /twilio/inbound-dial-fallback/:company_id) regardless of working hours; the first to answer takes the call, exhausted/error falls through to the AI.
  4. If inbound working hours are configured and it's within hours, the call is transferred to the human transfer_number instead of the AI.
  5. Otherwise the bridge "preheats" (signed URL, agent config, company context, visitor context) and returns TwiML/TeXML (<Connect><Stream>) pointing at the bridge WebSocket (/twilio/media-stream or /telnyx/media-stream — the path selects the carrier codec handling).
  6. ElevenLabsBridgeService opens two WebSockets — carrier (mu-law 8kHz; Telnyx A-law legs are normalized by AudioTranscodeHelper) and ElevenLabs (PCM 16kHz) — transcodes audio both ways, and fans the audio out to the ai-listen:{service_room_id} room for supervisors (carrier-agnostic).
  7. Turn-by-turn transcript is appended to InboundOutboundCallModel.transcript on every user_transcript / agent_response. On completion the record is finalized; the transcript email and post-call webhook follow (see Post-call below).

Inbound config lives on companies.ai_inbound_call (single agent) or companies.ai_inbound_call_agents[] (multi-agent, keyed by agent_key): agent_id, agent_key, name, transfer_number, working_hours, post_call_webhook_url / _header. Caller-ID identity verification fires an identity-graph lookup (/identity-graph/markdown-by-phone, /identity-graph/inbound-call-verify) so the AI greets a known caller with context.

Outbound phone call lifecycle (AI → lead)​

  1. ms-ai places the call (ms-communication does not dial directly); the number's provider picks the carrier (Twilio REST or Telnyx TeXML create-call). Before dialing, ms-ai hits POST /twilio/preheat-outbound/:company_id/:agent_id (or /telnyx/preheat-outbound/...) to warm caches.
  2. When the customer answers, the carrier hits POST /twilio/outbound-voice/:company_id / POST /telnyx/outbound-voice/:company_id, returning the same media-stream bridge TwiML/TeXML (purpose: 'outbound').
  3. Same transcoding/transcript/fanout as inbound; on completion the agent's post_call_webhook_url receives the full call metadata + transcript (see below).

Post-call pipeline​

The post-call webhook is held until the ElevenLabs analysis arrives (it carries summary, sentiment, recording), single-fired via an atomic claim on post_call_webhook_sent_at. Backend PostCallWebhookFallbackJob (every 1 min, 5-min grace, 24h max age, 5 attempts) fires calls whose analysis never lands — the claim guarantees exactly-once. Payload (PostCallWebhookService, namespaced next_level_call): agent_key, transcript[], transcript_text, summary, call_sentiment (positive|neutral|negative, classified from the caller's words via ms-ai POST /agents/classify-call-sentiment), signed recording_url (HMAC, ~365d, RECORDING_LINK_SECRET, fail-closed, served by backend GET /public/recordings/:id on api.nextlevelai.site). Outbound voicemail detection (analysis flag OR transcript heuristic → record voicemail) and call_outcome (booked|qualified|voicemail|no_answer|no_outcome, upgrade-only) drive GHL outcome tagging via backend /internal/ai-call/contact-extracted. Full detail: Telephony subsystem.

Outbound config: companies.ai_outbound_call_enabled (feature on), ai_outbound_call_allowed (admin gate / allowed initiators), ai_outbound_call_agents[] (per-agent: agent_key, name, webhook). Knox can place calls via its knox-place-call tool when granted.

Configuration & options​

Routing modes (companies.ai_config.call_routing)​

ModeBehavior
aiAI voice handles every call
humanCalls go to humans; if none online, fall back to missed-call / offline message
ai_to_humanAI starts; takeover transfers to a human when triggered. The escalation tool is only offered when AGENTS_STATUS == 'AVAILABLE' — named escalateToHuman on the PSTN bridge, legacy esclateToHuman in the widget path. See Telephony › Human escalation for creds/caller-ID behavior

Provider & call settings​

SettingPurpose
ai_config.call_model_providerSelects the AI tech (elevenlabs, fixie-ai/ultravox, openai-realtime) — verify exact field/values per tenant; some live in ms-ai/company schema
take_phone_number_before_callCapture the visitor's phone before starting a call
allow_twilio_callsEnable phone-call fallback
always_twilio_callsForce the Twilio path even when AI voice is available (compliance/recording)
company_phone_numberNumber used for phone fallback
ai_inbound_call.* / ai_inbound_call_agents[]Inbound AI agent(s), transfer number, working hours, post-call webhook
ai_outbound_call_enabled / ai_outbound_call_allowed / ai_outbound_call_agents[]Outbound calling gates and agents
communication_options.take_over_permissionsWhether agents may take over AI calls

Calls dashboard tabs​

The Calls screen (CallsView.vue) has: Call Recordings (WebRTC video/audio, heard/unheard, play/view-session/delete), AI Calls (in-browser AI voice + transcript), Forwarded Calls (phone-fallback recordings), and Inbound/Outbound Calls (AI phone transcripts with direction, caller, duration, status). Live phone calls show a LIVE badge and a Listen button (opens the live-call listener); completed ones show Play. Unheard badges and "Mark all heard" are per-tab; the list polls every ~10s while any live call exists.

Behaviors & edge cases​

  • Takeover (human joins an AI call). Socket ai-call-takeover-request ends the AI service room (ended_by: 'agent_takeover'), clears is_ai_call, creates a new service room (takeover_from_ai: true) with fresh Twilio tokens, and tells the widget (ai-call-takeover, auto-accepts) and the dashboard (twilio-webrtc-call-response). Gated by take_over_permissions. Inbound PSTN has its own takeover_inbound_call path where the agent's mic audio is injected into the live Twilio call via the bridge's pub/sub.
  • Monitor (silent listen for QA). Socket monitor_call (browser AI) / monitor_inbound_call (PSTN) joins the ai-listen:{service_room_id} room — carrier-agnostic (Twilio and Telnyx bridges feed the same room). Access is verified: the session's call must be active and AI, and the agent must belong to the company. The widget streams PCM audio_chunks (base64) only while a listener is present (request_audio_stream / stop_audio_stream); on WebRTC calls the widget taps the agent's LiveKit audio track to build that stream. Listener counts and streaming flags are tracked in Redis (~15-min TTL). The monitor confirms streaming within ~5s.
  • Permission fallbacks (WebRTC). Mic granted + camera denied → audio-only with cameraDenied. Mic denied → listener mode (micDenied): the visitor can see/hear the agent but cannot broadcast; they can publish tracks later. iOS Safari background suspension is detected (track ended) and the widget attempts to restart the video track.
  • Missed call. If a user-initiated call isn't answered before the ring window, it's logged (loki.logTimeout) and a missed-call email is sent (sendMissedCallEmail). A separate AI voice missed-call email fires when an inbound AI call ends with no action (e.g. no booking).
  • Recording. WebRTC: always on (recordParticipantsOnConnect: true), merged async by ms-sessions. AI in-browser: transcript captured (audio recording optional via ElevenLabs). PSTN: the bridge does not record; audio streams on demand from ElevenLabs via signed links (RecordingLinkService, HMAC ~365d, fail-closed) — never stored. always_twilio_calls forces the recorded Twilio path.
  • Stale-call sweep. In-progress phone calls with no update past a ~5-min cutoff are swept to failed so the dashboard doesn't show phantom live calls.
  • Timeouts (engineering reference). Max session duration is config-driven; signed-URL / WebRTC-token cache ~45s (widget waits 30s for elab-signed-url); agent-config cache ~10min; company-context cache ~2min; visitor-context init cap ~1.2s (inbound prewarm cap ~11s); transfer dial timeout 30s; routing-priority ring timeouts 5–120s (default 20s); AI max duration 900s.
  • Call rate limit: 3 points / 60s per connection (much stricter than chat) to prevent call spam.

Plan & limits​

  • Browser WebRTC human calls and basic AI voice are documented as available on all plans.
  • Inbound/outbound AI phone calling, provisioned phone numbers, and some recording/compliance features are likely add-on or higher-tier — verify against the current plan matrix before quoting.
  • Hard limits in code: AI calls cap at 900s; ring window ~35s; transfer dial 30s. There is no documented per-tenant monthly call cap here (billing/credits govern AI minutes) — verify.

Technical implementation​

  • Owning service: ms-communication.
    • app/controllers/CallsController.js — Twilio WebRTC, AI-call setup (WebRTC token / signed URL / Ultravox), takeover, monitor, missed-call email.
    • app/controllers/InboundCallController.js, OutboundCallController.js, InboundOutboundCallController.js — Twilio PSTN webhooks, routing priorities, wallet gate, preheat, transcript persistence, sentiment/voicemail/outcome classification.
    • app/controllers/TelnyxInboundCallController.js, TelnyxOutboundCallController.js + app/helpers/TelnyxHelper.js, AudioTranscodeHelper.js — Telnyx TeXML surface and A-law transcoding.
    • app/services/ElevenLabsBridgeService.js — the dual-WebSocket media-stream bridge (transcode, tool calls incl. escalateToHuman, transcript, takeover/monitor fanout).
    • app/services/PostCallWebhookService.js, RecordingLinkService.js — post-call webhook payload + signed recording links (backend holds the twin copy + fallback job PostCallWebhookFallbackJob).
  • Widget: widget — src/services/call.ts, twilio.ts (Twilio Video/Client SDK, listener mode, screen share, camera switch), aiCall.ts (ElevenLabs/Ultravox sessions, live-listen audio capture), sound.ts, socket.ts.
  • Agent UI: frontend — src/views/CallsView.vue, ChatsView.vue (one-click call from a chat).
  • Recording merge: ms-sessions — RoomVideoMerging job; see Session Recording.
  • AI voice tools / transcripts: ms-ai; Identity Graph (caller-ID verification, visitor_context).