Skip to main content

ms-communication — Chats, Calls & Notifications

Real-time messaging and voice. A Nodevel service on MongoDB, with Firestore as the live message store.

  • Stack: Express/Nodevel, Mongoose/MongoDB, Socket.IO (+ Redis adapter), Firebase/Firestore.
  • Port: 8080.

Responsibilities​

  • Messaging — text chat over Socket.IO + Firestore; Slack thread sync; message persistence.
  • Voice/video calls — Twilio and Telnyx PSTN ↔ ElevenLabs media-stream bridges, WebRTC (LiveKit) token relay for widget AI calls, recording links, inbound/outbound automation, post-call webhooks. See Telephony.
  • Speech-to-speech calls — the same carrier bridge can connect a call straight to GPT-Live (LiveAdapter, own OpenAI key) / Gemini Live / the retired Realtime models instead of ElevenLabs, relays widget S2S sessions (WebRTC offer, Gemini media relay, tool calls), records them, fans them out for live listen/takeover, and runs the greeting guard, call guard and Gemini caller transcriber. See Voice Engine.
  • Notifications — APN (iOS VoIP push), FCM (Android), email (SendGrid/ nodemailer).
  • Multi-tenant routing — associate users/teams/rooms with companies.
  • Webhooks — Twilio call events; Slack event subscriptions.

Boot flow​

Standard Nodevel (index.js → Application.run() → app/hooks.js). boot() loads Redis, Auth, Mongo, Cron, Logger, Mail, Permissions, Firebase, Firestore, Slack, APN, FCM, and an ElevenLabs voice bridge.

Directory map​

app/
├── models/ chats, inbound_outbound_calls, screened_calls, user_devices,
│ companies, agencies
├── services/ ApnService, FcmService, FirestoreService, MailService, SlackService,
│ MessagingService, SocketService, ElevenLabsBridgeService,
│ PostCallWebhookService, RecordingLinkService,
│ voiceDirect/ (RealtimeAdapter, GeminiLiveAdapter, LiveAdapter, CallerTranscriber,
│ DirectCallGuard, webCallAnalysis, VoiceDirectWebRelay,
│ VoiceDirectWebMediaRelay, VoiceDirectMsAi, recording/DirectCallRecorder
│ + mp3EncoderWorker, debug/DirectAudioDebugCapture), voiceVapi/
├── controllers/ MessageController (large — chat logic), CallsController,
│ InboundCallController (incl. keypress screening gate),
│ OutboundCallController (incl. Twilio AMD callback),
│ InboundOutboundCallController, TelnyxInboundCallController,
│ TelnyxOutboundCallController (incl. Telnyx AMD callback),
│ VoiceDirectRecordingController, SlackController
├── helpers/ TelnyxHelper (TeXML), AudioTranscodeHelper (mu-law + A-law),
│ FirstMessageHelper (continuation opener via ms-ai, 20s timeout),
│ VoiceProviderHelper (S2S model ids + TwiML provider hint), …
├── jobs/ cron jobs (e.g. recording merge)
├── keys/ APN cert (.p8), Firebase service account
└── routes/ api.js (Telnyx routes are declared in the controllers' routes())

Data models​

ModelPurpose
chatsthread metadata with embedded messages[] (sender, text, status, timestamp, slack_thread_id). Sticky has_bot_message / has_visitor_message flags stamped on write (never reset by "New chat" — backend daily-brief chat counts read them). widget_chat_starting_url = page the visitor was on when the conversation started (re-stamped on "New chat"); leadconnector_starting_url_note_id = GHL note id once posted (backend writes it, ms-comm clears on reset-chat).
inbound_outbound_callscall_sid, status, duration, recording_url, company_id, provider (twilio/telnyx), sentiment, voicemail, call_outcome, summary, post_call_webhook_sent_at, post_call_tags_applied, ghl_recording_state
screened_callsone row per inbound call turned away by the keypress screening gate (call_sid unique upsert, reason: no_input|wrong_digit, TTL 90d). Deliberately separate from inbound_outbound_calls so no call-count consumer changes.
user_devicespush tokens (apn_token, fcm_token, os)
companies / agenciestenant + Twilio/Slack config

Shared-collection indexes change in lock-step. chats and user_sessions schemas are declared in multiple services with autoIndex — a deploy of any one service with a divergent declaration rebuilds (or conflict-fails) the index for everyone. Recent examples: the { company_id, is_viewed, updated_at } index on user_sessions was removed, and the chats transcript-email sweep index was redefined as transcript_email_sweep_v2 ({ transcript_email_sent, created_at, updated_at }, partial on transcript_email_sent: false) — both must match the backend/ms-sessions declarations exactly.

Endpoints & socket events​

  • HTTP: health, POST /auth/login, POST /webhooks/twilio/inbound, Slack OAuth callback + POST /integrations/slack/events (raw-body signature verification).
  • Telephony: POST /twilio/{inbound,outbound}-voice/:company_id, POST /twilio/inbound-dial-fallback/:company_id (routing-priority chain), POST /twilio/inbound-screen/:company_id (keypress screening gate — stateless, digit rides ?d=, fail-open; see Telephony), POST /twilio/amd-status/:company_id + POST /telnyx/amd-status/:company_id (async AMD verdicts, always 204), POST /telnyx/{inbound,outbound}-voice/:company_id + -status callbacks, POST /telnyx/preheat-outbound/:company_id/:agent_id, POST /inbound-outbound-calls/save (post-call persistence; GET /inbound-outbound-calls/:id falls through to call_sid lookup). Media-stream WS upgrades on /twilio/media-stream and /telnyx/media-stream (path selects the carrier/codec path in ElevenLabsBridgeService).
  • Speech-to-speech: POST /voice-direct/recording (a direct web call's MP3 from the widget — session id + relay nonce in headers — forwarded to backend's recording route), POST /twilio/vapi-outbound-started/:company_id (Vapi-mode outbound), POST /voice-engine/post-call-analysis (engine-served calls). The bridge reads voice_provider off the TwiML <Stream> parameters; direct answers pre-warmed while ringing are shared across replicas via Redis vd:warm:<callSid> (GETDEL, exactly one adopter); a stream that finds the vd:warming:<callSid> marker waits for a pre-warm still minting on another replica instead of starting cold (5a0323a). Gemini turn-commit / stall / backpressure / echo-gate guards and the caller transcriber's model handoff: Voice Engine › Direct mode. S2S call records, service rooms and user_session.call_status carry s2s: true + voice_provider: 'voice-direct' + voice_vendor; listen/takeover work on them except xAI web and Vapi (s2s_not_supported). Widget S2S calls get a summary + outcome at call end (webCallAnalysis.js → ms-ai POST /voice-direct/web-analysis, fills empty call_summary / call_outcome).
  • Socket auth (helpers/SocketAccessHelper.js): sender fields must match the socket's own identity (bae25b2), agency users act on companies they manage, playground share token, unauthorized event, tokenless desktop fallback (5a41a6b) — see Socket room auth.
  • Call socket: twilio-webrtc-call-ring-again re-sends an unanswered agent call to the visitor (same room, server-minted token); call-log appends client steps to the room's call_logs (app/helpers/CallLogHelper.js, batched, capped at 1000).
  • Web AI calls (CallsController): S2S instructions (agent_instructions_s2s) and greeting (greeting_message_s2s) when the model is a permitted S2S model (GPT Live 1 / Gemini 3.8 Live are open to every company — VoiceProviderHelper mirrors the open/retired lists), GPT-Live's two instruction sets + gpt_live_reasoning_model on gpt-live-1, prefixed by exactly one You are <agent>; Headless SDK per-call context via _resolveSdkAgent — agent_key of a dashboard SDK voice agent (needs sdk_voice_agents_allowed > 0; its greeting, instructions, model, voice, speech settings and memory win) or inline instructions / greeting (sanitized, always allowed); an SDK agent's voice is applied per session as overrides.tts.voiceId, never onto the shared agent.
  • Inbound: agency pause gate — company.paused_at → dial the first human fallback (routing priority → transfer numbers → company phone) or "temporarily unavailable"; runs before screening and working hours, fails open. WhatsApp inbound for a paused company is ingested with the AI muted.
  • Socket.IO: message:send → message:received (room broadcast), call:initiate → Twilio bridge, notification:push → APN/FCM. A Firestore listener on /rooms/{room_id}/messages keeps clients in sync.

Integrations​

Twilio (calls/recording), Telnyx (TeXML second carrier), APN (iOS VoIP push), Firebase Admin (FCM + Firestore), SendGrid/nodemailer (email), Slack Web API, ElevenLabs (Convai voice — websocket bridge + WebRTC/LiveKit tokens), OpenAI GPT-Live / Realtime + realtime transcription, Gemini Live, xAI (direct S2S phone legs, the Gemini web relay and its caller transcriber), Vapi (S2S phone mode), AWS S3 (call recordings).

Email path​

MailService.send() no longer sends directly by default — it renders the EJS template locally and enqueues the pre-rendered HTML onto the backend's BullMQ email queue (the backend worker also accepts template jobs; pre-rendered { html, subject, to, … } jobs are supported for exactly this producer):

  • Queue Redis resolves EMAIL_QUEUE_REDIS_HOST → REDIS_HOST → localhost (app/configs/redis.js) — the DEV override points at the backend's Redis.
  • queueWithTimeout(payload, 5000) races BullMQ add() against a 5s timer (add() hangs while Redis is down); on timeout/error the send falls back to direct SMTP/SendGrid rather than losing the email. A late add() after the race is dropped to avoid duplicates.
  • Smtp() / sendGrid() now await and rethrow: the full nodemailer/ SendGrid error object (via mailError()) is persisted to email_logs — the only durable prod evidence of a failed send — and callers see the failure.
  • immediate: true skips the queue entirely.

One template shell (2026-08-27, d7ef3b8 / backend 142711093). app/views/templates/common/GENERAL.html is the only base (EMPTY.html stays for raw sends); MINIMAL.html and WHITE_LABEL.html were deleted in both repos and the 36 bodies are content-only fragments injected as body. common/_brand-logo.html decides the mark: white-label + agency logo → agency logo; white-label without logo → agency name text (a sub-account must never see the KK mark); otherwise the KK lockup with a prefers-color-scheme dark swap. Assets load from <base_url>/emails/* — ms-comm defaults base_url to Config.app('main_backend_url') because it has no /emails route. Preview locally with scripts/preview-emails.js → .preview/*.html. MailService.isArchivedRecipient(to, company_id) (mirrored from backend) runs at the top of send(): an explicit live company_id wins; otherwise all recipient addresses must resolve to archived users for the send to be blocked — one unknown address (lead, applicant, alias) keeps it alive. Blocks password resets too.

Transcript email sweep v3 (97edc74 / backend 7abd7c124): keyed on last_message_at, index transcript_email_sweep_v3 = {transcript_email_sent, created_at, last_message_at} partial on false, declared here and in backend lock-step; the claim is the timestamp transcript_email_processing_at (the old boolean was undeclared → dropped by strict mode → all four schedular pods raced), released on every exit path, STALE_CLAIM_MS = 15 min. Root cause of the hours-late batching was FactExtractionTriggerJob bumping updated_at every ~10 min; its no_person_id outcome is now a bounded wait (PERSON_WAIT_MS = 6 h) with no attempt spent. Call transcript emails lead with the ElevenLabs call_summary_title and render the summary above the transcript; the claim is gated on record.summary so the analysis-blind finalize pass doesn't burn it, and backend CallTranscriptEmailFallbackJob sends without a summary after 5 min.

Transcript email sweep v4 (ed2f23c, mirrors backend abba5cf9e): index transcript_email_sweep_v4 = {last_message_at, created_at} partial on transcript_email_sent: false, last_message_at > new Date(0) — empty per-visit chats are no longer indexed. Declared here in lock-step with backend; only backend runs the sweep.

Realtime​

Socket.IO 4.8 with @socket.io/redis-adapter for multi-pod fan-out; SocketService mirrors message state between Socket.IO and Firestore. Room joins are authorized by app/helpers/SocketAccessHelper.js (7813cfd): only dashboard users who can manage the company join the company room, dashboard-only events are refused from widget sockets, and Twilio grants are limited to the socket's own company. CORS preflights are cached 2 h (maxAge: 7200).

Cross-service​

Receives auth/tenant config from backend; pushes call-completion and read receipts back via webhook; queries ms-sessions presence to confirm a participant is live. Twilio in-progress on an outbound AI call fire-and-forgets a live-call push to backend POST /internal/knox/notifications/dispatch (type ai-call-live, dedup key per call SID) so the push governance pipeline applies. Calls to backend authenticate with backend_api_key (env BACKEND_API_KEY, falls back to WEBHOOK_API_KEY) — webhook_api_key is the ms-ai → ms-comm key and gets a 401 from backend. Request bodies are capped at 10 MB and express-fileupload is gone. See Cross-Service Communication.

Deployment​

Docker (port 8080) → ECR → EKS. The CI image (Dockerfile.azure, used by both GitHub workflows) is node:22-bookworm-slim, not Alpine: LiveKit's @livekit/rtc-node native binding ships for glibc only, so every LiveKit join failed on musl (8f3fc7d). The legacy Dockerfile / Dockerfile.stage still name the Alpine ECR base. HTTP pods scale on CPU; no separate worker pool (jobs are cron-based; the exactly-once post-call webhook fallback deliberately lives in the single-pod backend, not here). buildspec.yml builds, pushes, and rolls out. Secrets (TWILIO_*, SLACK_*, FIREBASE_KEY, TELNYX_MEDIA_STREAM_PATH, TELNYX_PUBLIC_WS_URL, RECORDING_LINK_SECRET, EMAIL_QUEUE_REDIS_HOST — DEV override for the backend email queue, BACKEND_INTERNAL_URL + secret VOICE_ENGINE_RECORDING_API_KEY for S2S recording uploads, OPENAI_API_KEY (GPT-Live phone, Gemini caller transcriber, end-intent classifier), optional GEMINI_API_KEY (Gemini phone API-key path; unset in PROD), VOICE_DIRECT_* guard/transcriber knobs, VOICE_S2S_PHONE_MODE, VOICE_ENGINE_TELEPHONY_ENABLED / _COMPANIES (engine pilot, default off), VOICE_DIRECT_DEBUG_CAPTURE — DEV audio tracing) come from the cluster.

Added 2026-10-20 → 11-16: SECRET_ENCRYPTION_KEY (secret) — app/helpers/SecretCrypto.js is a copy of the backend's and must hold the same keys (it decrypts WhatsApp tokens the backend wrote); FILE_LINK_SECRET (secret) — app/helpers/fileLink.js is a sign-only copy of the backend's signed private-file links (same test vectors), used for attachment links in transcripts and emails; LiveKit for the widget Gemini S2S media leg (VOICE_DIRECT_LIVEKIT, LIVEKIT_URL, LIVEKIT_PUBLIC_URL, secrets LIVEKIT_API_KEY / LIVEKIT_API_SECRET); xAI (XAI_API_KEY secret, XAI_SIP_ENABLED); APN_SANDBOX_FALLBACK (PROD "false": a production APNs BadDeviceToken is no longer retried on the sandbox gateway). See Deployment and Security & Secrets. (AMD tuning env AMD_ENABLED/AMD_TIMEOUT_SECONDS/AMD_SPEECH_THRESHOLD_MS lives in ms-ai, which appends the dial params; ms-comm only receives the callbacks.)