ms-communication — Chats, Calls & Notifications
Real-time messaging and voice. A Nodevel service on MongoDB, with Firestore as the live message store.
- Stack: Express/Nodevel, Mongoose/MongoDB, Socket.IO (+ Redis adapter), Firebase/Firestore.
- Port:
8080.
Responsibilities
- Messaging — text chat over Socket.IO + Firestore; Slack thread sync; message persistence.
- Voice/video calls — Twilio and Telnyx PSTN ↔ ElevenLabs media-stream bridges, WebRTC (LiveKit) token relay for widget AI calls, recording links, inbound/outbound automation, post-call webhooks. See Telephony.
- Speech-to-speech calls — the same carrier bridge can connect a call straight
to GPT-Live (
LiveAdapter, own OpenAI key) / Gemini Live / the retired Realtime models instead of ElevenLabs, relays widget S2S sessions (WebRTC offer, Gemini media relay, tool calls), records them, fans them out for live listen/takeover, and runs the greeting guard, call guard and Gemini caller transcriber. See Voice Engine. - Notifications — APN (iOS VoIP push), FCM (Android), email (SendGrid/ nodemailer).
- Multi-tenant routing — associate users/teams/rooms with companies.
- Webhooks — Twilio call events; Slack event subscriptions.
Boot flow
Standard Nodevel (index.js → Application.run() → app/hooks.js). boot() loads
Redis, Auth, Mongo, Cron, Logger, Mail, Permissions, Firebase, Firestore, Slack,
APN, FCM, and an ElevenLabs voice bridge.
Directory map
app/
├── models/ chats, inbound_outbound_calls, screened_calls, user_devices,
│ companies, agencies
├── services/ ApnService, FcmService, FirestoreService, MailService, SlackService,
│ MessagingService, SocketService, ElevenLabsBridgeService,
│ PostCallWebhookService, RecordingLinkService,
│ voiceDirect/ (RealtimeAdapter, GeminiLiveAdapter, LiveAdapter, CallerTranscriber,
│ DirectCallGuard, webCallAnalysis, VoiceDirectWebRelay,
│ VoiceDirectWebMediaRelay, VoiceDirectMsAi, recording/DirectCallRecorder
│ + mp3EncoderWorker, debug/DirectAudioDebugCapture), voiceVapi/
├── controllers/ MessageController (large — chat logic), CallsController,
│ InboundCallController (incl. keypress screening gate),
│ OutboundCallController (incl. Twilio AMD callback),
│ InboundOutboundCallController, TelnyxInboundCallController,
│ TelnyxOutboundCallController (incl. Telnyx AMD callback),
│ VoiceDirectRecordingController, SlackController
├── helpers/ TelnyxHelper (TeXML), AudioTranscodeHelper (mu-law + A-law),
│ FirstMessageHelper (continuation opener via ms-ai, 20s timeout),
│ VoiceProviderHelper (S2S model ids + TwiML provider hint), …
├── jobs/ cron jobs (e.g. recording merge)
├── keys/ APN cert (.p8), Firebase service account
└── routes/ api.js (Telnyx routes are declared in the controllers' routes())
Data models
| Model | Purpose |
|---|---|
chats | thread metadata with embedded messages[] (sender, text, status, timestamp, slack_thread_id). Sticky has_bot_message / has_visitor_message flags stamped on write (never reset by "New chat" — backend daily-brief chat counts read them). widget_chat_starting_url = page the visitor was on when the conversation started (re-stamped on "New chat"); leadconnector_starting_url_note_id = GHL note id once posted (backend writes it, ms-comm clears on reset-chat). |
inbound_outbound_calls | call_sid, status, duration, recording_url, company_id, provider (twilio/telnyx), sentiment, voicemail, call_outcome, summary, post_call_webhook_sent_at, post_call_tags_applied, ghl_recording_state |
screened_calls | one row per inbound call turned away by the keypress screening gate (call_sid unique upsert, reason: no_input|wrong_digit, TTL 90d). Deliberately separate from inbound_outbound_calls so no call-count consumer changes. |
user_devices | push tokens (apn_token, fcm_token, os) |
companies / agencies | tenant + Twilio/Slack config |
Shared-collection indexes change in lock-step. chats and user_sessions schemas are declared in multiple services with autoIndex — a deploy of any one service with a divergent declaration rebuilds (or conflict-fails) the index for everyone. Recent examples: the { company_id, is_viewed, updated_at } index on user_sessions was removed, and the chats transcript-email sweep index was redefined as transcript_email_sweep_v2 ({ transcript_email_sent, created_at, updated_at }, partial on transcript_email_sent: false) — both must match the backend/ms-sessions declarations exactly.
Endpoints & socket events
- HTTP: health,
POST /auth/login,POST /webhooks/twilio/inbound, Slack OAuth callback +POST /integrations/slack/events(raw-body signature verification). - Telephony:
POST /twilio/{inbound,outbound}-voice/:company_id,POST /twilio/inbound-dial-fallback/:company_id(routing-priority chain),POST /twilio/inbound-screen/:company_id(keypress screening gate — stateless, digit rides?d=, fail-open; see Telephony),POST /twilio/amd-status/:company_id+POST /telnyx/amd-status/:company_id(async AMD verdicts, always 204),POST /telnyx/{inbound,outbound}-voice/:company_id+-statuscallbacks,POST /telnyx/preheat-outbound/:company_id/:agent_id,POST /inbound-outbound-calls/save(post-call persistence;GET /inbound-outbound-calls/:idfalls through tocall_sidlookup). Media-stream WS upgrades on/twilio/media-streamand/telnyx/media-stream(path selects the carrier/codec path inElevenLabsBridgeService). - Speech-to-speech:
POST /voice-direct/recording(a direct web call's MP3 from the widget — session id + relay nonce in headers — forwarded to backend's recording route),POST /twilio/vapi-outbound-started/:company_id(Vapi-mode outbound),POST /voice-engine/post-call-analysis(engine-served calls). The bridge readsvoice_provideroff the TwiML<Stream>parameters; direct answers pre-warmed while ringing are shared across replicas via Redisvd:warm:<callSid>(GETDEL, exactly one adopter); a stream that finds thevd:warming:<callSid>marker waits for a pre-warm still minting on another replica instead of starting cold (5a0323a). Gemini turn-commit / stall / backpressure / echo-gate guards and the caller transcriber's model handoff: Voice Engine › Direct mode. S2S call records, service rooms anduser_session.call_statuscarrys2s: true+voice_provider: 'voice-direct'+voice_vendor; listen/takeover work on them except xAI web and Vapi (s2s_not_supported). Widget S2S calls get a summary + outcome at call end (webCallAnalysis.js→ ms-aiPOST /voice-direct/web-analysis, fills emptycall_summary/call_outcome). - Socket auth (
helpers/SocketAccessHelper.js): sender fields must match the socket's own identity (bae25b2), agency users act on companies they manage, playground share token,unauthorizedevent, tokenless desktop fallback (5a41a6b) — see Socket room auth. - Call socket:
twilio-webrtc-call-ring-againre-sends an unanswered agent call to the visitor (same room, server-minted token);call-logappends client steps to the room'scall_logs(app/helpers/CallLogHelper.js, batched, capped at 1000). - Web AI calls (
CallsController): S2S instructions (agent_instructions_s2s) and greeting (greeting_message_s2s) when the model is a permitted S2S model (GPT Live 1 / Gemini 3.8 Live are open to every company —VoiceProviderHelpermirrors the open/retired lists), GPT-Live's two instruction sets +gpt_live_reasoning_modelongpt-live-1, prefixed by exactly oneYou are <agent>; Headless SDK per-call context via_resolveSdkAgent—agent_keyof a dashboard SDK voice agent (needssdk_voice_agents_allowed> 0; its greeting, instructions, model, voice, speech settings and memory win) or inlineinstructions/greeting(sanitized, always allowed); an SDK agent's voice is applied per session asoverrides.tts.voiceId, never onto the shared agent. - Inbound: agency pause gate —
company.paused_at→ dial the first human fallback (routing priority → transfer numbers → company phone) or "temporarily unavailable"; runs before screening and working hours, fails open. WhatsApp inbound for a paused company is ingested with the AI muted. - Socket.IO:
message:send→message:received(room broadcast),call:initiate→ Twilio bridge,notification:push→ APN/FCM. A Firestore listener on/rooms/{room_id}/messageskeeps clients in sync.
Integrations
Twilio (calls/recording), Telnyx (TeXML second carrier), APN (iOS VoIP push), Firebase Admin (FCM + Firestore), SendGrid/nodemailer (email), Slack Web API, ElevenLabs (Convai voice — websocket bridge + WebRTC/LiveKit tokens), OpenAI GPT-Live / Realtime + realtime transcription, Gemini Live, xAI (direct S2S phone legs, the Gemini web relay and its caller transcriber), Vapi (S2S phone mode), AWS S3 (call recordings).
Email path
MailService.send() no longer sends directly by default — it renders the EJS
template locally and enqueues the pre-rendered HTML onto the backend's
BullMQ email queue (the backend worker also accepts template jobs; pre-rendered
{ html, subject, to, … } jobs are supported for exactly this producer):
- Queue Redis resolves
EMAIL_QUEUE_REDIS_HOST → REDIS_HOST → localhost(app/configs/redis.js) — the DEV override points at the backend's Redis. queueWithTimeout(payload, 5000)races BullMQadd()against a 5s timer (add() hangs while Redis is down); on timeout/error the send falls back to direct SMTP/SendGrid rather than losing the email. A late add() after the race is dropped to avoid duplicates.Smtp()/sendGrid()now await and rethrow: the full nodemailer/ SendGrid error object (viamailError()) is persisted toemail_logs— the only durable prod evidence of a failed send — and callers see the failure.immediate: trueskips the queue entirely.
One template shell (2026-08-27, d7ef3b8 / backend 142711093).
app/views/templates/common/GENERAL.html is the only base (EMPTY.html stays
for raw sends); MINIMAL.html and WHITE_LABEL.html were deleted in both
repos and the 36 bodies are content-only fragments injected as body.
common/_brand-logo.html decides the mark: white-label + agency logo → agency
logo; white-label without logo → agency name text (a sub-account must never
see the KK mark); otherwise the KK lockup with a prefers-color-scheme dark
swap. Assets load from <base_url>/emails/* — ms-comm defaults base_url to
Config.app('main_backend_url') because it has no /emails route. Preview
locally with scripts/preview-emails.js → .preview/*.html.
MailService.isArchivedRecipient(to, company_id) (mirrored from backend) runs
at the top of send(): an explicit live company_id wins; otherwise all
recipient addresses must resolve to archived users for the send to be blocked
— one unknown address (lead, applicant, alias) keeps it alive. Blocks password
resets too.
Transcript email sweep v3 (97edc74 / backend 7abd7c124): keyed on
last_message_at, index transcript_email_sweep_v3 = {transcript_email_sent, created_at, last_message_at} partial on false, declared here and in backend
lock-step; the claim is the timestamp transcript_email_processing_at (the old
boolean was undeclared → dropped by strict mode → all four schedular pods
raced), released on every exit path, STALE_CLAIM_MS = 15 min. Root cause of
the hours-late batching was FactExtractionTriggerJob bumping updated_at
every ~10 min; its no_person_id outcome is now a bounded wait
(PERSON_WAIT_MS = 6 h) with no attempt spent. Call transcript emails lead
with the ElevenLabs call_summary_title and render the summary above the
transcript; the claim is gated on record.summary so the analysis-blind
finalize pass doesn't burn it, and backend CallTranscriptEmailFallbackJob
sends without a summary after 5 min.
Transcript email sweep v4 (ed2f23c, mirrors backend abba5cf9e): index
transcript_email_sweep_v4 = {last_message_at, created_at} partial on
transcript_email_sent: false, last_message_at > new Date(0) — empty per-visit
chats are no longer indexed. Declared here in lock-step with backend; only
backend runs the sweep.
Realtime
Socket.IO 4.8 with @socket.io/redis-adapter for multi-pod fan-out; SocketService
mirrors message state between Socket.IO and Firestore. Room joins are
authorized by app/helpers/SocketAccessHelper.js (7813cfd): only dashboard
users who can manage the company join the company room, dashboard-only events
are refused from widget sockets, and Twilio grants are limited to the socket's
own company. CORS preflights are cached 2 h (maxAge: 7200).
Cross-service
Receives auth/tenant config from backend; pushes call-completion and read
receipts back via webhook; queries ms-sessions presence to confirm a participant
is live. Twilio in-progress on an outbound AI call fire-and-forgets a
live-call push to backend POST /internal/knox/notifications/dispatch
(type ai-call-live, dedup key per call SID) so the push governance pipeline
applies. Calls to backend authenticate with backend_api_key (env BACKEND_API_KEY, falls back to WEBHOOK_API_KEY) — webhook_api_key is the ms-ai → ms-comm key and gets a 401 from backend. Request bodies are capped at 10 MB and express-fileupload is gone. See Cross-Service Communication.
Deployment
Docker (port 8080) → ECR → EKS. The CI image (Dockerfile.azure, used by both GitHub workflows) is node:22-bookworm-slim, not Alpine: LiveKit's @livekit/rtc-node native binding ships for glibc only, so every LiveKit join failed on musl (8f3fc7d). The legacy Dockerfile / Dockerfile.stage still name the Alpine ECR base. HTTP pods scale on CPU; no
separate worker pool (jobs are cron-based; the exactly-once post-call webhook
fallback deliberately lives in the single-pod backend, not here). buildspec.yml
builds, pushes, and rolls out. Secrets (TWILIO_*, SLACK_*, FIREBASE_KEY,
TELNYX_MEDIA_STREAM_PATH, TELNYX_PUBLIC_WS_URL, RECORDING_LINK_SECRET,
EMAIL_QUEUE_REDIS_HOST — DEV override for the backend email queue,
BACKEND_INTERNAL_URL + secret VOICE_ENGINE_RECORDING_API_KEY for S2S recording
uploads, OPENAI_API_KEY (GPT-Live phone, Gemini caller transcriber, end-intent
classifier), optional GEMINI_API_KEY (Gemini phone API-key path; unset in PROD),
VOICE_DIRECT_* guard/transcriber knobs, VOICE_S2S_PHONE_MODE,
VOICE_ENGINE_TELEPHONY_ENABLED / _COMPANIES (engine pilot, default off),
VOICE_DIRECT_DEBUG_CAPTURE — DEV audio tracing) come from the cluster.
Added 2026-10-20 → 11-16: SECRET_ENCRYPTION_KEY (secret) — app/helpers/SecretCrypto.js
is a copy of the backend's and must hold the same keys (it decrypts WhatsApp
tokens the backend wrote); FILE_LINK_SECRET (secret) — app/helpers/fileLink.js
is a sign-only copy of the backend's signed private-file links (same test
vectors), used for attachment links in transcripts and emails; LiveKit for the
widget Gemini S2S media leg (VOICE_DIRECT_LIVEKIT, LIVEKIT_URL,
LIVEKIT_PUBLIC_URL, secrets LIVEKIT_API_KEY / LIVEKIT_API_SECRET); xAI
(XAI_API_KEY secret, XAI_SIP_ENABLED); APN_SANDBOX_FALLBACK (PROD "false":
a production APNs BadDeviceToken is no longer retried on the sandbox gateway).
See Deployment and
Security & Secrets. (AMD tuning env AMD_ENABLED/AMD_TIMEOUT_SECONDS/AMD_SPEECH_THRESHOLD_MS
lives in ms-ai, which appends the dial params; ms-comm only receives the callbacks.)