Skip to main content

Cross-Service Communication

There is no shared database. Services integrate over three channels:

  1. HTTP / REST — service-to-service calls and webhooks (the primary channel).
  2. Socket.IO — realtime, fanned out across pods via a Redis adapter.
  3. Redis — shared for socket adapters, BullMQ queues, throttling, and caches (each service uses its own logical Redis/keys; not a shared data model).

Who calls whom​

From → ToTransportPurpose
widget / frontend / desktop → backendHTTP + Socket.IOApp API, auth, live dashboards, call/chat signalling
widget → ms-sessionsSocket.IO (widget:event)rrweb event ingest
widget → ms-communicationSocket.IOChat messages, AI chat streaming
backend → ms-aiHTTP (Bearer MS_AI_SECRET_TOKEN)Fact extraction, Knox sync, voice handoff, post-call tag classification
backend → ms-sessionsHTTPEnqueue/​backfill recording analysis
ms-sessions → ms-aiHTTPHand analyzed recordings off for enrichment
ms-ai → backendHTTP + gRPC (identity)Identity-graph lookups, callbacks
ms-communication → backendHTTP (x-api-key = BACKEND_API_KEY)Identity-graph resolve, scheduling port, Simpro call notes, call/message callbacks
ms-communication / ms-sessions ↔ each otherHTTPVisitor presence checks (is the participant live?)
backend ↔ External CRMsHTTP webhooks + APIGHL/HubSpot/Slack/Stripe/Twilio

backend ↔ ms-ai​

The backend integrates with ms-ai through helper clients in backend/app/lib/:

  • msAiHandoff.js — send a finalized SessionFile/transcript for voice/AI processing.
  • msAiKnoxClient.js — Knox pipeline calls: /internal/knox/*, brief snapshots (/knox-agent/brief-snapshot), admin memory queries (/knox-agent/memories/admin), GHL sync events, action events.
  • Auth is a shared secret: backend resolves the key MS_AI_SECRET_TOKEN → SECRET_AUTH_TOKEN → MS_AI_API_KEY (legacy default last; prod sets MS_AI_SECRET_TOKEN) and sends it as Bearer; ms-ai validates SECRET_AUTH_TOKEN via its secret auth guard. POST /agents/classify-call-tags was @AuthType('public') and is now secret (3611d73f0) — its only callers are backend post-call jobs.

A sweeper cron (MsAiHandoffSweeperJob, every 5 min) retries failed handoffs so a transient ms-ai outage doesn't drop work.

backend ↔ ms-sessions​

  • backend/app/lib/msSessionsRecordingClient.js enqueues a SessionFile for analysis.
  • RecordingBackfillSweeperJob (every 15 min) backfills any SessionFile whose handoff to ms-sessions failed.

ms-communication → backend​

Backend's internal routes sit behind apiKeyAuth, which expects BACKEND_API_KEY (falls back to WEBHOOK_API_KEY, then a built-in key with a CRITICAL boot log). Since 276fdb8 (2026-09-28) ms-communication sends Config.app('backend_api_key') — env BACKEND_API_KEY, falling back to WEBHOOK_API_KEY — on every backend call (/identity-graph/resolve, SchedulingClientHelper /scheduling/*, /simpro/call-note, call and message callbacks, the ElevenLabs bridge). It used webhook_api_key before, which is the ms-ai → ms-communication key and is rejected by backend when the two differ (401 Invalid API key). BACKEND_API_KEY must be set in every deployed ms-communication environment.

Request body limits​

All four services were on 1000 MB–2 GB JSON limits; bounded on 2026-09-16:

ServiceJSON/urlencoded limitNotes
backend25 MBexpress-fileupload kept (100 MB, temp files) for real uploads
ms-ai20 MBsized against the ms-sessions recording handoff (capped at RECORDING_DOC_SIZE_LIMIT_BYTES, 5 MB); that producer does not retry a 4xx, so a 413 would drop the session's narration for good
ms-communication10 MBthe json verify hook copies the raw body into req.rawBody, so a body costs ~2× its size; unused express-fileupload removed
ms-sessions10 MB, /sessions/event 50 MBlarge-rrweb-event fallback; unused express-fileupload removed

Service URLs & secrets​

Backend resolves peer services from env (backend/app/configs/app.js):

Env varPoints at
MS_AI_URLms-ai base URL
MS_SESSION_URLms-sessions base URL
MS_COMMUNICATION_URLms-communication base URL
MS_AI_SECRET_TOKEN (→ SECRET_AUTH_TOKEN → MS_AI_API_KEY)shared secret for ms-ai calls
BACKEND_API_KEYkey backend's apiKeyAuth accepts from other services; also the x-api-key the ops AI agents send with x-ops-agent

Shared values (not just names) that must match across services: SECRET_ENCRYPTION_KEY (backend, schedular, ms-communication — encrypted tenant secrets), FILE_LINK_SECRET (backend, schedular, ms-communication — signed private-file links). ms-ai keeps its own encryption keyring. See Security & Secrets.

In the cluster these come from ConfigMaps + SOPS-encrypted secrets; locally from .env. See Local Development.

Realtime (Socket.IO)​

  • backend runs the main Socket.IO server with a Redis adapter (falls back to in-memory for single-pod dev). user-status-update is emitted only to the user's company room and user:<id> room, not platform-wide. It tracks session→socket mappings in Redis (session_sockets: prefix, 24h TTL) and throttles heartbeat writes (heartbeat_status: prefix, 5m TTL).
  • ms-sessions uses @socket.io/redis-streams-adapter and a cross-pod tab registry (session_tabs:{room} hashes, 90s TTL renewed every 30s) so visitor presence is globally visible regardless of which pod a tab connected to.
  • ms-communication uses the standard @socket.io/redis-adapter for chat/call fan-out and mirrors message state to Firestore.

The frontend holds two sockets: window.main_socket (agent/internal notifications) and window.socket (user-facing call/chat/session events). Key events are documented on the frontend page.

Socket room auth (2026-10-14)​

Company rooms carry live visitor data, so all three socket servers now refuse to put a socket in one on the strength of a company_id query param alone (backend 6c7dd2a7b, ms-communication 7813cfd, ms-sessions 9899eaf):

  • Dashboard sockets send the user JWT as handshake.auth.token (frontend socketManager.js reads it on every (re)connect, so impersonation and token refresh hand over the current token; the playground sends its owner token — 0ab6c8b0). The company room (and backend's user:<id> room) is joined only if the token is valid, the user isn't blocked or revoked (tokens_valid_from), and the user can manage that company — a port of backend GeneralHelper.canManageCompany (ms-sessions lib/companyAccess.js, ms-communication helpers/SocketAccessHelper.js).
  • Widget sockets (no token) join only their own session room; backend also puts them in a public widget:<company_id> room that only carries company-settings-updated.
  • Relays — backend dashboard relays (stop-live-session, update-live-session) need an authorized dashboard socket; ms-communication refuses dashboard-only events (agent call accept/end/transfer, monitor/takeover, chat-queue, …) from widget sockets, sends visitor/agent relays and Twilio token responses only to the company dashboards + that visitor's session room, and mints Twilio grants only for a room of the socket's company (and visitor). ms-sessions' public HTTP fallback (LiveSessionController) accepts only the page-url / send-event channels, into the caller's own session room, and reaches the company room only when the session really belongs to it. Backend's /custom-event/create relay was removed.

A client that connects without a token still gets a socket — it just never receives company events. If a dashboard stops getting live updates, check the handshake carries auth.token.

Follow-ups (2026-10-14/15) — backend f084fcacb, ms-communication 5a41a6b / bae25b2, ms-sessions ea6d515, frontend 44251b51:

  • unauthorized event — a token that fails verification, names a blocked/ revoked user, or can't manage the handshake company gets unauthorized { service } from all three servers. socketManager.js reconnects all three sockets once with the current token (at most every 60 s); store/app.js calls refreshSocketAuth() when a tab frozen by another tab's identity switch is released.
  • Playground share links — PlaygroundController returns a socket_token (scope: 'playground', playground_company_id, 12 h, no user id, so passport rejects it as an API token). All three socket servers admit it to that one company room only. usePlaygroundInit uses it when the visitor has no login; PlaygroundPreview.vue reuses window.socket only when it is for the same company, otherwise opens its own.
  • Desktop compatibility — a tokenless socket with query.user_id (and no user_session_id) that can manage query.company_id is admitted (socket.data.legacy) while SOCKET_LEGACY_DASHBOARD_AUTH is not 'false' (default on; set it to 'false' in backend, ms-communication and ms-sessions once desktop ships the token).
  • Agency users — ms-communication canActFor lets a dashboard socket act on any company its user manages (agency → client/playground), cached per socket; a user-only socket must name a session/company it manages.
  • Sender identity (ms-communication bae25b2) — authorizeEvent checks the sender field (message/save-chat-message username, summary-reply agent_id, chat-queue-claim/typing user_id, every save-chat-messages-bulk message): a widget may only be its own session or chatbot, a dashboard only its own user, a playground share socket a user who manages that company. A widget may mark only sender: 'agent' messages seen.
  • Public HTTP — backend 324576a72: /users/load-data-on-ghl and /agencies/load-data-on-ghl/load deleted; /chat|room/load-data-on-ghl admin-only; public /chats/user-sessions/:id and /all-conversations/user/:id return only messages + the agent's public card (GeneralHelper.publicUser) to callers who can't manage the company; agent-call-request is checked and sent to the session company's room only. Public PUT /service-room/:_id (backend and ms-communication) lets a non-manager set only forwarded_as_phone_call, forwarded_call_duration, is_chat_bot — backend on rooms under 24 h old, ms-communication only with the room's own user_session_id (requestManager checks the Bearer JWT). Backend's legacy call.js relay (call-decline, widget_interaction_rejoin_request) is restored with the company taken from the session.

Server → dashboard state pushes​

Backend RealtimeHelper emits id-only pings so open dashboards refetch: wallet-updated (any credit-ledger row, top-ups, grants) and company-settings-updated (Stripe/subscription changes, expiry crons, admin feature toggles, agency client-billing edits) to the company room — agency events to each agency user's user:<id> room (b060de28d). The frontend listens in App.vue and reloads the sidebar wallet, subscription and settings (0c066828).

Auth model summary​

CallerMechanism
Browser → backendJWT bearer (issued by backend; passport OAuth for social login)
Browser → ms-ai (public tools)rate-limited, no auth (e.g. calendar availability/booking)
Service → ms-aisecret guard: x-api-key or Bearer shared token
Webhooks (Shopify/ElevenLabs/Slack)HMAC signature over raw body
backend → CRMsper-integration OAuth tokens / API keys stored per company