Cross-Service Communication
There is no shared database. Services integrate over three channels:
- HTTP / REST — service-to-service calls and webhooks (the primary channel).
- Socket.IO — realtime, fanned out across pods via a Redis adapter.
- Redis — shared for socket adapters, BullMQ queues, throttling, and caches (each service uses its own logical Redis/keys; not a shared data model).
Who calls whom
| From → To | Transport | Purpose |
|---|---|---|
| widget / frontend / desktop → backend | HTTP + Socket.IO | App API, auth, live dashboards, call/chat signalling |
| widget → ms-sessions | Socket.IO (widget:event) | rrweb event ingest |
| widget → ms-communication | Socket.IO | Chat messages, AI chat streaming |
| backend → ms-ai | HTTP (Bearer MS_AI_SECRET_TOKEN) | Fact extraction, Knox sync, voice handoff, post-call tag classification |
| backend → ms-sessions | HTTP | Enqueue/backfill recording analysis |
| ms-sessions → ms-ai | HTTP | Hand analyzed recordings off for enrichment |
| ms-ai → backend | HTTP + gRPC (identity) | Identity-graph lookups, callbacks |
| ms-communication → backend | HTTP (x-api-key = BACKEND_API_KEY) | Identity-graph resolve, scheduling port, Simpro call notes, call/message callbacks |
| ms-communication / ms-sessions ↔ each other | HTTP | Visitor presence checks (is the participant live?) |
| backend ↔ External CRMs | HTTP webhooks + API | GHL/HubSpot/Slack/Stripe/Twilio |
backend ↔ ms-ai
The backend integrates with ms-ai through helper clients in
backend/app/lib/:
msAiHandoff.js— send a finalizedSessionFile/transcript for voice/AI processing.msAiKnoxClient.js— Knox pipeline calls:/internal/knox/*, brief snapshots (/knox-agent/brief-snapshot), admin memory queries (/knox-agent/memories/admin), GHL sync events, action events.- Auth is a shared secret: backend resolves the key
MS_AI_SECRET_TOKEN → SECRET_AUTH_TOKEN → MS_AI_API_KEY(legacy default last; prod setsMS_AI_SECRET_TOKEN) and sends it as Bearer; ms-ai validatesSECRET_AUTH_TOKENvia itssecretauth guard.POST /agents/classify-call-tagswas@AuthType('public')and is nowsecret(3611d73f0) — its only callers are backend post-call jobs.
A sweeper cron (MsAiHandoffSweeperJob, every 5 min) retries failed handoffs so
a transient ms-ai outage doesn't drop work.
backend ↔ ms-sessions
backend/app/lib/msSessionsRecordingClient.jsenqueues aSessionFilefor analysis.RecordingBackfillSweeperJob(every 15 min) backfills anySessionFilewhose handoff to ms-sessions failed.
ms-communication → backend
Backend's internal routes sit behind apiKeyAuth, which expects
BACKEND_API_KEY (falls back to WEBHOOK_API_KEY, then a built-in key with a
CRITICAL boot log). Since 276fdb8 (2026-09-28) ms-communication sends
Config.app('backend_api_key') — env BACKEND_API_KEY, falling back to
WEBHOOK_API_KEY — on every backend call (/identity-graph/resolve,
SchedulingClientHelper /scheduling/*, /simpro/call-note, call and message
callbacks, the ElevenLabs bridge). It used webhook_api_key before, which is
the ms-ai → ms-communication key and is rejected by backend when the two
differ (401 Invalid API key). BACKEND_API_KEY must be set in every deployed
ms-communication environment.
Request body limits
All four services were on 1000 MB–2 GB JSON limits; bounded on 2026-09-16:
| Service | JSON/urlencoded limit | Notes |
|---|---|---|
| backend | 25 MB | express-fileupload kept (100 MB, temp files) for real uploads |
| ms-ai | 20 MB | sized against the ms-sessions recording handoff (capped at RECORDING_DOC_SIZE_LIMIT_BYTES, 5 MB); that producer does not retry a 4xx, so a 413 would drop the session's narration for good |
| ms-communication | 10 MB | the json verify hook copies the raw body into req.rawBody, so a body costs ~2× its size; unused express-fileupload removed |
| ms-sessions | 10 MB, /sessions/event 50 MB | large-rrweb-event fallback; unused express-fileupload removed |
Service URLs & secrets
Backend resolves peer services from env (backend/app/configs/app.js):
| Env var | Points at |
|---|---|
MS_AI_URL | ms-ai base URL |
MS_SESSION_URL | ms-sessions base URL |
MS_COMMUNICATION_URL | ms-communication base URL |
MS_AI_SECRET_TOKEN (→ SECRET_AUTH_TOKEN → MS_AI_API_KEY) | shared secret for ms-ai calls |
BACKEND_API_KEY | key backend's apiKeyAuth accepts from other services; also the x-api-key the ops AI agents send with x-ops-agent |
Shared values (not just names) that must match across services: SECRET_ENCRYPTION_KEY
(backend, schedular, ms-communication — encrypted tenant secrets), FILE_LINK_SECRET
(backend, schedular, ms-communication — signed private-file links). ms-ai keeps its own
encryption keyring. See Security & Secrets.
In the cluster these come from ConfigMaps + SOPS-encrypted secrets; locally from
.env. See Local Development.
Realtime (Socket.IO)
- backend runs the main Socket.IO server with a Redis adapter (falls back to
in-memory for single-pod dev).
user-status-updateis emitted only to the user's company room anduser:<id>room, not platform-wide. It tracks session→socket mappings in Redis (session_sockets:prefix, 24h TTL) and throttles heartbeat writes (heartbeat_status:prefix, 5m TTL). - ms-sessions uses
@socket.io/redis-streams-adapterand a cross-pod tab registry (session_tabs:{room}hashes, 90s TTL renewed every 30s) so visitor presence is globally visible regardless of which pod a tab connected to. - ms-communication uses the standard
@socket.io/redis-adapterfor chat/call fan-out and mirrors message state to Firestore.
The frontend holds two sockets: window.main_socket (agent/internal
notifications) and window.socket (user-facing call/chat/session events). Key
events are documented on the frontend page.
Socket room auth (2026-10-14)
Company rooms carry live visitor data, so all three socket servers now refuse
to put a socket in one on the strength of a company_id query param alone
(backend 6c7dd2a7b, ms-communication 7813cfd, ms-sessions 9899eaf):
- Dashboard sockets send the user JWT as
handshake.auth.token(frontendsocketManager.jsreads it on every (re)connect, so impersonation and token refresh hand over the current token; the playground sends its owner token —0ab6c8b0). The company room (and backend'suser:<id>room) is joined only if the token is valid, the user isn't blocked or revoked (tokens_valid_from), and the user can manage that company — a port of backendGeneralHelper.canManageCompany(ms-sessionslib/companyAccess.js, ms-communicationhelpers/SocketAccessHelper.js). - Widget sockets (no token) join only their own session room; backend
also puts them in a public
widget:<company_id>room that only carriescompany-settings-updated. - Relays — backend dashboard relays (
stop-live-session,update-live-session) need an authorized dashboard socket; ms-communication refuses dashboard-only events (agent call accept/end/transfer, monitor/takeover, chat-queue, …) from widget sockets, sends visitor/agent relays and Twilio token responses only to the company dashboards + that visitor's session room, and mints Twilio grants only for a room of the socket's company (and visitor). ms-sessions' public HTTP fallback (LiveSessionController) accepts only thepage-url/send-eventchannels, into the caller's own session room, and reaches the company room only when the session really belongs to it. Backend's/custom-event/createrelay was removed.
A client that connects without a token still gets a socket — it just never
receives company events. If a dashboard stops getting live updates, check the
handshake carries auth.token.
Follow-ups (2026-10-14/15) — backend f084fcacb, ms-communication 5a41a6b
/ bae25b2, ms-sessions ea6d515, frontend 44251b51:
unauthorizedevent — a token that fails verification, names a blocked/ revoked user, or can't manage the handshake company getsunauthorized { service }from all three servers.socketManager.jsreconnects all three sockets once with the current token (at most every 60 s);store/app.jscallsrefreshSocketAuth()when a tab frozen by another tab's identity switch is released.- Playground share links —
PlaygroundControllerreturns asocket_token(scope: 'playground',playground_company_id, 12 h, no user id, so passport rejects it as an API token). All three socket servers admit it to that one company room only.usePlaygroundInituses it when the visitor has no login;PlaygroundPreview.vuereuseswindow.socketonly when it is for the same company, otherwise opens its own. - Desktop compatibility — a tokenless socket with
query.user_id(and nouser_session_id) that can managequery.company_idis admitted (socket.data.legacy) whileSOCKET_LEGACY_DASHBOARD_AUTHis not'false'(default on; set it to'false'in backend, ms-communication and ms-sessions once desktop ships the token). - Agency users — ms-communication
canActForlets a dashboard socket act on any company its user manages (agency → client/playground), cached per socket; a user-only socket must name a session/company it manages. - Sender identity (ms-communication
bae25b2) —authorizeEventchecks the sender field (message/save-chat-messageusername,summary-replyagent_id,chat-queue-claim/typinguser_id, everysave-chat-messages-bulkmessage): a widget may only be its own session orchatbot, a dashboard only its own user, a playground share socket a user who manages that company. A widget may mark onlysender: 'agent'messages seen. - Public HTTP — backend
324576a72:/users/load-data-on-ghland/agencies/load-data-on-ghl/loaddeleted;/chat|room/load-data-on-ghladmin-only; public/chats/user-sessions/:idand/all-conversations/user/:idreturn only messages + the agent's public card (GeneralHelper.publicUser) to callers who can't manage the company;agent-call-requestis checked and sent to the session company's room only. PublicPUT /service-room/:_id(backend and ms-communication) lets a non-manager set onlyforwarded_as_phone_call,forwarded_call_duration,is_chat_bot— backend on rooms under 24 h old, ms-communication only with the room's ownuser_session_id(requestManagerchecks the Bearer JWT). Backend's legacycall.jsrelay (call-decline,widget_interaction_rejoin_request) is restored with the company taken from the session.
Server → dashboard state pushes
Backend RealtimeHelper emits id-only pings so open dashboards refetch:
wallet-updated (any credit-ledger row, top-ups, grants) and
company-settings-updated (Stripe/subscription changes, expiry crons, admin
feature toggles, agency client-billing edits) to the company room — agency
events to each agency user's user:<id> room (b060de28d). The frontend
listens in App.vue and reloads the sidebar wallet, subscription and settings
(0c066828).
Auth model summary
| Caller | Mechanism |
|---|---|
| Browser → backend | JWT bearer (issued by backend; passport OAuth for social login) |
| Browser → ms-ai (public tools) | rate-limited, no auth (e.g. calendar availability/booking) |
| Service → ms-ai | secret guard: x-api-key or Bearer shared token |
| Webhooks (Shopify/ElevenLabs/Slack) | HMAC signature over raw body |
| backend → CRMs | per-integration OAuth tokens / API keys stored per company |