ms-sessions — Recording & Presence
Captures visitor sessions (rrweb), turns them into analyzed recordings, and is the source of truth for visitor presence. A Nodevel service on MongoDB + Redis + S3.
- Port:
3003. A separate recording-analysis worker process runs the heavy pipeline.
Responsibilities
- Capture — receive rrweb DOM events from the widget over Socket.IO, buffer in Redis, flush to disk, enqueue for processing.
- Video synthesis — stitch event streams into MP4 (ffmpeg /
webm-to-mp4/video-stitch/ handbrake), tag for AI analysis. - Session state —
user_sessions,session_files, lead scoring, visitor profiles. - Presence — a cross-pod tab registry queried by ms-ai/Knox and ms-communication to know if a visitor is live.
- Lead qualification — engagement metrics (pages, time on high-intent pages, chat/call acceptance, bookings).
Boot flow
Standard Nodevel; app/hooks.js boot() loads Redis, Auth, Mongo, Cron, Logger,
Mail, Permissions, Socket, S3, and a 30 req/min rate limiter on session endpoints.
Body parsers are capped at 10 MB, except /sessions/event at 50 MB (the
large-rrweb-event fallback — see Session Recording);
they were 1000 MB. express-fileupload was removed (c6d54e1): it was
mounted with no options (unbounded multipart buffered in memory) and no route
reads req.files (app/helpers/UploadHelper.js is unregistered dead code).
The HTTP server binds port 3003; the worker boots the same app with
__skip_server = true.
Directory map
app/
├── models/ UserSessionModel, SessionFileModel, CompanyModel, UserModel
├── services/ S3Service, SocketService (rrweb buffering), AuthService, MailService
├── controllers/ RecordedSessionController, LiveSessionController, KnoxPresenceController
├── jobs/ DeleteExpiredVideos, RoomVideoMerging, RoomVideoProcessing
├── queues/ recording-analysis.queue.js, ms-ai-handoff.queue.js
├── workers/ recording-analysis/ (processor, chunk-fetcher, dom-resolver,
│ intent-synth, stream-stitcher, geo-backfill-runner,
│ visitor-profile-rollup, phantom-form-scrub, extractors/*)
└── routes/ api.js, internalKnox.js
bin/
├── www HTTP server
└── recording-analysis-worker.js BullMQ consumer (separate k8s Deployment)
Data models
| Model | Purpose |
|---|---|
user_sessions | session lifetime, current URL, nested lead_score |
session_files | per-page rrweb recording; chunk_count counter (S3 objects named <n>.ndjson; legacy files[] arrays still honored — app/lib/chunkNames.js resolveChunkNames() reconstructs); recording_status (active/stale/finalized/analyzed, or skipped with skipped_reason, e.g. revivable no_events); first/last event timestamps (first_event_at has no default). Index: compound user_session_id + file_id |
Recording pipeline
Global gate: app/configs/app.js session_recordings_enabled (hardcoded true
at HEAD) short-circuits ingest and playback.
- Browser streams rrweb events →
widget:event(Socket.IO) → buffered in Redis, drained to S3 in 100-event chunks (FLUSH_CHUNK_EVENTS,drainListInChunks— OOM fix); each chunk$incschunk_count. - When a session goes stale, a recording-analysis BullMQ job is
enqueued. Finalize grace is
FINALIZE_GRACE_TTL_SECONDS(default 90s), sweep everyFINALIZE_SWEEP_INTERVAL_MS(default 15s) → ~2min end-to-end. The sweep is bounded to a 2-day window (FINALIZE_SWEEP_MAX_AGE_MSconst), batches capped at 200, adopts status-less backend-created rows, and the crash sweep addresses docs by_id(STALE_FINALIZE_THRESHOLD_MS15min safety net). Since3dcb260it runs once per interval across pods (Redis lockfinalize-sweep-lock), skips event-less rows idle 15 min asno_events(revived by a late chunk viaSessionFileModel.reviveIfNoEventsSkip), and session-file creation upserts instead of findOne+create. The ms-ai-handoff failure marker now matchesanalyzedrows, so failed handoffs actually retry. Details: Session Recording › Orphan rows. - The worker (
bin/recording-analysis-worker.js, concurrency 4, 120s timeout;ms-ai-handoffalso runs at concurrency 4):- fetches chunks (
chunk-fetcher.js— parallel/batched, plus a processor time budget; fixedstitch_timeouton many-chunk sessions), - rebuilds the DOM (
dom-resolver), - extracts intent signals (
intent-synth— high-intent pages, form submits; the form-abandonment extractorextractors/form.jswas rewritten in 2026-07 after minting phantom rows;extractors/scroll.jsandvisitor-profile-rollup.jsalso fixed), Since 2026-10 the form extractor ignores inputs the visitor never focused (page scripts writing hidden/untouched fields) and stamps exact timing (page_entered_at,last_activity_at,submitted_at) — see Per-Session Signals, - after the visitor-profile rollup, a rise in engagement score is posted
fire-and-forget to backend
POST /internal/knox/visit-engagement{ company_id, user_session_id, from, to }(x-api-key: BACKEND_API_KEY;lib/backendVisitorLive.jsnotifyVisitEngagement,c3d05dd) — feeds Agent K's "highly engaged" setting / automation trigger. Deploy after backend, - geo-backfills from visitor IP,
- stitches video frames → MP4 (handbrake),
- uploads to S3,
- enqueues ms-ai-handoff to send the analyzed session + transcripts to ms-ai for enrichment.
- fetches chunks (
Cron sweeps prune expired videos and raw .webm chunks. See
Session Recording for the end-to-end view.
Presence
Per-pod sockets register tabs in a Redis hash session_tabs:{user_session_id}
(90s TTL, renewed every 30s) so any pod can answer "is this visitor live, and on
which tabs?". Since PR #102 (d10da51, 2026-09) the registry is
liveness-filtered, not just TTL'd: the heartbeat writes a per-tab
last_beat, and _getLiveTabs / _getLiveTabCount hide grace-marked or stale
entries and lazily unregister them — the hash key's TTL is shared by every
field, so one live tab kept a dead pod's tabs "warm" and snapshot requests
targeted a tab that could never answer. Disconnect marks disconnected_at and
finalizes after WIDGET_DISCONNECT_GRACE_MS (5000) so hard navigations don't
flap the dashboard. Protocol additions: snapshot-unavailable
(server → viewer when no live tab; widget → server → viewer when not streaming)
and snapshot request retargeting (a request-snapshot for a dead tab is
rewritten to the focused/most-recent live tab with retargeted_from);
user-event fan-out is socket.to(room), not io.to(room) — the widget was
being echoed its own stream; broadcastEvent emits to the company room and
the session room. POST custom events (LiveSessionController) fall back to the
validated top-level company_id for the room (671d1c4) — the recorder
posts it at the body root, not inside value, so page-url stopped reaching
the company room once the global fallback was removed. Tests: test/socket-service-registry.test.js,
test/socket-service-live.test.js (ioredis-mock harness), and
tools/fake-widget/ — seven scenario scripts speaking the exact recorder wire
protocol. Exposed via GET /internal/knox/presence/snapshot and
/internal/knox/presence/visitor. The snapshot carries identity per visitor:
person_id (verified_person_id preferred) + display_name
(KnoxPresenceController).
Storage maintenance
POST /migrations/session-files-cleanup (internalKnox.js →
MigrationController@sessionFilesCleanup) — dry-run by default, requires
older_than_days / min_files to match anything, apply: true to execute;
job-poll endpoint alongside. Called by backend's
POST /admin/storage/session-files-cleanup proxy + Storage Maintenance admin
screen.
POST /migrations/scrub-phantom-forms (+ GET /migrations/scrub-phantom-forms/jobs/:id)
removes phantom "abandoned form" data minted by the old form extractor
(MigrationController → phantom-form-scrub.js). Backend proxy:
POST /admin/storage/scrub-phantom-forms (adminStorageRoutes.js), surfaced in
the same Storage Maintenance panel — dry-run first.
bin/scrub-phantom-form-abandonments.js is local-dev only.
Index lock-step: UserSessionModel index declarations are duplicated in
backend/ms-communication — change them together. 2026-08: dropped
{company_id, has_recordings} and {company_id, is_viewed, updated_at: -1}
(7.9 GB, heartbeat churn); transcript_email_sweep_v2 on ChatModel
(backend + ms-comm) is now partial on transcript_email_sent: false.
Tests
npm test (mocha 11, bumped from 9.2.2 in c4bc34c: mocha 9's vendored
yargs calls require() inside an ES module, a hard error on Node 26, so the
suite couldn't start locally; prod images are node-22 where 9 worked).
package-lock.json is gitignored here and the Dockerfile runs npm install.
Specs that need global.resolveOnce set and restore it around their own run —
mocha requires every spec before running any, so module-scope globals are
last-require-wins.
Realtime
2026-10 presence fixes (fd76e75): the liveness beat is now the newest of
last_beat / last_event_time / ready_time (a stale last_beat used to
outrank a tab streaming events right now), and the tab heartbeat interval runs
without Redis too — the local map is what _getLiveTabs reads on a single
pod, so skipping it made every tab "stale" TAB_STALE_MS after widget-ready
(new-user answered widget-offline, snapshots NACKed). /custom-event/create
(the live stream's large-event fallback, kkSendLargeEvent) accepts 50 MB JSON.
Large live events (c2e402e, 4283459, 9e165d5): live events of
LIVE_EVENT_OFFLOAD_CHARS (32 K) or more are parked in Redis
(live-event:<ref>, LIVE_EVENT_TTL_SECONDS 120) and relayed as an
event_ref pointer instead of riding the adapter stream; the viewer fetches
GET /live-event/:ref (?format=utf16le for raw bytes). request-latest-event
replays a tab's latest pointer (live-latest:<room>:<tab>) to the asking socket
and answers latest-event {found}. The adapter stream is explicitly
maxLen SOCKET_ADAPTER_STREAM_MAX_LEN (10000, the library default — it also
backs connectionStateRecovery). See
Session Recording › Live streaming, 2026-10.
Company room auth (9899eaf, 2026-10-14): a socket joins the
company_id room only with a valid user JWT in handshake.auth.token for a
user who can manage that company (lib/companyAccess.js, a port of backend
GeneralHelper.canManageCompany: admin, owner/member, agency/agency_staff of
the owning agency; blocked users and tokens older than tokens_valid_from
refused). Widget sockets join only their session room. The public HTTP relay
(LiveSessionController) accepts only page-url / send-event, writes into
the session room, and broadcasts to the company room only when that session
belongs to the claimed company. Since ea6d515 a playground share-link
socket_token (scope: 'playground') joins only its own company room, a bad
token gets an unauthorized event, and tokenless desktop sockets are admitted
by query.user_id while SOCKET_LEGACY_DASHBOARD_AUTH ≠ 'false' — see
Socket room auth.
Socket.IO 4.5 with @socket.io/redis-streams-adapter on a dedicated
redis.duplicate() connection (the adapter's blocking XREAD on the shared
client serialized all Redis traffic — prod outage root cause, fixed 2026-07-20).
Admin UI gated behind SOCKET_ADMIN_UI (was unauthenticated in prod).
Cross-service
Reads company/auth config from backend; emits session-finalized + lead scores back to backend; hands analyzed recordings to ms-ai; serves presence to ms-communication and Knox.
Deployment
Docker (Node 22 Alpine, 6 GB heap, port 3003) → ECR → EKS. Two deployments: a
stateless HTTP pod and an autoscaling recording worker (HPA on queue depth, 1–10
pods). Worker env: RECORDING_WORKER_CONCURRENCY, RECORDING_WORKER_JOB_TIMEOUT_MS.
Finalization env: FINALIZE_GRACE_TTL_SECONDS (90), FINALIZE_SWEEP_INTERVAL_MS
(15000); presence env: TAB_HEARTBEAT_INTERVAL_MS (30000), TAB_STALE_MS
(120000), WIDGET_DISCONNECT_GRACE_MS (5000). SERVICE_REDIS=TRUE is
required for any multi-pod deploy — without it there's no adapter, no
cross-pod registry, in-memory buffers, and misrouting; boot logs a warning.
SOCKET_ADMIN_UI gates the Socket.IO admin UI.