Skip to main content

Session Recording

The recording subsystem spans three services: the widget captures, ms-sessions buffers and processes, and ms-ai enriches.

End-to-end flow​

widget (rrweb)
  │  Socket.IO: widget:event
  ▼
ms-sessions HTTP/socket  ──► Redis buffer ──► disk flush (~30s)
  │  session goes "stale"
  ▼
BullMQ: recording-analysis  (worker pod, concurrency 4)
  ├─ dom-resolver     rebuild DOM tree from rrweb chunks
  ├─ intent-synth     high-intent pages, form submits, engagement signals
  ├─ geo-backfill     resolve visitor IP → geo
  ├─ stream-stitcher  frames → MP4 (handbrake / webm-to-mp4 / video-stitch)
  └─ upload to S3
  │  BullMQ: ms-ai-handoff
  ▼
ms-ai  ──► transcript/intent enrichment, signals (knox-signals)
  │
  ▼
backend ◄── lead scores / session-finalized (updates UserSession.lead_score)

Capture (widget)​

rrweb records DOM mutations and interactions. Events are emitted to the host page via postMessage and streamed to ms-sessions over Socket.IO, keyed by company_id + user_session_id. A widget:disconnect on tab close starts a finalization grace period.

Consent gating — when a tenant enables consent gating with the behaviour category, the loader never loads rrweb / live_session.js / recording scripts until the visitor consents (live view and replay; forward-only after a mid-session grant). Chat is never gated.

Kill switch — ms-sessions/app/configs/app.js session_recordings_enabled (guards in SocketService + RecordedSessionController); hardcoded true at HEAD. A per-session chunk cap was added and then reverted after a prod incident (its $expr/$size filter matched nothing and stopped recording after the first chunk) — never reintroduce $expr in update filters there.

Buffer & store (ms-sessions)​

Events land in a Redis buffer and are drained to S3 in 100-event chunks (FLUSH_CHUNK_EVENTS, drainListInChunks — sequential flush with a re-entrancy guard; the old whole-backlog drain OOM'd pods), recorded in session_files (recording_status: active → stale → finalized → analyzed). Chunk bookkeeping uses a counter, not filename arrays: _writeChunk does $inc: { chunk_count: 1 } and names the object <n>.ndjson; readers reconstruct names via app/lib/chunkNames.js resolveChunkNames() (legacy files[] still honored, concatenated first). The HTTP endpoints (POST /sessions, POST /sessions/event) are rate-limited (30/min).

Finalization timing​

SocketService.js: finalize grace FINALIZE_GRACE_TTL_SECONDS (env, default 90s — was 5min) and sweep interval FINALIZE_SWEEP_INTERVAL_MS (env, default 15s — was 30s) give ~2min tab-close → analyzed. The sweep only considers rows touched in the last 2 days (FINALIZE_SWEEP_MAX_AGE_MS, hardcoded const), caps batches at 200 (FINALIZE_SWEEP_*_LIMIT), adopts status-less rows (backend- created lean session_files without recording_status are claimed to active so they're sweepable), and the crash sweep addresses docs by _id (a session_id filter can hit an analyzed twin). STALE_FINALIZE_THRESHOLD_MS (15min) is the crashed-tab safety net.

Analysis (ms-sessions worker)​

A separate worker deployment consumes the recording-analysis queue (HPA on queue depth, 1–10 pods; 120s job timeout). It rebuilds the DOM, extracts intent signals, geo-backfills, stitches the MP4, uploads to S3, then enqueues the ms-ai-handoff.

Enrichment (ms-ai)​

ms-ai receives the analyzed session and produces transcript/intent enrichment and insight signals (knox-signals), which feed visitor profiles and Nox's insight engine.

The same ms-sessions sockets maintain the cross-pod presence registry (session_tabs:{user_session_id} in Redis, 90s TTL) used by Knox and ms-communication to know if a visitor is live. The Knox presence snapshot now carries identity: person_id (verified preferred) + display_name per visitor (KnoxPresenceController). See ms-sessions.

Storage maintenance​

POST /migrations/session-files-cleanup on ms-sessions (internalKnox.js → MigrationController@sessionFilesCleanup) — dry-run by default (apply: true to execute), refuses a whole-collection match without older_than_days / min_files. Super-admins reach it via the backend proxy POST /admin/storage/session-files-cleanup (+ /admin/storage/stats, adminStorageRoutes.js → msSessionsRecordingClient.js) and the Storage Maintenance admin screen (frontend/src/views/admin/StorageMaintenance.vue, dry-run/apply + job polling). Retention policy is still undecided.

Reliability​

  • backend's RecordingBackfillSweeperJob (every 15 min) backfills any SessionFile whose handoff failed.
  • Cron sweeps delete expired videos and raw .webm chunks.
  • Prod-stability pass (2026-07-20): the Socket.IO redis-streams adapter runs on a dedicated duplicated Redis connection (its blocking XREAD was serializing all Redis traffic — outage root cause), the broadcast ping loop was removed, the Socket.IO admin UI is gated behind SOCKET_ADMIN_UI, and partial cron indexes were added on UserSessionModel / SessionAnalysisModel.

Replay​

The dashboard replays recordings with rrweb-player via frontend/src/components/SessionPlayer/ and useLiveReplayer (live sessions can be watched in near-real-time, recorded ones replayed from S3).