Skip to main content

Session Recording

The recording subsystem spans three services: the widget captures, ms-sessions buffers and processes, and ms-ai enriches.

End-to-end flow​

widget (rrweb)
  │  Socket.IO: widget:event
  ▼
ms-sessions HTTP/socket  ──► Redis buffer ──► disk flush (~30s)
  │  session goes "stale"
  ▼
BullMQ: recording-analysis  (worker pod, concurrency 4)
  ├─ dom-resolver     rebuild DOM tree from rrweb chunks
  ├─ intent-synth     high-intent pages, form submits, engagement signals
  ├─ geo-backfill     resolve visitor IP → geo
  ├─ stream-stitcher  frames → MP4 (handbrake / webm-to-mp4 / video-stitch)
  └─ upload to S3
  │  BullMQ: ms-ai-handoff
  ▼
ms-ai  ──► transcript/intent enrichment, signals (knox-signals)
  │
  ▼
backend ◄── lead scores / session-finalized (updates UserSession.lead_score)

Capture (widget)​

rrweb records DOM mutations and interactions. Events are emitted to the host page via postMessage and streamed to ms-sessions over Socket.IO, keyed by company_id + user_session_id. A widget:disconnect on tab close starts a finalization grace period.

Consent gating — when a tenant enables consent gating with the behaviour category, the loader never loads rrweb / live_session.js / recording scripts until the visitor consents (live view and replay; forward-only after a mid-session grant). Chat is never gated.

Kill switch — ms-sessions/app/configs/app.js session_recordings_enabled (guards in SocketService + RecordedSessionController); hardcoded true at HEAD. A per-session chunk cap was added and then reverted after a prod incident (its $expr/$size filter matched nothing and stopped recording after the first chunk) — never reintroduce $expr in update filters there.

Buffer & store (ms-sessions)​

Events land in a Redis buffer and are drained to S3 in 100-event chunks (FLUSH_CHUNK_EVENTS, drainListInChunks — sequential flush with a re-entrancy guard; the old whole-backlog drain OOM'd pods), recorded in session_files (recording_status: active → stale → finalized → analyzed). Chunk bookkeeping uses a counter, not filename arrays: _writeChunk does $inc: { chunk_count: 1 } and names the object <n>.ndjson; readers reconstruct names via app/lib/chunkNames.js resolveChunkNames() (legacy files[] still honored, concatenated first). The HTTP endpoints (POST /sessions, POST /sessions/event) are rate-limited (30/min).

Finalization timing​

SocketService.js: finalize grace FINALIZE_GRACE_TTL_SECONDS (env, default 90s — was 5min) and sweep interval FINALIZE_SWEEP_INTERVAL_MS (env, default 15s — was 30s) give ~2min tab-close → analyzed. The sweep only considers rows touched in the last 2 days (FINALIZE_SWEEP_MAX_AGE_MS, hardcoded const), caps batches at 200 (FINALIZE_SWEEP_*_LIMIT), adopts status-less rows (backend- created lean session_files without recording_status are claimed to active so they're sweepable), and the crash sweep addresses docs by _id (a session_id filter can hit an analyzed twin). STALE_FINALIZE_THRESHOLD_MS (15min) is the crashed-tab safety net.

Analysis (ms-sessions worker)​

A separate worker deployment consumes the recording-analysis queue (HPA on queue depth, 1–10 pods; 120s job timeout). It rebuilds the DOM, extracts intent signals, geo-backfills, stitches the MP4, uploads to S3, then enqueues the ms-ai-handoff.

Recent fixes (2026-07/08):

  • stitch_timeout on many-chunk sessions — app/workers/recording-analysis/chunk-fetcher.js now fetches chunks in parallel batches, and the processor has a time budget.
  • Form-abandonment extractor rewritten (app/workers/recording-analysis/extractors/form.js, with tests) — the old one minted phantom "abandoned form" rows. Cleanup: POST /migrations/scrub-phantom-forms (+ /jobs/:id poll) on internalKnox.js → MigrationController, proxied by backend POST /admin/storage/scrub-phantom-forms (adminStorageRoutes.js) and surfaced in the Storage Maintenance admin panel (dry-run first). bin/scrub-phantom-form-abandonments.js is local-dev only.
  • Scroll extractor fix (extractors/scroll.js) and visitor-profile rollup corrections (visitor-profile-rollup.js).

Enrichment (ms-ai)​

ms-ai receives the analyzed session and produces transcript/intent enrichment and insight signals (knox-signals), which feed visitor profiles and Nox's insight engine.

Indexes (lock-step warning)​

UserSessionModel / ChatModel index declarations are duplicated across backend, ms-communication, and ms-sessions — change them together or one service recreates what another dropped. 2026-08 pass:

  • Dropped {company_id, has_recordings} and {company_id, is_viewed, updated_at: -1} (7.9 GB combined; updated_at heartbeat churn) — removed in both ms-sessions and ms-communication UserSessionModel.
  • transcript_email_sweep_v2 redefined as a partial index (partialFilterExpression: { transcript_email_sent: false }) in backend and ms-communication ChatModel.

Live-session accuracy​

  • Backend ActiveSessionJob (1 min) reworked — live list no longer shows stale sessions.
  • Compressed rrweb frames fixed in backend/public/live_session_for_mobile.js (mobile live view).
  • Visitor presence for engage/Nox = Redis heartbeat (backend/app/lib/sessionPresence.js), replacing the old nox_live_at field.

The same ms-sessions sockets maintain the cross-pod presence registry (session_tabs:{user_session_id} in Redis, 90s TTL, liveness-filtered per tab since PR #102 — see ms-sessions › Presence) used by Knox and ms-communication to know if a visitor is live. The Knox presence snapshot now carries identity: person_id (verified preferred) + display_name per visitor (KnoxPresenceController). See ms-sessions.

Storage maintenance​

POST /migrations/session-files-cleanup on ms-sessions (internalKnox.js → MigrationController@sessionFilesCleanup) — dry-run by default (apply: true to execute), refuses a whole-collection match without older_than_days / min_files. Super-admins reach it via the backend proxy POST /admin/storage/session-files-cleanup (+ /admin/storage/stats, adminStorageRoutes.js → msSessionsRecordingClient.js) and the Storage Maintenance admin screen (frontend/src/views/admin/StorageMaintenance.vue, dry-run/apply + job polling). Retention policy is still undecided.

Reliability​

  • backend's RecordingBackfillSweeperJob (every 15 min) rescues any SessionFile whose finalize/handoff failed.
  • Cron sweeps delete expired videos and raw .webm chunks.
  • Prod-stability pass (2026-07-20): the Socket.IO redis-streams adapter runs on a dedicated duplicated Redis connection (its blocking XREAD was serializing all Redis traffic — outage root cause), the broadcast ping loop was removed, the Socket.IO admin UI is gated behind SOCKET_ADMIN_UI, and partial cron indexes were added on UserSessionModel / SessionAnalysisModel.

Replay​

The dashboard replays recordings with rrweb-player via frontend/src/components/SessionPlayer/ and useLiveReplayer (live sessions can be watched in near-real-time, recorded ones replayed from S3).

rrweb is self-hosted and version-locked at 2.1.1 (33ef2e988): the widget loads WIDGET_API_URL + '/widget/rrweb-all.min.js' (vendored from rrweb@2.1.1 dist/rrweb.umd.min.cjs; live-user-session.html loads the same plus /widget/rrweb.min.css), widget-iframe.js derives the base from document.currentScript.src and falls back to a pinned CDN (rrweb@2.0.0-alpha.4, never @latest), and the frontend pins rrweb / rrweb-player 2.1.1. Recorder, replayer, and player must move together. public/.gitignore ignores by default — the bundle 404'd until !rrweb-all.min.js / !rrweb.min.css were added.

Live streaming fixes (PR #2127 / #986 / #102, 2026-09): blockClass regex fixed (/^tkzx-rr-(block|hidden|no-display|)$/ — the empty alternative matched tkzx-rr-); live_streaming is now paired with live_streaming_meta {session, at} honoured only for the same session and < 12h; snapshot throttle KK_SNAPSHOT_MIN_INTERVAL_MS = 2000 with one trailing send; snapshot-sent and page-url carry tab_id; the widget re-announces on socket_sessions connect with widget-ready {reconnect: true} + a fresh snapshot; duplicated tabs (sessionStorage is copied) regenerate tab_id via a BroadcastChannel('kk_tab_claim') probe. Frontend useLiveReplayer rebases the sequence only on snapshot/meta and holds incrementals before the first full snapshot of a cycle — a mousemove used to latch expectedSeq past the throttled snapshot, which was then dropped as a duplicate ("User's Browser Not Responding"); auto-follow and auto-recovery are debounced (1.2s). Frontend gotcha: 'reconnect' is a Manager event in socket.io-client v4 — socket.on('reconnect') never fires; socket.io.on('reconnect', …) does (utils/socketManager.js, 0cc93512).