Skip to main content

Session Recording

The recording subsystem spans three services: the widget captures, ms-sessions buffers and processes, and ms-ai enriches.

End-to-end flow​

widget (rrweb)
│ Socket.IO: widget:event
▼
ms-sessions HTTP/socket ──► Redis buffer ──► disk flush (~30s)
│ session goes "stale"
▼
BullMQ: recording-analysis (worker pod, concurrency 4)
├─ dom-resolver rebuild DOM tree from rrweb chunks
├─ intent-synth high-intent pages, form submits, engagement signals
├─ geo-backfill resolve visitor IP → geo
├─ stream-stitcher frames → MP4 (handbrake / webm-to-mp4 / video-stitch)
└─ upload to S3
│ BullMQ: ms-ai-handoff
▼
ms-ai ──► transcript/intent enrichment, signals (knox-signals)
│
▼
backend ◄── lead scores / session-finalized (updates UserSession.lead_score)

Capture (widget)​

rrweb records DOM mutations and interactions. Events are emitted to the host page via postMessage and streamed to ms-sessions over Socket.IO, keyed by company_id + user_session_id. A widget:disconnect on tab close starts a finalization grace period.

Consent gating — when a tenant enables consent gating with the behaviour category, the loader never loads rrweb / live_session.js / recording scripts until the visitor consents (live view and replay; forward-only after a mid-session grant). Chat is never gated.

GDPR safe mode (2026-11-10, backend 99b94dcc6) — EU/EEA/UK/CH and unknown-country visitors (user_sessions.gdpr_restricted) are recorded only after an explicit behaviour grant (banner, stored choice, kkSetConsent/preset, Cookiebot/OneTrust/Google Consent Mode), with or without consent gating; a site/CMP "no" stops recording for anyone mid-page (stopRecording(), window.__kkRecordingOff). ms-sessions backstops it: SocketService.canRecord refuses restricted sessions without consent.explicit && consent.behaviour (allow cached 5 min, refusal 30 s). See GDPR & Privacy.

What is captured — the chat widget's own rrweb recorder was removed (widget ff382f6); the widget hosts are blocked from the page recording. Recording scripts load alongside the widget iframe rather than before it (7175e1dd5). With per-company recording_masking_v2, the shared public/recording-config.js masks every typed value except allowlisted fields (never passwords/cards/IDs), honours data-kk-mask / data-kk-unmask, and records errors as kk-error custom events without console text; field metadata (never values) is reported to POST /recording/fields. public/widget-iframe.js (customer cross-origin iframes) follows the masking config but has no consent check of its own.

Kill switch — ms-sessions/app/configs/app.js session_recordings_enabled (guards in SocketService + RecordedSessionController); hardcoded true at HEAD. A per-session chunk cap was added and then reverted after a prod incident (its $expr/$size filter matched nothing and stopped recording after the first chunk) — never reintroduce $expr in update filters there.

Buffer & store (ms-sessions)​

Events land in a Redis buffer and are drained to S3 in 100-event chunks (FLUSH_CHUNK_EVENTS, drainListInChunks — sequential flush with a re-entrancy guard; the old whole-backlog drain OOM'd pods), recorded in session_files (recording_status: active → stale → finalized → analyzed). Chunk bookkeeping uses a counter, not filename arrays: _writeChunk does $inc: { chunk_count: 1 } and names the object <n>.ndjson; readers reconstruct names via app/lib/chunkNames.js resolveChunkNames() (legacy files[] still honored, concatenated first). The HTTP endpoints (POST /sessions, POST /sessions/event) are rate-limited (30/min). Body limit is 10 MB except /sessions/event (50 MB) — public/recoded_session.js routes any rrweb event over SOCKET_MAX_SIZE_KB (800 KB) there uncompressed because Socket.IO drops it; a 413 is retried 3× then buffered locally, losing the FullSnapshot and stopping replay at that point.

Finalization timing​

SocketService.js: finalize grace FINALIZE_GRACE_TTL_SECONDS (env, default 90s — was 5min) and sweep interval FINALIZE_SWEEP_INTERVAL_MS (env, default 15s — was 30s) give ~2min tab-close → analyzed. The sweep only considers rows touched in the last 2 days (FINALIZE_SWEEP_MAX_AGE_MS, hardcoded const), caps batches at 200 (FINALIZE_SWEEP_*_LIMIT), adopts status-less rows (backend- created lean session_files without recording_status are claimed to active so they're sweepable), and the crash sweep addresses docs by _id (a session_id filter can hit an analyzed twin). STALE_FINALIZE_THRESHOLD_MS (15min) is the crashed-tab safety net.

Orphan rows (3dcb260, 2026-09-29): rows that never received an event sat active forever — the crash query filters on last_event_at (not in the index), so every 15 s sweep fetched and discarded all of them. Now:

  • Rows active, untouched for 15 min, with last_event_at: null are set to recording_status: 'skipped', skipped_reason: 'no_events'. SessionFileModel.reviveIfNoEventsSkip() puts one back to active if a chunk lands later (flush outage, late adoption). _enqueueAnalysis also terminates event-less rows as no_events instead of analysing nothing.
  • One sweep per interval across pods — Redis SET finalize-sweep-lock NX PX (interval − 1 s; the lock just expires).
  • _initSessionDb and the HTTP upload path upsert ($setOnInsert) instead of findOne+create, so concurrent first chunks no longer twin a row (no unique index on session_id, so a race can still twin — the no-events skip clears it).
  • first_event_at has no null default ($min never replaces a stored null).

Analysis (ms-sessions worker)​

A separate worker deployment consumes the recording-analysis queue (HPA on queue depth, 1–10 pods; 120s job timeout). It rebuilds the DOM, extracts intent signals, geo-backfills, stitches the MP4, uploads to S3, then enqueues the ms-ai-handoff.

Recent fixes (2026-07/08):

  • stitch_timeout on many-chunk sessions — app/workers/recording-analysis/chunk-fetcher.js now fetches chunks in parallel batches, and the processor has a time budget.
  • Form-abandonment extractor rewritten (app/workers/recording-analysis/extractors/form.js, with tests) — the old one minted phantom "abandoned form" rows. Cleanup: POST /migrations/scrub-phantom-forms (+ /jobs/:id poll) on internalKnox.js → MigrationController, proxied by backend POST /admin/storage/scrub-phantom-forms (adminStorageRoutes.js) and surfaced in the Storage Maintenance admin panel (dry-run first). bin/scrub-phantom-form-abandonments.js is local-dev only.
  • Scroll extractor fix (extractors/scroll.js) and visitor-profile rollup corrections (visitor-profile-rollup.js).
  • No typed values in analysis (e060401, 2026-11-10): dom-resolver.js drops attributes.value except as a button caption, and textarea / contenteditable text never becomes textContent, a label or an accessible name; extractors/form.js keeps length buckets only; extractors/error.js reads kk-error as well as console events. Masking-v2 companies get form abandonment from form_events (FormEventAbandonmentJob) and ms-sessions skips its recording-based trigger for them.
  • Telemetry (04be833): no default OTLP endpoint — tracing starts only when OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set.

Enrichment (ms-ai)​

ms-ai receives the analyzed session and produces transcript/intent enrichment and insight signals (knox-signals), which feed visitor profiles and Nox's insight engine.

Indexes (lock-step warning)​

UserSessionModel / ChatModel index declarations are duplicated across backend, ms-communication, and ms-sessions — change them together or one service recreates what another dropped. 2026-08 pass:

  • Dropped {company_id, has_recordings} and {company_id, is_viewed, updated_at: -1} (7.9 GB combined; updated_at heartbeat churn) — removed in both ms-sessions and ms-communication UserSessionModel.
  • transcript_email_sweep_v2 redefined as a partial index (partialFilterExpression: { transcript_email_sent: false }) in backend and ms-communication ChatModel.

Live-session accuracy​

  • Backend ActiveSessionJob (1 min) reworked — live list no longer shows stale sessions.
  • Compressed rrweb frames fixed in backend/public/live_session_for_mobile.js (mobile live view).
  • Visitor presence for engage/Nox = Redis heartbeat (backend/app/lib/sessionPresence.js), replacing the old nox_live_at field.

The same ms-sessions sockets maintain the cross-pod presence registry (session_tabs:{user_session_id} in Redis, 90s TTL, liveness-filtered per tab since PR #102 — see ms-sessions › Presence) used by Knox and ms-communication to know if a visitor is live. The Knox presence snapshot now carries identity: person_id (verified preferred) + display_name per visitor (KnoxPresenceController). See ms-sessions.

Storage maintenance​

POST /migrations/session-files-cleanup on ms-sessions (internalKnox.js → MigrationController@sessionFilesCleanup) — dry-run by default (apply: true to execute), refuses a whole-collection match without older_than_days / min_files. Super-admins reach it via the backend proxy POST /admin/storage/session-files-cleanup (+ /admin/storage/stats, adminStorageRoutes.js → msSessionsRecordingClient.js) and the Storage Maintenance admin screen (frontend/src/views/admin/StorageMaintenance.vue, dry-run/apply + job polling).

Per-visitor retention (2026-11-04): backend RecordingRetentionJob (every 15 min, schedular) drives ms-sessions POST /internal/recordings/expire, deleting the recordings of visitors with no recorded visit for recording_retention.retention_days (default 60, floor 30; off by default; Admin → Settings → Recording Retention). A visitor who came back keeps everything. Details: Recorded Sessions.

Reliability​

  • backend's RecordingBackfillSweeperJob (every 15 min) rescues any SessionFile whose finalize/handoff failed. See Sweepers.
  • Cron sweeps delete expired videos and raw .webm chunks.
  • Prod-stability pass (2026-07-20): the Socket.IO redis-streams adapter runs on a dedicated duplicated Redis connection (its blocking XREAD was serializing all Redis traffic — outage root cause), the broadcast ping loop was removed, the Socket.IO admin UI is gated behind SOCKET_ADMIN_UI, and partial cron indexes were added on UserSessionModel / SessionAnalysisModel.

Sweepers​

Until 60118219c (2026-09-29) the backend SessionFileModel was strict and did not declare recording_status, skipped_reason, finalized_at or handoff_attempt_count, so every $set from RecordingBackfillSweeperJob and MsAiHandoffSweeperJob was silently dropped — only updated_at changed, which kept stuck rows in the ms-sessions sweep window indefinitely and retried failed handoffs forever. The fields are now declared (no defaults or indexes — ms-sessions owns row creation and the index list). RecordingBackfillSweeperJob also:

  • lower-bounds updated_at in every branch (index is {company_id, recording_status, updated_at}), so the scan is finite;
  • skips active/stale/finalized rows with no events as no_events (same reason ms-sessions uses, so a late chunk revives them);
  • bumps updated_at on enqueue failure so an always-failing row rotates out of the oldest-first batch.

On the ms-sessions side the handoff failure marker targeted non-analyzed rows it could never match (the analysis worker sets analyzed before enqueueing the handoff), so failed handoffs were never retried; it now matches recording_status: 'analyzed' and starts a new cycle at handoff_attempt_count: 1.

Replay​

Access: ms-sessions recording reads/writes are company-scoped and recordings are no longer broadcast to every socket (7790b27); call/room video (GET /video-recordings/:id) needs login + company access (9f4d058). The live replay iframe is script-free (UNSAFE_replayCanvas: false, frontend 61263b55 — visitor HTML used to run in the dashboard).

The dashboard replays recordings with rrweb-player via frontend/src/components/SessionPlayer/ and useLiveReplayer (live sessions can be watched in near-real-time, recorded ones replayed from S3).

rrweb is self-hosted and version-locked at 2.1.1 (33ef2e988): the widget loads WIDGET_API_URL + '/widget/rrweb-all.min.js' (vendored from rrweb@2.1.1 dist/rrweb.umd.min.cjs; live-user-session.html loads the same plus /widget/rrweb.min.css), widget-iframe.js derives the base from document.currentScript.src and falls back to a pinned CDN (rrweb@2.0.0-alpha.4, never @latest), and the frontend pins rrweb / rrweb-player 2.1.1. Recorder, replayer, and player must move together. public/.gitignore ignores by default — the bundle 404'd until !rrweb-all.min.js / !rrweb.min.css were added.

Live streaming fixes (PR #2127 / #986 / #102, 2026-09): blockClass regex fixed (/^tkzx-rr-(block|hidden|no-display|)$/ — the empty alternative matched tkzx-rr-); live_streaming is now paired with live_streaming_meta {session, at} honoured only for the same session and < 12h; snapshot throttle KK_SNAPSHOT_MIN_INTERVAL_MS = 2000 with one trailing send; snapshot-sent and page-url carry tab_id; the widget re-announces on socket_sessions connect with widget-ready {reconnect: true} + a fresh snapshot; duplicated tabs (sessionStorage is copied) regenerate tab_id via a BroadcastChannel('kk_tab_claim') probe. Frontend useLiveReplayer rebases the sequence only on snapshot/meta and holds incrementals before the first full snapshot of a cycle — a mousemove used to latch expectedSeq past the throttled snapshot, which was then dropped as a duplicate ("User's Browser Not Responding"); auto-follow and auto-recovery are debounced (1.2s). Frontend gotcha: 'reconnect' is a Manager event in socket.io-client v4 — socket.on('reconnect') never fires; socket.io.on('reconnect', …) does (utils/socketManager.js, 0cc93512).

Live streaming, 2026-10 (backend 2b65e49e3 / 5f7e837e3, ms-sessions c2e402e / 4283459 / 9e165d5, frontend 37525a28 / 3fa41df9 / 3575cf5b / e92f26bc):

  • Own rrweb instance — live_session.js streams with window.rrwebLive, loaded on demand from /widget/rrweb-live.min.js. CompanyController serves that name from rrweb-all.min.js with the UMD global renamed, so the live recorder never shares the recording's rrweb global. Streaming starts (and a restored stream resumes) only once it has loaded and socket_sessions exists (500 ms × 30). Non-checkout full snapshots log Snapshot emitted (seq, size, socket vs HTTP); a large event that fails every retry logs that it never reached the server.
  • Large events as pointers — the Redis streams adapter copies every broadcast into one stream all pods read; ~1 MB snapshots there stalled 30 s+ or never arrived ("live session won't load"). ms-sessions now parks any live event of LIVE_EVENT_OFFLOAD_CHARS (32 K chars) or more under live-event:<128-bit ref> (LIVE_EVENT_TTL_SECONDS 120, JSON-encoded so the LZ string's lone surrogates survive Redis) and broadcasts {…, event_ref, event_size} instead — socket and HTTP send-event paths alike; single pod / no Redis emits inline. The viewer fetches GET /live-event/:ref?format=utf16le (raw UTF-16LE, ~40% smaller than JSON; frontend utils/utf16.js). Downloads run one at a time; a queued pointer superseded by a newer one for the same tab is skipped (except seq 1–2, the restart baseline) and a ref relayed twice is fetched once.
  • Lost-pointer recovery — the latest pointer per room+tab is also kept (live-latest:<room>:<tab>). On snapshot-sent with no snapshot the viewer emits request-latest-event at 1.5 / 3.5 / 7 s; the server answers that socket only with the pointer (user-event) and latest-event {found, seq} (found: false = the widget's snapshot never reached the server).
  • Recorder restart (hard navigation) — useLiveReplayer treats seq 1 meta or a seq 2 full snapshot below expectedSeq as a restart and holds the new page's incrementals until its (pointer-fetched, possibly late) snapshot lands; after 4 s without it useLiveSocket asks again.
  • Live baseline — startLive() is deferred to the first event and uses its timestamp, so visitor/dashboard clock skew no longer makes rrweb skip cursor animation (visitor behind) or delay it (visitor ahead).