Session Recording
The recording subsystem spans three services: the widget
captures, ms-sessions buffers and processes, and
ms-ai enriches.
End-to-end flow
widget (rrweb)
│ Socket.IO: widget:event
▼
ms-sessions HTTP/socket ──► Redis buffer ──► disk flush (~30s)
│ session goes "stale"
▼
BullMQ: recording-analysis (worker pod, concurrency 4)
├─ dom-resolver rebuild DOM tree from rrweb chunks
├─ intent-synth high-intent pages, form submits, engagement signals
├─ geo-backfill resolve visitor IP → geo
├─ stream-stitcher frames → MP4 (handbrake / webm-to-mp4 / video-stitch)
└─ upload to S3
│ BullMQ: ms-ai-handoff
▼
ms-ai ──► transcript/intent enrichment, signals (knox-signals)
│
▼
backend ◄── lead scores / session-finalized (updates UserSession.lead_score)
Capture (widget)
rrweb records DOM mutations and interactions. Events are emitted to the host page
via postMessage and streamed to ms-sessions over Socket.IO, keyed by
company_id + user_session_id. A widget:disconnect on tab close starts a
finalization grace period.
Consent gating — when a tenant enables
consent gating with the behaviour category,
the loader never loads rrweb / live_session.js / recording scripts until the
visitor consents (live view and replay; forward-only after a mid-session
grant). Chat is never gated.
GDPR safe mode (2026-11-10, backend 99b94dcc6) — EU/EEA/UK/CH and
unknown-country visitors (user_sessions.gdpr_restricted) are recorded only
after an explicit behaviour grant (banner, stored choice, kkSetConsent/preset,
Cookiebot/OneTrust/Google Consent Mode), with or without consent gating; a
site/CMP "no" stops recording for anyone mid-page (stopRecording(),
window.__kkRecordingOff). ms-sessions backstops it: SocketService.canRecord
refuses restricted sessions without consent.explicit && consent.behaviour
(allow cached 5 min, refusal 30 s). See GDPR & Privacy.
What is captured — the chat widget's own rrweb recorder was removed (widget
ff382f6); the widget hosts are blocked from the page recording. Recording
scripts load alongside the widget iframe rather than before it (7175e1dd5).
With per-company recording_masking_v2, the shared public/recording-config.js
masks every typed value except allowlisted fields (never passwords/cards/IDs),
honours data-kk-mask / data-kk-unmask, and records errors as kk-error
custom events without console text; field metadata (never values) is reported to
POST /recording/fields. public/widget-iframe.js (customer cross-origin
iframes) follows the masking config but has no consent check of its own.
Kill switch — ms-sessions/app/configs/app.js session_recordings_enabled
(guards in SocketService + RecordedSessionController); hardcoded true at
HEAD. A per-session chunk cap was added and then reverted after a prod
incident (its $expr/$size filter matched nothing and stopped recording after
the first chunk) — never reintroduce $expr in update filters there.
Buffer & store (ms-sessions)
Events land in a Redis buffer and are drained to S3 in 100-event chunks
(FLUSH_CHUNK_EVENTS, drainListInChunks — sequential flush with a re-entrancy
guard; the old whole-backlog drain OOM'd pods), recorded in session_files
(recording_status: active → stale → finalized → analyzed). Chunk bookkeeping
uses a counter, not filename arrays: _writeChunk does
$inc: { chunk_count: 1 } and names the object <n>.ndjson; readers
reconstruct names via app/lib/chunkNames.js resolveChunkNames() (legacy
files[] still honored, concatenated first). The HTTP endpoints
(POST /sessions, POST /sessions/event) are rate-limited (30/min). Body limit
is 10 MB except /sessions/event (50 MB) — public/recoded_session.js
routes any rrweb event over SOCKET_MAX_SIZE_KB (800 KB) there uncompressed
because Socket.IO drops it; a 413 is retried 3× then buffered locally, losing
the FullSnapshot and stopping replay at that point.
Finalization timing
SocketService.js: finalize grace FINALIZE_GRACE_TTL_SECONDS (env, default
90s — was 5min) and sweep interval FINALIZE_SWEEP_INTERVAL_MS (env, default
15s — was 30s) give ~2min tab-close → analyzed. The sweep only considers rows
touched in the last 2 days (FINALIZE_SWEEP_MAX_AGE_MS, hardcoded const), caps
batches at 200 (FINALIZE_SWEEP_*_LIMIT), adopts status-less rows (backend-
created lean session_files without recording_status are claimed to active
so they're sweepable), and the crash sweep addresses docs by _id (a
session_id filter can hit an analyzed twin). STALE_FINALIZE_THRESHOLD_MS
(15min) is the crashed-tab safety net.
Orphan rows (3dcb260, 2026-09-29): rows that never received an event sat
active forever — the crash query filters on last_event_at (not in the
index), so every 15 s sweep fetched and discarded all of them. Now:
- Rows
active, untouched for 15 min, withlast_event_at: nullare set torecording_status: 'skipped',skipped_reason: 'no_events'.SessionFileModel.reviveIfNoEventsSkip()puts one back toactiveif a chunk lands later (flush outage, late adoption)._enqueueAnalysisalso terminates event-less rows asno_eventsinstead of analysing nothing. - One sweep per interval across pods — Redis
SET finalize-sweep-lock NX PX(interval − 1 s; the lock just expires). _initSessionDband the HTTP upload path upsert ($setOnInsert) instead of findOne+create, so concurrent first chunks no longer twin a row (no unique index onsession_id, so a race can still twin — the no-events skip clears it).first_event_athas no null default ($minnever replaces a stored null).
Analysis (ms-sessions worker)
A separate worker deployment consumes the recording-analysis queue (HPA on
queue depth, 1–10 pods; 120s job timeout). It rebuilds the DOM, extracts intent
signals, geo-backfills, stitches the MP4, uploads to S3, then enqueues the
ms-ai-handoff.
Recent fixes (2026-07/08):
stitch_timeouton many-chunk sessions —app/workers/recording-analysis/chunk-fetcher.jsnow fetches chunks in parallel batches, and the processor has a time budget.- Form-abandonment extractor rewritten
(
app/workers/recording-analysis/extractors/form.js, with tests) — the old one minted phantom "abandoned form" rows. Cleanup:POST /migrations/scrub-phantom-forms(+/jobs/:idpoll) oninternalKnox.js→MigrationController, proxied by backendPOST /admin/storage/scrub-phantom-forms(adminStorageRoutes.js) and surfaced in the Storage Maintenance admin panel (dry-run first).bin/scrub-phantom-form-abandonments.jsis local-dev only. - Scroll extractor fix (
extractors/scroll.js) and visitor-profile rollup corrections (visitor-profile-rollup.js). - No typed values in analysis (
e060401, 2026-11-10):dom-resolver.jsdropsattributes.valueexcept as a button caption, and textarea / contenteditable text never becomestextContent, a label or an accessible name;extractors/form.jskeeps length buckets only;extractors/error.jsreadskk-erroras well as console events. Masking-v2 companies get form abandonment fromform_events(FormEventAbandonmentJob) and ms-sessions skips its recording-based trigger for them. - Telemetry (
04be833): no default OTLP endpoint — tracing starts only whenOTEL_EXPORTER_OTLP_ENDPOINT/OTEL_EXPORTER_OTLP_TRACES_ENDPOINTis set.
Enrichment (ms-ai)
ms-ai receives the analyzed session and produces transcript/intent enrichment and
insight signals (knox-signals), which feed visitor profiles and Nox's
insight engine.
Indexes (lock-step warning)
UserSessionModel / ChatModel index declarations are duplicated across
backend, ms-communication, and ms-sessions — change them together or one
service recreates what another dropped. 2026-08 pass:
- Dropped
{company_id, has_recordings}and{company_id, is_viewed, updated_at: -1}(7.9 GB combined;updated_atheartbeat churn) — removed in both ms-sessions and ms-communicationUserSessionModel. transcript_email_sweep_v2redefined as a partial index (partialFilterExpression: { transcript_email_sent: false }) in backend and ms-communicationChatModel.
Live-session accuracy
- Backend
ActiveSessionJob(1 min) reworked — live list no longer shows stale sessions. - Compressed rrweb frames fixed in
backend/public/live_session_for_mobile.js(mobile live view). - Visitor presence for engage/Nox = Redis heartbeat
(
backend/app/lib/sessionPresence.js), replacing the oldnox_live_atfield.
Presence (related)
The same ms-sessions sockets maintain the cross-pod presence registry
(session_tabs:{user_session_id} in Redis, 90s TTL, liveness-filtered per tab
since PR #102 — see ms-sessions › Presence) used by Knox and
ms-communication to know if a visitor is live. The Knox presence snapshot now
carries identity: person_id (verified preferred) + display_name per visitor
(KnoxPresenceController). See ms-sessions.
Storage maintenance
POST /migrations/session-files-cleanup on ms-sessions (internalKnox.js →
MigrationController@sessionFilesCleanup) — dry-run by default (apply: true
to execute), refuses a whole-collection match without older_than_days /
min_files. Super-admins reach it via the backend proxy
POST /admin/storage/session-files-cleanup (+ /admin/storage/stats,
adminStorageRoutes.js → msSessionsRecordingClient.js) and the Storage
Maintenance admin screen (frontend/src/views/admin/StorageMaintenance.vue,
dry-run/apply + job polling).
Per-visitor retention (2026-11-04): backend RecordingRetentionJob (every
15 min, schedular) drives ms-sessions POST /internal/recordings/expire,
deleting the recordings of visitors with no recorded visit for
recording_retention.retention_days (default 60, floor 30; off by default;
Admin → Settings → Recording Retention). A visitor who came back keeps
everything. Details: Recorded Sessions.
Reliability
backend'sRecordingBackfillSweeperJob(every 15 min) rescues anySessionFilewhose finalize/handoff failed. See Sweepers.- Cron sweeps delete expired videos and raw
.webmchunks. - Prod-stability pass (2026-07-20): the Socket.IO redis-streams adapter runs on a
dedicated duplicated Redis connection (its blocking XREAD was serializing
all Redis traffic — outage root cause), the broadcast ping loop was removed,
the Socket.IO admin UI is gated behind
SOCKET_ADMIN_UI, and partial cron indexes were added onUserSessionModel/SessionAnalysisModel.
Sweepers
Until 60118219c (2026-09-29) the backend SessionFileModel was strict and did
not declare recording_status, skipped_reason, finalized_at or
handoff_attempt_count, so every $set from RecordingBackfillSweeperJob
and MsAiHandoffSweeperJob was silently dropped — only updated_at changed,
which kept stuck rows in the ms-sessions sweep window indefinitely and retried
failed handoffs forever. The fields are now declared (no defaults or indexes —
ms-sessions owns row creation and the index list). RecordingBackfillSweeperJob
also:
- lower-bounds
updated_atin every branch (index is{company_id, recording_status, updated_at}), so the scan is finite; - skips
active/stale/finalizedrows with no events asno_events(same reason ms-sessions uses, so a late chunk revives them); - bumps
updated_aton enqueue failure so an always-failing row rotates out of the oldest-first batch.
On the ms-sessions side the handoff failure marker targeted non-analyzed
rows it could never match (the analysis worker sets analyzed before
enqueueing the handoff), so failed handoffs were never retried; it now matches
recording_status: 'analyzed' and starts a new cycle at
handoff_attempt_count: 1.
Replay
Access: ms-sessions recording reads/writes are company-scoped and recordings are
no longer broadcast to every socket (7790b27); call/room video
(GET /video-recordings/:id) needs login + company access (9f4d058). The
live replay iframe is script-free (UNSAFE_replayCanvas: false, frontend
61263b55 — visitor HTML used to run in the dashboard).
The dashboard replays recordings with rrweb-player via
frontend/src/components/SessionPlayer/ and useLiveReplayer (live sessions can
be watched in near-real-time, recorded ones replayed from S3).
rrweb is self-hosted and version-locked at 2.1.1 (33ef2e988): the widget
loads WIDGET_API_URL + '/widget/rrweb-all.min.js' (vendored from
rrweb@2.1.1 dist/rrweb.umd.min.cjs; live-user-session.html loads the same
plus /widget/rrweb.min.css), widget-iframe.js derives the base from
document.currentScript.src and falls back to a pinned CDN
(rrweb@2.0.0-alpha.4, never @latest), and the frontend pins rrweb /
rrweb-player 2.1.1. Recorder, replayer, and player must move together.
public/.gitignore ignores by default — the bundle 404'd until
!rrweb-all.min.js / !rrweb.min.css were added.
Live streaming fixes (PR #2127 / #986 / #102, 2026-09): blockClass regex
fixed (/^tkzx-rr-(block|hidden|no-display|)$/ — the empty alternative matched
tkzx-rr-); live_streaming is now paired with live_streaming_meta {session, at} honoured only for the same session and < 12h; snapshot throttle
KK_SNAPSHOT_MIN_INTERVAL_MS = 2000 with one trailing send; snapshot-sent and
page-url carry tab_id; the widget re-announces on socket_sessions
connect with widget-ready {reconnect: true} + a fresh snapshot; duplicated
tabs (sessionStorage is copied) regenerate tab_id via a
BroadcastChannel('kk_tab_claim') probe. Frontend useLiveReplayer rebases
the sequence only on snapshot/meta and holds incrementals before the first full
snapshot of a cycle — a mousemove used to latch expectedSeq past the
throttled snapshot, which was then dropped as a duplicate ("User's Browser Not
Responding"); auto-follow and auto-recovery are debounced (1.2s). Frontend
gotcha: 'reconnect' is a Manager event in socket.io-client v4 —
socket.on('reconnect') never fires; socket.io.on('reconnect', …) does
(utils/socketManager.js, 0cc93512).
Live streaming, 2026-10 (backend 2b65e49e3 / 5f7e837e3, ms-sessions
c2e402e / 4283459 / 9e165d5, frontend 37525a28 / 3fa41df9 /
3575cf5b / e92f26bc):
- Own rrweb instance —
live_session.jsstreams withwindow.rrwebLive, loaded on demand from/widget/rrweb-live.min.js.CompanyControllerserves that name fromrrweb-all.min.jswith the UMD global renamed, so the live recorder never shares the recording'srrwebglobal. Streaming starts (and a restored stream resumes) only once it has loaded andsocket_sessionsexists (500 ms × 30). Non-checkout full snapshots logSnapshot emitted(seq, size, socket vs HTTP); a large event that fails every retry logs that it never reached the server. - Large events as pointers — the Redis streams adapter copies every
broadcast into one stream all pods read; ~1 MB snapshots there stalled 30 s+
or never arrived ("live session won't load"). ms-sessions now parks any live
event of
LIVE_EVENT_OFFLOAD_CHARS(32 K chars) or more underlive-event:<128-bit ref>(LIVE_EVENT_TTL_SECONDS120, JSON-encoded so the LZ string's lone surrogates survive Redis) and broadcasts{…, event_ref, event_size}instead — socket and HTTPsend-eventpaths alike; single pod / no Redis emits inline. The viewer fetchesGET /live-event/:ref?format=utf16le(raw UTF-16LE, ~40% smaller than JSON; frontendutils/utf16.js). Downloads run one at a time; a queued pointer superseded by a newer one for the same tab is skipped (except seq 1–2, the restart baseline) and a ref relayed twice is fetched once. - Lost-pointer recovery — the latest pointer per room+tab is also kept
(
live-latest:<room>:<tab>). Onsnapshot-sentwith no snapshot the viewer emitsrequest-latest-eventat 1.5 / 3.5 / 7 s; the server answers that socket only with the pointer (user-event) andlatest-event {found, seq}(found: false= the widget's snapshot never reached the server). - Recorder restart (hard navigation) —
useLiveReplayertreats seq 1 meta or a seq 2 full snapshot belowexpectedSeqas a restart and holds the new page's incrementals until its (pointer-fetched, possibly late) snapshot lands; after 4 s without ituseLiveSocketasks again. - Live baseline —
startLive()is deferred to the first event and uses its timestamp, so visitor/dashboard clock skew no longer makes rrweb skip cursor animation (visitor behind) or delay it (visitor ahead).