Skip to main content

ms-sessions — Recording & Presence

Captures visitor sessions (rrweb), turns them into analyzed recordings, and is the source of truth for visitor presence. A Nodevel service on MongoDB + Redis + S3.

  • Port: 3003. A separate recording-analysis worker process runs the heavy pipeline.

Responsibilities​

  • Capture — receive rrweb DOM events from the widget over Socket.IO, buffer in Redis, flush to disk, enqueue for processing.
  • Video synthesis — stitch event streams into MP4 (ffmpeg / webm-to-mp4 / video-stitch / handbrake), tag for AI analysis.
  • Session state — user_sessions, session_files, lead scoring, visitor profiles.
  • Presence — a cross-pod tab registry queried by ms-ai/Knox and ms-communication to know if a visitor is live.
  • Lead qualification — engagement metrics (pages, time on high-intent pages, chat/call acceptance, bookings).

Boot flow​

Standard Nodevel; app/hooks.js boot() loads Redis, Auth, Mongo, Cron, Logger, Mail, Permissions, Socket, S3, and a 30 req/min rate limiter on session endpoints. Body parsers are capped at 10 MB, except /sessions/event at 50 MB (the large-rrweb-event fallback — see Session Recording); they were 1000 MB. express-fileupload was removed (c6d54e1): it was mounted with no options (unbounded multipart buffered in memory) and no route reads req.files (app/helpers/UploadHelper.js is unregistered dead code). The HTTP server binds port 3003; the worker boots the same app with __skip_server = true.

Directory map​

app/
├── models/ UserSessionModel, SessionFileModel, CompanyModel, UserModel
├── services/ S3Service, SocketService (rrweb buffering), AuthService, MailService
├── controllers/ RecordedSessionController, LiveSessionController, KnoxPresenceController
├── jobs/ DeleteExpiredVideos, RoomVideoMerging, RoomVideoProcessing
├── queues/ recording-analysis.queue.js, ms-ai-handoff.queue.js
├── workers/ recording-analysis/ (processor, chunk-fetcher, dom-resolver,
│ intent-synth, stream-stitcher, geo-backfill-runner,
│ visitor-profile-rollup, phantom-form-scrub, extractors/*)
└── routes/ api.js, internalKnox.js
bin/
├── www HTTP server
└── recording-analysis-worker.js BullMQ consumer (separate k8s Deployment)

Data models​

ModelPurpose
user_sessionssession lifetime, current URL, nested lead_score
session_filesper-page rrweb recording; chunk_count counter (S3 objects named <n>.ndjson; legacy files[] arrays still honored — app/lib/chunkNames.js resolveChunkNames() reconstructs); recording_status (active/stale/finalized/analyzed, or skipped with skipped_reason, e.g. revivable no_events); first/last event timestamps (first_event_at has no default). Index: compound user_session_id + file_id

Recording pipeline​

Global gate: app/configs/app.js session_recordings_enabled (hardcoded true at HEAD) short-circuits ingest and playback.

  1. Browser streams rrweb events → widget:event (Socket.IO) → buffered in Redis, drained to S3 in 100-event chunks (FLUSH_CHUNK_EVENTS, drainListInChunks — OOM fix); each chunk $incs chunk_count.
  2. When a session goes stale, a recording-analysis BullMQ job is enqueued. Finalize grace is FINALIZE_GRACE_TTL_SECONDS (default 90s), sweep every FINALIZE_SWEEP_INTERVAL_MS (default 15s) → ~2min end-to-end. The sweep is bounded to a 2-day window (FINALIZE_SWEEP_MAX_AGE_MS const), batches capped at 200, adopts status-less backend-created rows, and the crash sweep addresses docs by _id (STALE_FINALIZE_THRESHOLD_MS 15min safety net). Since 3dcb260 it runs once per interval across pods (Redis lock finalize-sweep-lock), skips event-less rows idle 15 min as no_events (revived by a late chunk via SessionFileModel.reviveIfNoEventsSkip), and session-file creation upserts instead of findOne+create. The ms-ai-handoff failure marker now matches analyzed rows, so failed handoffs actually retry. Details: Session Recording › Orphan rows.
  3. The worker (bin/recording-analysis-worker.js, concurrency 4, 120s timeout; ms-ai-handoff also runs at concurrency 4):
    • fetches chunks (chunk-fetcher.js — parallel/batched, plus a processor time budget; fixed stitch_timeout on many-chunk sessions),
    • rebuilds the DOM (dom-resolver),
    • extracts intent signals (intent-synth — high-intent pages, form submits; the form-abandonment extractor extractors/form.js was rewritten in 2026-07 after minting phantom rows; extractors/scroll.js and visitor-profile-rollup.js also fixed), Since 2026-10 the form extractor ignores inputs the visitor never focused (page scripts writing hidden/untouched fields) and stamps exact timing (page_entered_at, last_activity_at, submitted_at) — see Per-Session Signals,
    • after the visitor-profile rollup, a rise in engagement score is posted fire-and-forget to backend POST /internal/knox/visit-engagement { company_id, user_session_id, from, to } (x-api-key: BACKEND_API_KEY; lib/backendVisitorLive.js notifyVisitEngagement, c3d05dd) — feeds Agent K's "highly engaged" setting / automation trigger. Deploy after backend,
    • geo-backfills from visitor IP,
    • stitches video frames → MP4 (handbrake),
    • uploads to S3,
    • enqueues ms-ai-handoff to send the analyzed session + transcripts to ms-ai for enrichment.

Cron sweeps prune expired videos and raw .webm chunks. See Session Recording for the end-to-end view.

Presence​

Per-pod sockets register tabs in a Redis hash session_tabs:{user_session_id} (90s TTL, renewed every 30s) so any pod can answer "is this visitor live, and on which tabs?". Since PR #102 (d10da51, 2026-09) the registry is liveness-filtered, not just TTL'd: the heartbeat writes a per-tab last_beat, and _getLiveTabs / _getLiveTabCount hide grace-marked or stale entries and lazily unregister them — the hash key's TTL is shared by every field, so one live tab kept a dead pod's tabs "warm" and snapshot requests targeted a tab that could never answer. Disconnect marks disconnected_at and finalizes after WIDGET_DISCONNECT_GRACE_MS (5000) so hard navigations don't flap the dashboard. Protocol additions: snapshot-unavailable (server → viewer when no live tab; widget → server → viewer when not streaming) and snapshot request retargeting (a request-snapshot for a dead tab is rewritten to the focused/most-recent live tab with retargeted_from); user-event fan-out is socket.to(room), not io.to(room) — the widget was being echoed its own stream; broadcastEvent emits to the company room and the session room. POST custom events (LiveSessionController) fall back to the validated top-level company_id for the room (671d1c4) — the recorder posts it at the body root, not inside value, so page-url stopped reaching the company room once the global fallback was removed. Tests: test/socket-service-registry.test.js, test/socket-service-live.test.js (ioredis-mock harness), and tools/fake-widget/ — seven scenario scripts speaking the exact recorder wire protocol. Exposed via GET /internal/knox/presence/snapshot and /internal/knox/presence/visitor. The snapshot carries identity per visitor: person_id (verified_person_id preferred) + display_name (KnoxPresenceController).

Storage maintenance​

POST /migrations/session-files-cleanup (internalKnox.js → MigrationController@sessionFilesCleanup) — dry-run by default, requires older_than_days / min_files to match anything, apply: true to execute; job-poll endpoint alongside. Called by backend's POST /admin/storage/session-files-cleanup proxy + Storage Maintenance admin screen.

POST /migrations/scrub-phantom-forms (+ GET /migrations/scrub-phantom-forms/jobs/:id) removes phantom "abandoned form" data minted by the old form extractor (MigrationController → phantom-form-scrub.js). Backend proxy: POST /admin/storage/scrub-phantom-forms (adminStorageRoutes.js), surfaced in the same Storage Maintenance panel — dry-run first. bin/scrub-phantom-form-abandonments.js is local-dev only.

Index lock-step: UserSessionModel index declarations are duplicated in backend/ms-communication — change them together. 2026-08: dropped {company_id, has_recordings} and {company_id, is_viewed, updated_at: -1} (7.9 GB, heartbeat churn); transcript_email_sweep_v2 on ChatModel (backend + ms-comm) is now partial on transcript_email_sent: false.

Tests​

npm test (mocha 11, bumped from 9.2.2 in c4bc34c: mocha 9's vendored yargs calls require() inside an ES module, a hard error on Node 26, so the suite couldn't start locally; prod images are node-22 where 9 worked). package-lock.json is gitignored here and the Dockerfile runs npm install. Specs that need global.resolveOnce set and restore it around their own run — mocha requires every spec before running any, so module-scope globals are last-require-wins.

Realtime​

2026-10 presence fixes (fd76e75): the liveness beat is now the newest of last_beat / last_event_time / ready_time (a stale last_beat used to outrank a tab streaming events right now), and the tab heartbeat interval runs without Redis too — the local map is what _getLiveTabs reads on a single pod, so skipping it made every tab "stale" TAB_STALE_MS after widget-ready (new-user answered widget-offline, snapshots NACKed). /custom-event/create (the live stream's large-event fallback, kkSendLargeEvent) accepts 50 MB JSON.

Large live events (c2e402e, 4283459, 9e165d5): live events of LIVE_EVENT_OFFLOAD_CHARS (32 K) or more are parked in Redis (live-event:<ref>, LIVE_EVENT_TTL_SECONDS 120) and relayed as an event_ref pointer instead of riding the adapter stream; the viewer fetches GET /live-event/:ref (?format=utf16le for raw bytes). request-latest-event replays a tab's latest pointer (live-latest:<room>:<tab>) to the asking socket and answers latest-event {found}. The adapter stream is explicitly maxLen SOCKET_ADAPTER_STREAM_MAX_LEN (10000, the library default — it also backs connectionStateRecovery). See Session Recording › Live streaming, 2026-10.

Company room auth (9899eaf, 2026-10-14): a socket joins the company_id room only with a valid user JWT in handshake.auth.token for a user who can manage that company (lib/companyAccess.js, a port of backend GeneralHelper.canManageCompany: admin, owner/member, agency/agency_staff of the owning agency; blocked users and tokens older than tokens_valid_from refused). Widget sockets join only their session room. The public HTTP relay (LiveSessionController) accepts only page-url / send-event, writes into the session room, and broadcasts to the company room only when that session belongs to the claimed company. Since ea6d515 a playground share-link socket_token (scope: 'playground') joins only its own company room, a bad token gets an unauthorized event, and tokenless desktop sockets are admitted by query.user_id while SOCKET_LEGACY_DASHBOARD_AUTH ≠ 'false' — see Socket room auth.

Socket.IO 4.5 with @socket.io/redis-streams-adapter on a dedicated redis.duplicate() connection (the adapter's blocking XREAD on the shared client serialized all Redis traffic — prod outage root cause, fixed 2026-07-20). Admin UI gated behind SOCKET_ADMIN_UI (was unauthenticated in prod).

Cross-service​

Reads company/auth config from backend; emits session-finalized + lead scores back to backend; hands analyzed recordings to ms-ai; serves presence to ms-communication and Knox.

Deployment​

Docker (Node 22 Alpine, 6 GB heap, port 3003) → ECR → EKS. Two deployments: a stateless HTTP pod and an autoscaling recording worker (HPA on queue depth, 1–10 pods). Worker env: RECORDING_WORKER_CONCURRENCY, RECORDING_WORKER_JOB_TIMEOUT_MS. Finalization env: FINALIZE_GRACE_TTL_SECONDS (90), FINALIZE_SWEEP_INTERVAL_MS (15000); presence env: TAB_HEARTBEAT_INTERVAL_MS (30000), TAB_STALE_MS (120000), WIDGET_DISCONNECT_GRACE_MS (5000). SERVICE_REDIS=TRUE is required for any multi-pod deploy — without it there's no adapter, no cross-pod registry, in-memory buffers, and misrouting; boot logs a warning. SOCKET_ADMIN_UI gates the Socket.IO admin UI.