Skip to main content

Collections & Tables Reference

Audience: Engineering · QA · Support · Where in app: Internal · Plan availability: All plans

A high-density map of where data lives. Each service owns its own persistence — there is no cross-service shared schema; data crosses boundaries via API calls and shared IDs (company_id, user_session_id, person_id, file_id), never database joins. Use this page when you need to know which store holds a given record, or how to follow a record from one store to another.

Engineering deep-dive: Data Stores covers which service owns which database and the schema features (pgvector HNSW, JSONB, soft-deletes). This page is the collection/table catalogue. For the visitor-specific records (UserSession / VisitorProfile / SessionAnalysis / Person) see Visitor data; for companies.* config see Company configuration.

MongoDB — backend, ms-communication, ms-sessions​

The three Nodevel services use MongoDB via Mongoose (locally mongo:7.0; backend is the largest schema, ~70 models).

Core tenant + identity​

CollectionWhat
companiesTenants. Owns all per-tenant config (see Company configuration).
usersTeam members (admins + agents). A self-serve signup holds its business details in pending_company until onboarding answers "my own business" (POST /user/company creates the company); an agency signup never gets a company (ee0dff872).
peopleGlobal Person records (name/email/phone/match_keys).
person_profilesPer-tenant Person view (extracted_facts, form_submissions, bookings, crm_enrichment).
user_sessionsOne row per visit. widget_user_session_id, lead_score, page visits, contact info, traffic source.
user_devicesDevice fingerprints for identity verification (also APN/FCM push tokens in ms-communication).
person_ip_signals / session_ipsIP-assisted identity-resolution signals.
playground_companiesSandbox tenants.

Communication​

CollectionWhat
chatsFinal chat records (transcript messages[], knox enrichment). Every website visit creates an empty one, so the 5-min sweeps run on partial indexes that only hold chats with a message: transcript_email_sweep_v4 ({last_message_at, created_at}, also declared in ms-communication so its autoIndex matches) and fact_extraction_sweep_v2 (abba5cf9e; the jobs repeat the last_message_at > HAS_MESSAGE_FLOOR term and hint the index).
service_roomsActive session containers (chat + call). call_logs (select:false, last 1000 entries) = per-call step log written by ms-communication CallLogHelper — server lifecycle, socket events, widget + dashboard call-log steps; read via /admin/calls/logs.
chat_queuePending chats waiting for an agent.
chatbotsFlow-builder definitions.
inbound_outbound_callsCall records (Twilio SID, recording URL, transcript) — in ms-communication.
call_eventsReal-time call event log.

Sessions & recording (ms-sessions)​

CollectionWhat
session_filesrrweb recordings — chunk metadata, S3 paths, recording_status.
session_analysesPer-visit analysis artifact.
visitor_profilesCross-visit rollup keyed on (company_id, user_session_id).

Privacy / GDPR​

See GDPR.

CollectionWhat
consent_eventsAppend-only proof of each visitor consent decision (categories, source, banner version); latest per session mirrored on user_sessions.consent.
person_erasuresProof of each erasure (ids + counts only); ConversationPurgeJob moves it pending_vendors → done. Also the never-re-identify stop list.
person_exportsAccess requests built by PersonExportJob; file removed after 7 days (expired), row kept.
recording_fieldsForm fields the recorder has seen (never values) — feeds Settings → Recording → Visible fields.
form_eventsStructured form behaviour (viewed / focused / completed / exit / submitted, never values), read by FormEventAbandonmentJob; 30 days.
snitcher_reuse_cacheOne-hour reuse of a company-level lookup for anonymous GDPR-mode visits, keyed by sha256(company + IP).

Bookings & leads​

CollectionWhat
bookingsAll meetings (Google / Calendly / GHL / manual).
form_submissionsCaptured forms (audit).
form_abandonment_reviewsSuper-admin verdicts on forms the recording analyser flagged as abandoned, keyed (file_id, form_index) with a history[] — see Identity Graph admin.
leadsOne Leads-page row per lead, rebuilt from chats, calls, forms and Agent K (LeadsController); only edits / deleted_at are tenant-written — see Leads.
tags / contact_tagsA company's lead tags (slug = lower-cased name) · one person's tags keyed by contact keys, shared by Leads, campaigns and automations — see Lead tags.
campaigns / campaign_leadsOutbound campaigns (steps compiled to an automation graph) · one lead per campaign, queued → active → done/skipped — see Campaigns.
form_leadsUnlinked website-form leads from GDPR-restricted visitors (not tied to session, device or identity graph).

Agent K & automations​

CollectionWhat
agent_k_prospectsThe Agent K queue — every feeder's leads (source outbound / reactivation / knockid / import) through trickle → push.
agent_k_importsA tenant-uploaded lead list; its rows wait in agent_k_prospects (source: 'import') and AgentKImportJob releases them.
do_not_contactsPer-company do-not-contact / unsubscribe list, honoured by every agent.
lead_lifecycle_eventsAppend-only Agent K lead stages (pushed → sent → clicked → replied → booked).
funnel_contactsNox funnel stage state, one per (company, contact), keyed by ghl_contact_id or person_id.
automation_rules / automation_events / automation_claims / automation_contactsAutomation workflows · run log · exactly-once claims and stop markers · non-Agent-K people an automation runs on — see Automations. Re-engage has no collection of its own (company.ai_agent_k.re_engage — see Re-engage).

CRM / integrations / enrichment​

CollectionWhat
webhooksOutbound webhook subscriptions.
webhook_logsWebhook delivery audit (the old webhook_events catalog model was removed — the catalogue lives in code, app/configs/webhookEvents.js).
credentialsCredential vault — every tenant/agency OAuth token and API key, one SecretCrypto blob per owner + provider (data select:false); credential_keys holds key fingerprints. Only via CredentialVault — see Security & Secrets.
hubspot_requestsHubSpot sync audit.
hubspot_oauth_handshakesSingle-use HubSpot install state (1h).
outbound_call_ghl_syncGHL outbound-call sync state.
clearbit_sessions, rb2b_sessions, snitcher_sessions, apollo_recordsB2B enrichment caches (HappierLeads was removed 2026-09).

AI & automation​

CollectionWhat
ai_conversationsQ+A training pairs (FAQ).
ai_credit_logsEvery credit consume/refill (180-day expiry).
ai_auto_trigger_messagesAuto-trigger message templates.
nox_actionsNox-initiated action records (NoxActionModel).
nox_* (Nox engine v2)nox_days, nox_plan_items, nox_rapport, nox_email_accounts / nox_email_messages, nox_tasks, nox_group_followups, nox_admin_calls, nox_contact_notes, nox_stalls, nox_event_outbox, nox_invoke_keys, nox_suggestion_decisions, nox_agentk_changes, nox_agent_lessons / nox_reply_lessons / nox_sales_lessons, plus sales_reply_drafts — see Nox engine v2.
knox_visitor_directivesAuto-engage-on-return directives (KnoxVisitorDirectiveModel).
knox_visitor_watch_eventsReturn-visit watch events (KnoxVisitorWatchEventModel).
identity_graph_defaultsSingleton global IG config.

Billing & subscription​

CollectionWhat
subscriptionsActive subscriptions (credits, expiry, trial).
packagesProduct tier SKUs. Agency client packages carry archived_at (archived = kept by existing clients, no longer offered; /agencies/client-packages/:id/{usage,archive,unarchive}, delete only when unused — c0a749e47).
addonsAdd-on SKUs.
invoicesStripe invoice mirror.
agencies, agency_packages, agency_addons, agency_subscriptionsReseller objects.
appsumo_packages / appsumo_licensesAppSumo tiers (display price, no Stripe product) · one row per license key → company — see AppSumo.

Operational​

CollectionWhat
notificationsIn-app notifications.
admin_notificationsAdmin broadcasts (optionally an alert popup — popup subdoc); popup_receipts tracks each user's views/snoozes/clicks.
email_logsEmail send audit.
api_requestsAPI call audit (credential headers dropped, secret-looking body keys [redacted] — lib/requestRedaction.js).
audit_logsRecord-change audit trail on 39 models (180-day TTL) — see Audit Trail.
logsGeneral backend activity log.
filesUploaded assets (videos, logos, PDFs). Private folders (KB docs, certification proof, chat/WhatsApp attachments, certificates, product-update videos) have a private ACL and no url — served via signed links.
permissions, rolesRBAC.
teamsTeam groupings.
settingsGlobal key-value config: admin settings, Feature Flags platform switches (agency_client_tickets, nox_quality_check_enabled, nox_voice_server_reports, …; one-minute cache), job settings (recording_retention, archived_company_deletion, partner_hub, …).
admin_allowed_ipsSuper-admin IP allowlist.
ops_*, product_update_*Ops workspace: Zij cards/stages/scrum notes, support tickets + messages, AI-draft approvals, roadmap votes/requests, announcements, local-worker jobs — see Ops workspace.
partner_hub_*Partner Hub tasks, claims, taskers, creators, outreach, payouts, agent runs — see Partner Hub.
voice_previews, api_subscriptionsAdmin: S2S voice preview clips; third-party subscription register.
chats_empty_archiveEmpty per-visit chats moved out of chats by ArchiveEmptyChatsJob (TTL).
blockedBlock-list for phone/email.

PostgreSQL + pgvector — ms-ai​

ms-ai uses Postgres 17 with the pgvector extension via TypeORM; schema is migration-managed (ms-ai/migrations/).

Knox knowledge base (RAG)​

TableWhat
knox_knowledge_chunksEmbeddings + metadata (this docs corpus's chunks; dual 1536-dim, HNSW index).
knox_knowledge_documentsSource markdown (pre-chunk).
knox_knowledge_collectionsFolders/groups of docs.
knox_knowledge_sourcesExternal sources (URLs) ingested.
knox_page_registryInternal app URL map for navigation.
knox_search_indexCross-tenant semantic-search embeddings (visitor narratives, session intents) — results filtered by company_id.

Knox analytics & memory​

TableWhat
knox_session_analysesPer-visit analysis (PK: file_id).
knox_visitor_profilesPer-visitor cumulative profile (unique (company_id, user_session_id)).
knox_call_transcriptsVoice call transcripts + quality scores.
knox_agent_conversationsEvery Nox Q+A turn (audit + feedback).
knox_admin_memoryDurable per-admin facts (pgvector).

Knox tenant config + actions​

TableWhat
knox_ai_modelsLLM config per tenant.
knox_tenant_configCustom Nox behaviors.
knox_usage_eventsToken / cost usage tracking per tool call.
knox_engagement_rulesAutomation conditions.
knox_action_eventsUnified write-side action stream (GHL / Nox / ElevenLabs / calendar / email).
knox_action_auditAudit of Knox-initiated actions.
knox_ghl_automation_eventsRaw GHL webhook events.

Nox engine v2​

src/knox-v2/entities — purposes in Nox engine v2.

TableWhat
knox_v2_company_flags, knox_v2_runs, knox_v2_tasks, knox_v2_events, knox_v2_run_notesPer-company switch · runs · planned work · event ledger (SSE replay) · per-run worker notes
knox_v2_run_effects, knox_v2_undo_records, knox_v2_park, knox_v2_confirm_prefs, knox_v2_screen_consent, knox_v2_escalationsCostly/outward effects · undo versions · parked cards · confirm prefs · screen consent · hand-offs
nox_trigger, nox_schedule, nox_event, nox_email_queueProactive runs · per-admin wake time · judgeable events · delayed group emails
nox_contact_log, nox_guard_rules, nox_guard_logReach-the-admin decisions · admin limits · guard decisions
nox_rule (+ nox_rule_event), nox_case (+ nox_case_event), nox_goalRulebook · followed matters · goals
nox_standup_item, nox_thread_summary, nox_business_profile, nox_conversation_outcome, nox_product_notesPlan items · thread summaries · business profile · conversation outcomes · learned product notes
nox_jev_shadow, nox_quality_runsJev shadow judgements · daily quality checks

Identity & integrations​

TableWhat
agent_memoriesPersistent conversation facts per visitor.
elabs_agentsElevenLabs Conversational AI agent configs.
elabs_phone_numberMapped phone numbers for inbound AI calls (Twilio/Telnyx keys in meta encrypted at rest).
integrationsTenant platform credentials (Shopify / WooCommerce); secret keys in credentials encrypted with ms-ai's own keyring. Google tokens ms-ai no longer uses are dropped by the encrypt-at-rest job.
rag_productsSynced ecommerce products with embeddings.

Redis — cross-service (shared infra, not shared data)​

Isolated instances/keys per service (redis-backend, redis-communication, redis-sessions).

UseWhere
Socket.IO adapter (multi-pod fan-out)All sockets
Cache (subscription gate, CRM enrichment)Backend, ms-ai
Rate limitingms-communication call/message limiters
Session storageBackend auth
Visitor presence registryms-sessions (powers knox_presence)
BullMQ queuesms-sessions analysis worker, ms-ai fact-extraction + embedding workers

Firestore — ms-communication​

UseWhat
Real-time chat/call stateEach socket subscribes to /rooms/{room_id}/messages; SocketService keeps Firestore + Socket.IO in sync. Twilio WebRTC call tracking.

S3 — object storage​

UseWhere
Recording NDJSON chunks (raw rrweb)ms-sessions
Stitched recordings / video files (call recordings, default videos)backend
Uploaded files (logos, PDFs, agent knowledge)backend

Join-key cheat sheet​

The IDs that stitch records across stores:

WantJoin
Sessions for a personpeople._id → person_profiles.person_id (per tenant) → user_sessions.person_id
Chats for a personpeople._id → person_profiles.person_id → chats.person_id
Recording for a sessionuser_sessions.widget_user_session_id ← session_files.user_session_id (then session_files._id = file_id)
Analysis for a recordingsession_files._id = session_analyses.file_id = knox_session_analyses.file_id
Visitor profile from session(user_sessions.company_id, user_sessions.widget_user_session_id) = visitor_profiles.(company_id, user_session_id)
Cross-tenant Person checkpeople.match_keys contains sha256(normalized_email)
Booking attributionbookings.user_session_id → user_sessions → full journey