Skip to main content

Data Stores

Each service owns its own persistence. There is no cross-service shared schema — data crosses boundaries via API calls and IDs (company_id, user_session_id, person_id), not joins.

MongoDB (backend, ms-communication, ms-sessions)​

The three Nodevel services use MongoDB via Mongoose. They run against mongo:7.0 locally (MongoDB 8.0 segfaults on some kernels).

backend — the largest schema (~70 models)​

GroupKey models
Auth / RBACUserModel, TeamModel, RoleModel, PermissionModel
Tenancy / billingCompanyModel, AgencyModel, SubscriptionModel, AgencySubscriptionModel, PackageModel, AddonModel, InvoiceModel
ConversationsUserSessionModel, ChatModel, CallModel (inbound/outbound)
Identity graphPersonModel (unique match_keys_hash_unique), PersonProfileModel, PersonIpSignalModel (keyed by HMAC ip_hash), SessionIpsModel (ipdetective class + device churn)
AI / KnoxAIConversationModel, AICreditLogModel, KnoxVisitorDirectiveModel, KnoxVisitorWatchEventModel, NoxActionModel
CRM / enrichmentOutboundCallGhlSyncModel, HubspotRequestModel, ApolloRecordModel, ClearbitSessionModel, WebhookModel
PlatformChatBotModel, CompanyFaqModel, FileModel, SessionFileModel, LogModel, NotificationModel
Audit / reviewAuditLogModel (audit_logs, 180-day TTL, written by the lib/auditLog plugin on 39 models), FormAbandonmentReviewModel, ApiRequestModel (credentials redacted)
Secrets / securityCredentialModel (credentials — every tenant OAuth token/API key as one SecretCrypto blob per owner + provider, only via CredentialVault), CredentialKeyModel (credential_keys — per-version key fingerprints), AdminAllowedIpModel (admin_allowed_ips). See Security & Secrets
SettingsSettingModel (settings — key/value: admin settings, Feature Flags platform switches read with a one-minute cache, job settings such as recording_retention, archived_company_deletion, partner_hub)
Ops / supportops_cards, ops_stages, ops_scrum_notes, ops_tickets, ops_ticket_messages, ops_agent_actions, ops_announcements, ops_announcement_seen, ops_roadmap_votes / _guests / _comments, ops_feature_requests, ops_video_jobs, ops_video_workers, ops_state, ops_counters; product updates product_update_sends / _suppressions / _videos / _agency_decisions / _rollouts. See Ops workspace
Partner Hubpartner_hub_tasks, _claims, _taskers, _creators, _creator_messages, _outreach, _topics, _subreddits, _payouts, _agent_runs, _api_usage. See Partner Hub
Admin miscadmin_notifications + popup_receipts (alert popups), voice_previews, api_subscriptions, chats_empty_archive (empty chats moved by ArchiveEmptyChatsJob, TTL)

Index notes (2026-09): person_id is indexed on every collection the Person merge repoints (and the session claimed/verified/device_shadow person ids) — without them each repoint updateMany was a full scan past the 45 s socket timeout. service_rooms carries the partial fact_extraction_sweep_v1; chats moved (2026-10-14) to fact_extraction_sweep_v2 and transcript_email_sweep_v4, which hold only chats with a message (every visit creates an empty chat); ai_conversations, chat_queues, users (online_status) and companies.go_high_level.location_id gained indexes they had lacked. user_sessions deliberately has no fp_hash index (storage) — sweeps that need it slice by time instead. The strict backend SessionFileModel declares the sweeper fields (recording_status, skipped_reason, finalized_at, handoff_attempt_count, …) without indexes; ms-sessions owns that index list.

ms-sessions​

user_sessions (lifetime + nested lead_score), session_files (per-page rrweb recording + video metadata + recording_status), plus mirrored companies / users.

ms-communication​

chats (with embedded messages[]), inbound_outbound_calls (Twilio SID, recording URL, transcript), user_devices (APN/FCM tokens), plus companies / agencies. service_rooms.call_logs (per-call step log, last 1,000 entries, select:false) is written here by CallLogHelper.

Postgres + pgvector (ms-ai)​

ms-ai uses Postgres 17 with the pgvector extension via TypeORM. Schema is migration-managed (ms-ai/migrations/, npm run migration:run).

GroupKey entities
Knowledge / RAGknox_knowledge_chunks (dual 1536-dim embeddings, HNSW index), knox_knowledge_documents, knox_knowledge_collections
Knox conversationsknox_agent_conversations, knox_admin_memory (pgvector)
Visitor insightknox_visitor_profile, knox_session_analysis
Action auditknox_action_events, knox_action_audit, knox_ghl_automation_events
Integrations / costCompany, Integration (integrations.credentials secret keys encrypted at rest with ms-ai's own keyring), elabs_phone_number (carrier keys in meta encrypted), AgentMemory, PdfContent, cost-tracking tables

Schema features: pgvector columns with HNSW indexes for sub-100ms similarity search, JSONB for structured payloads, is_active soft-delete flags for re-ingestion.

Redis (shared infra, not shared data)​

Used by every realtime service, but with isolated instances/keys per service (redis-backend, redis-communication, redis-sessions in the cluster):

  • Socket.IO adapters (pub/sub or streams) for multi-pod fan-out.
  • BullMQ queues (ms-ai and ms-sessions).
  • Visitor presence registry (ms-sessions).
  • Rate limiting and short-lived caches.
  • Backend cron firing claims (cron:claim:<job>:<minute>, 120 s) so one schedular replica runs each firing.

Firestore (ms-communication)​

Real-time chat message store — each socket subscribes to /rooms/{room_id}/messages; SocketService keeps Firestore and Socket.IO state in sync.

Object storage (S3)​

Session recordings (stitched MP4 + raw rrweb chunks), uploaded files, and call recordings live in AWS S3, fronted by CloudFront / Azure CDN. Managed by backend's S3Service and ms-sessions' upload pipeline.