Skip to main content

Knowledge-Base RAG

Nox's product knowledge comes from the nox-knowledge-base repo — markdown files ingested by ms-ai's KnoxDocumentService into a pgvector store and retrieved with reranking.

Source: nox-knowledge-base​

A markdown repo of ~60+ files across 15 category dirs (platform, widget, communication, ai-agents, sessions, identity-graph, leads-and-scoring, bookings, integrations, billing, settings, agency, desktop, data-model, nox-self), indexed by INDEX.md.

Each file carries frontmatter that drives retrieval:

title: <human-readable>
content_type: feature_guide | how_to | troubleshooting | policy | api_reference |
faq | concept | screen_guide | setting | workflow | ...
product_area: dashboard | chat | voice | visitors | settings | integrations |
widget | analytics | billing | knox | identity-graph | ...
plan_scope: all | starter | growth | pro | enterprise
tags: [keywords]
related_urls: [/settings/integrations/*, ...] # enables URL-filtered retrieval

Ingestion pipeline (KnoxDocumentService)​

markdown file
→ save as KnoxKnowledgeDocument
→ semantic chunking (split on H2/H3 headings)
→ per chunk: contextual prefix (section hierarchy) + AI summary
→ dual embeddings (OpenAI text-embedding-3-small, 1536-dim):
embedding = contextual_prefix + content_text
summary_embedding = intent-level matching
→ store in knox_knowledge_chunks (Postgres + pgvector, HNSW index)
→ organized under KnoxKnowledgeCollection

Re-ingestion (POST /knowledge/ingest, knoxDocumentQueue) soft-replaces prior chunks for the same source (is_active=false) rather than deleting.

Retrieval (3-prong + rerank)​

For a query, retrieval combines:

  1. URL-filtered search — chunks whose related_urls match the admin's current page rank higher.
  2. Global vector search over embedding (HNSW).
  3. Summary embedding search over summary_embedding.

The prongs are merged with reciprocal rank fusion (1/(60+rank) summed per chunk, KnoxKnowledgeService merge step) — without Cohere this is the final ranking. Prong priority order used to let any chunk tagged with the admin's page outrank a better global match, and raw distances aren't comparable across the two embeddings. Candidates are then reranked with Cohere rerank and the top-K (topK: 6) injected into the Nox prompt. If COHERE_API_KEY is unset or the call fails, CohereService returns the caller's order cut to topN (it used to return every candidate).

knox_knowledge_search takes only query — the filter_feature_area / filter_content_type parameters were removed (2026-10, ms-ai 37644d6a0): the model set them on unscoped questions and the AI-derived labels hid the right chunks. product_area / content_type frontmatter is still stored, just not used as a model-set filter.

Authoring workflow​

  1. Write markdown with the required frontmatter.
  2. Use H2/H3 headings (the chunk split boundary); lead each section with the canonical feature/object name; prefer fact lists and tables over prose (~200–800 words/file).
  3. Cross-reference related files; add an entry to INDEX.md.
  4. POST /knowledge/ingest to re-ingest — the soft-replace handles stale chunks.

Or sync the whole repo from the super-admin Knowledge screen (/admin/knox, see Admin → Knox): Bulk Import opens Sync Knowledge Base (KnoxBulkImportModal.vue). Drop the nox-knowledge-base folder (subfolders = collections, files = documents) and it diffs against what's stored before writing anything:

  • Documents match on collection slug + title (utils/knoxKbSync.js matchCollection). Same markdown + same related_urls = Unchanged (skipped); otherwise Updated (saved with the existing id) or New. A stored doc in failed status is always re-sent.
  • Stored documents with no matching file (deleted files, renamed titles, duplicates from older imports) and whole collections missing from the folder are listed as Removed; the "Remove N item(s) that aren't in this folder" checkbox (on by default) deletes them after the uploads.
  • An unreadable file aborts the scan — syncing without it would delete its stored doc. The button reads Apply N changes / Already up to date.
  • Gotcha: the stored-docs list is one getKnoxDocuments call per collection, capped server-side at 100. A collection past 100 docs gets unseen docs re-created as duplicates — page the call before that happens.

New product routes Nox can deep-link to must also be added to the page registry (ms-ai/src/knox-agent/page-registry/route-map.ts) — the 2026-10 KB sync added 25.

  • The same pgvector approach backs Knox admin memory (operator-specific context) — see Nox Assistant.
  • Company website/FAQ/PDF content is separately embedded for the AI chat agent (scraping/embedding/PDF queues in ms-ai), distinct from the Nox product KB. Its search returns KB_TOP_RESULTS passages (env, default 4) and product search PRODUCT_TOP_RESULTS (default 4), with the same original-order fallback when Cohere is missing; website crawls are capped by CRAWL_MAX_LINKS (default 5000 — a hardcoded 500 in scrapping-queue.processor.ts used to override it).