Skip to main content

Knowledge Base

Audience: Sales · Management · QA · Support · Engineering · Where in app: Settings → Knowledge Base (also reachable from Settings → AI Agents) · Plan availability: All plans; document/token/page caps are plan-dependent (verify)

The Knowledge Base is the content your company's AI agents read to answer visitor questions. You feed it your website pages, PDFs, FAQs, and free text; the platform turns that into searchable embeddings; and at conversation time the Chat Agent and Call Agent retrieve the most relevant pieces to ground their answers. The more comprehensive and accurate it is, the better your AI performs.

Not the same as Nox's knowledge base. This page is the company KB (what visitors' AI reads). Nox (the in-app admin assistant) has its own, separate product-docs KB. See the comparison table at the end.

What it does​

  • Stores company-specific content (products, services, pricing, policies, FAQs) as embeddings.
  • Powers RAG grounding for the Chat Agent, Call Agent, and Auto-Connect Agent — any content you add benefits all of them.
  • Syncs an e-commerce product catalog (Shopify/WooCommerce) for product lookups during conversations.
  • Auto-seeds itself from the company website on signup so the agent can answer from day one.

How it works​

Source types. Content comes from five source types:

SourceHow it's ingested
Website / URLPaste a URL → scraping / scrapping-bee fetches the page → markdownified → chunked. (Walks the sitemap, capped.)
PDFUpload → pdf-extraction parses → chunked. Needs selectable text (not scanned images). Max 10 MB per file (verify).
TextType/paste custom content directly.
FAQ / Q&AManually entered question+answer pairs (AIConversationModel).
Auto-scrape (signup)On signup the system scrapes the company homepage + sitemap as a starter set.

Ingestion pipeline.

  1. Source ingested into ms-ai (via the relevant queue).
  2. Text extracted (PDF/HTML).
  3. Split into chunks (sliding window with overlap).
  4. Each chunk embedded with OpenAI text-embedding-3-small (1536-dim).
  5. Stored in the ms-ai embeddings pgvector table, keyed by company_id.
  6. Retrievable at chat/call time via semantic search.

Backend tracks uploaded documents on companies.agent_knowledge_base_documents[]:

{
file: ObjectId (FileModel),
status: pending | processing | ready | failed,
job_id: string,
page_count: number,
error: string,
frontmatter: { ... }
}

Retrieval (how agents use it). Per visitor query the agent:

  1. Embeds the question.
  2. Runs an approximate-nearest-neighbour search over the company's chunks — top ~20 candidates by cosine distance (embeddings <=> $vector).
  3. Cohere-reranks down to the top KB_TOP_RESULTS (env, default 4; falls back to vector order, cut to the same count, if Cohere is unavailable). Product search uses PRODUCT_TOP_RESULTS (default 4); website crawls are capped by CRAWL_MAX_LINKS (default 5000).
  4. Injects those chunks as a "Knowledge" section in the system prompt.
  5. The LLM answers grounded in those chunks.

If retrieval returns nothing relevant, the agent answers from its persona prompt only — less grounded, more likely to hallucinate. Keep the KB tight and current.

FAQs (manual Q&A). AIConversationModel (company_id, optional chat_id, question, answer, question_id, answer_id). Short, high-confidence answers. The agent prefers a strongly-matching FAQ over scraped content — ideal for nailing brand voice on common questions.

Product catalog (RAG). For e-commerce, Shopify and WooCommerce sync the product catalog into rag_products (ms-ai). Each product is embedded for semantic lookup, so the agent can search products by name/description, recommend related items, quote pricing, and link to product pages. Sync state lives on the integration row: sync_status, last_sync_at, product_count.

Configuration & options​

  • Add sources in Settings → Knowledge Base → Add Source (Website / PDF / Text / Q&A). Editable in-dashboard.
  • Refresh / re-train:
    • PDFs — re-upload to replace.
    • URLs — "Refresh" re-scrapes and re-embeds if content changed.
    • FAQs — edit in dashboard.
    • Bulk re-embed via admin endpoint when the embedding model/strategy changes.
  • Preview tool — test agent responses against the KB before going live.

Performance tips for admins (QA/support):

  • Use focused FAQs for high-frequency questions.
  • Trim scraped content — strip nav/footer boilerplate.
  • Add product PDFs (spec sheets, integration guides).
  • Sync your FAQ/help-center page first — it already contains the most-asked Q&A.

Behaviors & edge cases​

  • Status lifecycle. Each document is pending → processing → ready (or failed with an error). A failed doc isn't retrieved; re-upload or re-scrape to retry.
  • Ingestion vs. credits. Embedding new content consumes a small amount of AI Credits per chunk. If credits are exhausted, ingestion stays pending until refilled. See AI Credits.
  • Scrape gating on the combined balance. backend/app/lib/aiCredits.js is the single balance rule — plan allowance plus purchased credits — shared by the manual "Add Website" scrape (CompanyWebsiteController.scrape) and CompanyController's agent-switch check, so both give the same verdict. Zero combined balance rejects the scrape with a "purchase more credits" message.
  • Scanned PDFs / OCR. Mistral OCR (ms-ai/src/pdf-extraction/providers/mistral-ocr.service.ts) handles image content; its annotation prompt now transcribes all visible text — only purely decorative elements/logos are dropped.
  • Stale content. URL sources don't auto-refresh — re-scrape when your site changes.
  • Empty retrieval → hallucination risk. Sparse or off-topic KBs leave the agent ungrounded; this is the main lever on answer quality.
  • Scanned PDFs. Image-only PDFs yield no text and won't train the agent.

Plan & limits​

  • Available on all plans.
  • Number of documents, total token count, and page-count caps are plan-dependent — verify under billing/plans.
  • PDF max size 10 MB per file (verify current limit).
  • Embedding/ingestion draws on AI Credits (small per-chunk cost). See AI Credits.

What Nox can tell you​

  • How many docs are in the KB, what sources exist, which are pending/failed.
  • How often the agent has cited each chunk (via session analyses + tool-call logs).

Nox can't add docs — that's a manual admin action in Settings → Knowledge Base.

Technical implementation​

  • Service: ms-ai. Ingestion modules: scraping/ + scrapping-bee/, pdf-extraction/, embedding/, company/ (RAG queries + Cohere rerank), cohere/.
  • Endpoints: POST /context/.../:company_id (RAG context for chatbot), POST /products/.../:company_id (product search), POST /create-embeddings, POST /enqueue-embeddings, POST /fetch-urls, POST /enqueue-scrapping, POST /filter-links, POST /pdf-extraction/extract, GET /pdf-extraction/content/:company_id/:file_id. See ms-ai/API_ENDPOINTS.md.
  • Queues (BullMQ): scrappingQueue, embeddingQueue, pdfExtractionQueue, shopiProductsQueue / wooProductsQueue.
  • Retrieval code: company/company.service.ts — questionEmbeddings() then getUserContext() (top-20 by cosine, CohereService.rerankItems(..., KB_TOP_RESULTS)).
  • Embedding model: EMBEDDINGS_MODEL env (default text-embedding-3-small) in constants.ts.
  • Storage: embeddings pgvector table (company KB chunks), rag_products (catalog). The separate Nox product KB uses knox_knowledge_chunks — see Knowledge-Base RAG.

Agency default knowledge base (2026-10)​

Backend 4245f7061, frontend 15dae45e. An agency keeps PDFs and text entries every new client sub-account starts with (agencies.default_knowledge_base; max 10 PDFs × 10 MB, 20 texts × 50k chars). UI: agency Settings → Knowledge Base (/agency/settings/knowledge-base, "Default Knowledge Base", needs agency.manage_settings). Routes (agencyKnowledgeBaseRoutes.js, AgencyKnowledgeBaseController): GET /agencies/default-knowledge-base, POST …/documents, DELETE …/documents/:id, PUT …/texts.

lib/agencyDefaultKnowledgeBase.js runs on sub-account create (Add Client, GHL create, signup page, Platform API — not the agency's own account), best-effort:

  • each PDF is copied (own FileModel + storage object in the private agent-knowledge-base folder) and sent to ms-ai extraction like a normal upload, flagged agent_knowledge_base_documents[].auto_train; when every flagged document has finished extracting, the extraction webhook trains them server-side (trainDocuments, the replica of the frontend's trainAIOnDocuments — nobody has the client's page open);
  • text entries are appended to additional_contexts and trained immediately.

Existing clients are never changed; a client can edit or remove its copy. In client setup, a sub-account with starter knowledge (an auto_train doc, a non-failed upload or a non-empty text) skips the knowledge step when it has no website ("I'll add my website later").

Business-profile prefill from PDFs (ms-ai 92381ee5f): POST /business-profile accepts pdf_file_ids (≤ 50); ScraperService.getCompanyPdfContent reads the named pdf_contents rows (only ids the caller names — rows outlive a removed document) and appends them after the website text (the extractor keeps the first 14k chars, so the website leads). Sent by the client-setup BusinessStep.vue.

Private KB files (2026-10)​

Backend 0842e3077: uploads under agent-knowledge-base/ (and certification/) are stored without public-read (lib/privateFiles.isPrivatePath) and GET /files/:id/get serves them only to the uploader, a manager of the owning company, ms-ai (webhook key) or an allowlisted admin, with Cache-Control: private, no-store; non-media files download (nosniff + attachment). Existing objects were flipped by the admin maintenance job POST /admin/maintenance/private-files-acl (dry-run first).

Adjacent KB/cost fixes (2026-08)​

  • Nox-KB collection delete now cascades (ms-ai/src/knox-agent/services/knox-document.service.ts): deleting a collection soft-deletes its documents and chunks in one transaction; retrieval filters chunk.is_active, so deleted content can no longer produce stale answers. Recreating a collection with the same slug revives it. (Nox KB — knox_knowledge_* tables — not the company KB.)
  • Cost-tracking leader lock (ms-ai/src/cost-tracking/schedulers/leader-lock.util.ts): Postgres advisory locks (keys 4823001/4823002) gate the 03:00 refresh-estimates and 04:15 elevenlabs-catchup schedulers to one replica (2026-08-06 Postgres CPU incident — every replica ran them). Retry/prune/reconcile schedulers deliberately remain all-replica.

Difference from Nox's own knowledge base​

Company KB (this page)Nox KB
AudienceVisitors talking to the company's AIAdmin asking Nox
ContentCompany-specific (their products, FAQs)Knock Knock product docs
Storageembeddings per companyknox_knowledge_chunks (global)
Retrieved byChat/Call agent at conversation timeNox knox-knowledge-search tool