Skip to main content

AI Cost-Tracking (Admin)

Audience: Finance, super-admins, engineering · Where in app: /admin/cost-tracking/... · Access: Super-admin only — every endpoint is gated by AdminOnlyGuard in ms-ai.

Cost-Tracking is the super-admin view of what the AI providers cost us — OpenAI, Anthropic, ElevenLabs (voice), Google, Baseten, ScrapingBee, plus flat-USD providers (Agent K's Apollo/PredictLeads/Unipile and telephony) — across every company, agent, and model. It is the cost side of the ledger (provider spend), distinct from revenue.md (customer subscription income). It lives in the AI microservice (ms-ai), not the main backend, and is backed by a PostgreSQL schema (usage_event / usage_charge / pricing tables).

What it does​

  • Aggregates fleet-wide USD and credit spend over a time window, broken down by provider and by agent.
  • Computes empirical unit economics ($ per chat message, per auto-trigger, per voice minute).
  • Ranks companies by spend (leaderboard) and drills into one company's per-agent / per-model breakdown.
  • Holds the rate-card catalog: providers, models, per-unit prices, and plan rates.
  • Tracks ElevenLabs (credit-based) and ScrapingBee subscription state and reconciles our recorded cost against what the provider reports (variance / drift).
  • Flags charges that ran without a resolvable price ("missing-price") so ops can fix the catalog.

How it works​

Data model (entities)​

All under ms-ai/src/cost-tracking/entities/:

  • usage_event — one row per provider API call or webhook ("what happened"). Carries company_id, provider_id/provider_name_raw, model_id/model_name_raw, agent (free-text use-case label), external_id (idempotency: UNIQUE(provider_id, external_id)), occurred_at (event time), billable_duration_sec, is_test, and a request_meta jsonb blob. No money lives here.
  • usage_charge — the atomic billable component ("how much"). Carries amount_usd, credits, quantity, unit (a BillingUnit), charge_source (a ChargeSource), price_missing, links to model/price/plan_rate/elevenlabs_plan_rate, and voided_at / reversal_of. Charges are immutable — corrections insert a negative-quantity reversal row pointing back via reversal_of and set the original's voided_at.
  • ai_provider — provider catalog (ProviderKind: openai, anthropic, elevenlabs, google, baseten, scrapingbee, …).
  • ai_model — model catalog (ModelKind: llm, embedding, voice, tts, stt, unknown, scrape) with is_active / is_visible.
  • ai_model_price — per-unit USD price, unique on (model_id, unit, service_tier). unit is a BillingUnit (input_token, output_token, cached/cache_read/cache_write tokens, minute, second, character, message, request, credit, …).
  • ai_plan_rate — provider plan-level rates (e.g. per-message).
  • elevenlabs_plan_rate — one row per ElevenLabs tier (starter/creator/pro/scale) with plan_price_usd, included_credit_rate, overage_credit_rate, burst_multiplier. Numeric columns come back from TypeORM as strings — callers must coerce.
  • elevenlabs_subscription_state / scrapingbee_subscription_state — singleton (id = 'singleton') live-account snapshots, written by webhooks + cron, read-only in admin.
  • provider_cost_snapshot — daily reconciliation rows (provider-reported vs our-computed). provider_invoice — imported provider invoices for variance checks.

Correct aggregation pattern (from the usage_charge entity doc): always filter voided_at IS NULL AND reversal_of IS NULL; dropping either double-counts refunds.

How credits map to USD​

Credits are the customer-facing unit. The conversion is fixed: CREDITS_PER_USD (default 100) — i.e. 1 credit = $0.01 (ms-ai/src/constants.ts). The unit-economics cards multiply USD by 100 to show credits. This is an env var; it is no longer surfaced as an editable admin field.

Time windows​

Windowed endpoints accept either { hours } (anchored to now, default 24×30 = 30 days) or { from, to } (ISO 8601, fixed [from, to) range). The repo helper appendWindow (frontend/src/repositories/admin/cost-tracking.js) chooses which to send. Custom ranges are capped server-side at 90 days for the stats endpoints; /missing-prices accepts only hours (1–720). Windows filter on occurred_at (event time), not write time, and always exclude is_test = true.

Reconciliation & drift​

A daily cron (≈ 05:30 UTC) fetches each provider's reported usage for the prior UTC day and writes a provider_cost_snapshot with provider_reported_value, our_computed_value, variance_pct, and a status (reconciled / variance / disputed / missing). variance_pct = (provider − ours) / provider × 100, signed:

  • > 0 → we recorded less than the provider says → loss exposure (e.g. a missed ElevenLabs post-call webhook, or under-billing vs the OpenAI invoice).
  • < 0 → we recorded more than the provider says → over-recorded (duplicate/phantom event, or over-billing).
  • ≈ 0 → reconciled.

The fleet headline drift_pct is the our_computed_value-weighted absolute variance over the user-selected time window (no longer hardcoded to 7 days).

Screens & fields​

The five tabs share CostTrackingAdminTabs.vue. The knox agent is displayed as "Nox" in these screens (display override only).

Spend Overview — /admin/cost-tracking​

frontend/src/views/admin/cost-tracking/Overview.vue. Filters: time window (24h / 7d / 30d / 90d / custom), provider chips, agent chips (chips are populated from GET /stats/dimensions).

KPISource
Total spend (USD + credits)GET /stats → total_usd, total_credits
AI requestsevent_count, company_count
Avg per requestavg_per_event_usd
Active dimensionsby_provider.length / by_agent.length
OpenAI / ElevenLabs reconciliationlatest non-missing provider_cost_snapshot per provider; sign + status drive the tone (loss/profit/reconciled)
Missing-price chargesmissing_price_count

Plus: Spend by provider and Spend by agent breakdown panels (top 10, with %-of-total bars and a natural per-agent volume unit — chat=messages, voice=calls, auto_trigger=triggers, knox=turns); Unit economics cards ($ per chat message / auto-trigger / voice minute, plus dedicated inbound / outbound call-minute cards, from GET /stats/fleet-estimates); a Latest reconciliation table (last 7d snapshots); and a Missing-price charges table rolled up by (charge_source, provider, model, agent, unit, service_tier) with a reason of model_unresolved / unit_unpriced / tier_unpriced / plan_tier_unpriced.

Company Spend — /admin/cost-tracking/companies​

Companies.vue. Default view is a leaderboard of top-spending companies in the window (GET /companies/leaderboard) — columns: rank, company (name + company_id), USD, credits, AI events, agents, last activity. ms-ai does not store company names, so the FE resolves them via the main backend (POST /company/admin/names-by-ids). Searching by name resolves names → IDs on the backend, then restricts the leaderboard via company_ids. Clicking a row drills into per-agent / per-model / per-unit breakdown (GET /usage?company_id=...).

Models & Prices — /admin/cost-tracking/models​

Models.vue. Read-only in this view (CRUD ships separately). Three tables:

  • Providers — name, kind, active, visible.
  • Models & per-unit prices — provider, model (+ display_label), kind, active, visible, Used by (distinct agents observed in usage_event for that model), and Prices (per-unit USD chips; "No prices" warning in amber). Filter chips: All / LLMs / Embeddings / Voice. Per-token prices are tiny so they render with up to 8 decimals; the API also exposes per_million (USD per 1M tokens) for LLMs.
  • Provider plan rates — provider, plan, unit, USD/unit.

ElevenLabs Plans — /admin/cost-tracking/elevenlabs-plans​

ElevenLabsPlans.vue. Live subscription state (GET /elevenlabs-subscription-state): tier, status, overage flag, credit_limit + max_credit_extension → total_allowance, credits_remaining (vs ElevenLabs' api_credits_used), drift = api_credits_used − local_credits_used, last sync source, next reset/invoice. Auto-refreshes every 60s. Plan-rate editor (GET /elevenlabs-plan-rates, PATCH /elevenlabs-plan-rates/:tier): per-tier plan_price_usd, included_credit_rate, overage_credit_rate, burst_multiplier. Only fields present in the PATCH body are updated; the seeder owns row creation (PATCH 404s if the tier row does not exist); updated_by/updated_at are stamped automatically.

ScrapingBee — /admin/cost-tracking/scrapingbee​

ScrapingBee.vue. Live state (GET /scrapingbee-subscription-state): inferred plan, max_api_credit allotment, used_api_credit / credits_remaining, utilization_pct, per_credit_usd (from the scrapingbee-request model's credit price), derived plan_price_usd (rate × allotment), effective_per_credit_usd (plan price ÷ used), and ledger drift_credits / drift_pct (provider used − our recorded). "Sync now" (POST /scrapingbee/sync) forces an immediate /usage pull + drift recompute; the cron also runs every 30 min.

Endpoints​

All under ms-ai, base GET|POST|PATCH /api/cost-tracking/admin/... (guarded by AdminOnlyGuard):

EndpointPurpose
GET /providersList providers
GET /models (?provider_id,?kind,?visible)Model catalog + prices + used_by_agents
POST /modelsUpsert a model
GET /prices (?model_id) / POST /prices / DELETE /prices/:idPrice CRUD (upsert is in-place; existing charge snapshots are not mutated)
GET /plan-rates / POST /plan-ratesProvider plan rates
GET /elevenlabs-plan-rates / PATCH /elevenlabs-plan-rates/:tierElevenLabs tier rate card
GET /elevenlabs-subscription-stateLive ElevenLabs account state
GET /scrapingbee-subscription-state / POST /scrapingbee/syncScrapingBee state + forced sync
GET /stats (?provider[],?agent[],window)Fleet totals + by-provider + by-agent + drift
GET /stats/dimensionsDistinct agents/providers in window (filter chips)
GET /stats/fleet-estimatesEmpirical $ per chat msg / knox / auto-trigger / voice min
GET /companies/leaderboard (?search,?company_ids,?limit)Top-spending companies
GET /usage (?company_id,?agent,?model_id)Per-company breakdown
GET /voice-split (?days)Voice cost: platform (duration) vs llm_passthrough (inner-LLM)
GET /missing-prices (?hours)Unpriced charges, rolled up with a reason
GET /variance-scan (?hours,?threshold_pct)Events with high variance vs provider total
POST /invoices, GET /invoices, POST /invoices/:id/recomputeProvider-invoice reconciliation
POST /reconcile/run (?date), GET /reconcile/snapshots (?provider_name,?days)Daily reconciliation (cron 05:30 UTC + manual)
POST /elevenlabs/catchupRe-ingest missing ElevenLabs conversations
GET /estimates, POST /estimates/refresh (under /api/cost-tracking)Per-company per-agent cost estimates

Flat-USD provider usage (Agent K + telephony)​

Non-token costs go through recordFlatUsdCharge (cost-tracker.service.ts) — flat USD converted to credits at the global rate ($1 = 100 credits), landing in the same usage_event/usage_charge tables:

  • Agent K providers. Backend meters Apollo/PredictLeads/Unipile per company/provider/day (provider_usage); the hourly AgentKUsageEmitJob forwards CLOSED-day rows to POST /api/cost-tracking/internal/provider-usage (internal secret-token auth, not AdminOnlyGuard). Idempotent on external_id agentk:<co>:<provider>:<day>, with an emitted watermark for retries. Pricing: src/cost-tracking/agent-k-provider-pricing.ts — USD/call defaults apollo_match $0.05, predictleads $0.02, unipile $0.01, apollo_search free; override via AGENT_K_PROVIDER_USD env (JSON, no deploy). Rows with company_id: 'platform' (shared Apify SoS pull) are never emitted.
  • Telephony. ms-ai's number-rental / Twilio call-cost crons record charges the same way, under separate providers so reconciliation stays clean — see Wallet.
  • New dimensions. agent_k is a usage agent, and knox_intent is split out from knox (intent-classifier spend visible separately). Baseten GLM-4.7 is priced at 0 credits — platform-absorbed, still tracked in USD.

These credits flow into the existing token-usage sum, so the backend's AITokensCreditJob deducts them like any other AI usage. Reconciliation + statements come from the admin cost-tracking endpoints below.

Behaviors & edge cases​

  • Voided/reversed charges must be filtered out of any manual SQL aggregation (voided_at IS NULL AND reversal_of IS NULL), or refunds double-count.
  • llm_passthrough is not the platform model. When an inner LLM runs during an ElevenLabs voice call, its charge is tagged llm_passthrough and the parent event's model_name_raw is the voice model. Missing-price rows for passthrough show <unresolved inner LLM> — fix the inner-LLM resolver, do not add token prices to the voice model.
  • Missing-price reasons are actionable: model_unresolved (raw name not in catalog → add the model or fix name casing), unit_unpriced (model exists, no price for that unit), tier_unpriced (price exists but not for that service_tier), plan_tier_unpriced (ElevenLabs webhook tier not in elevenlabs_plan_rates → add the tier row).
  • primary_event_count is used so "5 messages" means 5 user asks, not 20 (it excludes embedding sub-calls and tool follow-ups). Voice merges voice / voice_inbound / voice_outbound.
  • ms-ai has no company names — always resolve via the main backend; leaderboard rows alone carry only company_id.
  • Decimal-as-string: numeric money/rate fields from TypeORM often arrive as strings (to avoid float-precision loss). Coerce before arithmetic.
  • Singletons require the seeder. The ElevenLabs/ScrapingBee state endpoints 404 if the seeder has not created the singleton row.

Technical implementation​

  • Controller: ms-ai/src/cost-tracking/controllers/admin-cost-tracking.controller.ts (and estimate.controller.ts, ai-model-pricing.controller.ts)
  • Guard: ms-ai/src/auth/admin-only.guard.ts
  • Enums: ms-ai/src/cost-tracking/cost-tracking.enums.ts
  • Entities: ms-ai/src/cost-tracking/entities/*
  • Services: cost-reconciliation.service.ts, reconciliation.service.ts, usage-estimate.service.ts, scrapingbee-subscription-sync.service.ts, elevenlabs-*
  • Schedulers (cron): ms-ai/src/cost-tracking/schedulers/*
  • Credit conversion: ms-ai/src/constants.ts (CREDITS_PER_USD)
  • Frontend views: frontend/src/views/admin/cost-tracking/* · Repo: frontend/src/repositories/admin/cost-tracking.js
  • Related service: ../services/ms-ai.md