docs/usage/usage-reports.md
This document defines what an Onyx usage report must tell an organization admin, and why. It starts from the admin's job, not from the data we happen to store.
The rule that governs every decision here: a number belongs in the report only if the admin can finish the sentence "so I will...". If no action follows, cut the number.
create_new_usage_report builds a zip with three CSVs and a PDF review pack:
| File | Contents |
|---|---|
chat_messages.csv | One row per message: session, user, flow, time, agent, email, tokens, model |
users.csv | user_id, is_active |
usage_by_user.csv | Per user, per day, per model/flow/provider: tokens, cache reads, cost |
usage_report.pdf | Summary of spend, adoption, seats, and usage attribution |
The raw CSV files are still a data dump. They have three problems:
users.csv has no email, so it cannot join to the other files.The raw export is still valuable. It is just the wrong artifact for every job.
An org admin is accountable for three things:
Every useful metric comes from one of these. The sections below derive the metrics from the decisions.
The admin needs a seat ledger, not a message count.
The named lists are the deliverable. The admin acts on them directly. They reclaim a seat, retrain the person, or drop the seat at renewal.
Total cost drives no action. Cost concentration does.
cache_read_tokens today.Cost split by team or user group. Most orgs must charge the cost back, or at least explain the invoice internally. An admin who cannot attribute cost to a cost center cannot grow the deployment. This blocks expansion.
User__UserGroup already gives us the join.
Message volume is a trap. It rises when three people go heavy.
Measure breadth first, then habit, then depth:
The drop-off point in the funnel tells the admin what to fix. A drop before the first message means onboarding. A drop after it means answer quality.
The admin cannot read conversations. Give them proxies, and give them trends. Nobody knows what a good absolute thumbs-down rate is.
This section is the most actionable, and it does not exist today.
Each item maps to one concrete action: add a source, write a document, delete an agent.
Admins do not want a security console here. They want a short "look at this" list, with names on it.
Every vendor can report spend and logins. Only Onyx knows what the organization tries to learn and fails to find.
A monthly artifact that lists the top unanswered questions, clustered by topic, with the missing sources named, is worth more than the full cost breakdown. It tells the admin something about their own company that they cannot get anywhere else. It also makes the strongest renewal argument that exists.
Treat this as the headline of the report, not an extra tab.
The current report tries to be one thing. The admin needs three, at three cadences. This is the main structural change.
| Artifact | Cadence | Form | Purpose |
|---|---|---|---|
| Pulse | Monthly | Pushed to email or Slack, no download | Tell the admin if anything changed |
| Review pack | Quarterly | A PDF the admin forwards to their boss | Defend the spend and the rollout |
| Investigation export | On demand | The raw CSVs we build today | Answer a specific question, feed BI |
The pulse must be pushed. An artifact that needs a download and a spreadsheet gets read once.
If the pulse were a single screen, it holds eight items:
Everything else lives one click deeper.
The review pack must be a PDF. An admin forwards it to a VP or a CFO. Those people do not open a zip of CSVs, and they do not log in to an admin panel. The PDF is the artifact that travels.
Use ReportLab (BSD licensed). Write the document in Python. Do not template markdown, do not render HTML, and do not drive a browser.
ReportLab supplies everything the pack needs:
platypus flows content into pages. It splits long tables across pages and
repeats the header row.graphics.charts draws bar, line, and pie charts as native PDF vectors.Measured on a representative pack (40-row table across 2 pages, one line chart, one bar chart): 13 ms, 4.4 KB. No subprocess, no browser.
Build one typed aggregate object, and give each output its own renderer:
| Output | Renderer |
|---|---|
| ReportLab document builder | |
| Email digest | HTML with inlined styles |
| Slack digest | Slack blocks |
| CSV rollups | The existing writers |
All four read the same aggregate object, so the numbers always agree. Sharing a data model is stronger than sharing a markdown string. Markdown cannot express a chart, a page break, or a repeated table header, so it would have leaked layout concerns into the shared layer anyway.
report_id.This runs in the existing Celery report task.
backend/Dockerfile:124) and
already launches it for the web connector. It also renders correctly with no
network (verified below). It still loses: a browser launch costs about 470 ms
and a few hundred MB of RSS, versus 13 ms for ReportLab, and it drags a
browser process, a template layer, and hand-written SVG into the report path.Verified inside the shipped image with docker run --network none:
Note that the air-gap CI job only starts api_server, inference_model_server,
and minio. It does not exercise the background worker or a browser. Any
air-gap claim for a new render path needs its own check.
Do not put these in a report. They look informative and drive no decision:
The investigation export stays. Fix its defects:
summary.csv, manifest.json (schema version, period, timezone, row
counts, generator version), and a README.md that defines every column.users.csv: email, role, groups, created date, last active, seat state.Ordered by value per unit of work. The first two are independent of each other.
users.csv and derive the dormant-seat list. Small
change, and it makes every other file joinable.