internal-packages/observability-map/README.md
Scores every webapp entry point on whether it could explain itself during an incident, and prints
the ones worth fixing. An entry point is a Remix loader or action under
apps/webapp/app/routes, 427 of them today.
The score is 19 out of 100. It is low because the webapp does not attach tenant identity to its failures: 11 of the 412 measured entry points name an environment, project, organization or user on a failure path. Everything else, when it breaks at 3am, gives you the route and the request id and nothing about whose request it was.
Every figure here comes from a run against the tree as it stands. How the scanner reaches a verdict, and what it refuses to decide, is in INTERNALS.md.
pnpm --filter @internal/observability-map run map # the whole tree
pnpm --filter @internal/observability-map run map --json # same, as JSON on stdout
pnpm --filter @internal/observability-map run map /api/v1/token # one route, with its check results
The whole-tree run also writes observability-map.json at the repo root, which --no-write
suppresses. Single-route mode takes either the route path the report prints (/api/v1/token) or the
file name (api.v1.token.ts). An exact match wins over the routes it is a prefix of, and an
ambiguous prefix warns and names the alternatives rather than picking one.
A pull request touching apps/webapp/app/routes or this package gets a sticky comment scanning head
against the tip of the base branch, with the score, what changed, and the current fix list. Every
comment names the head commit it was rendered for, as a link to the compare range, because the
comment is edited in place across pushes and otherwise says nothing about which push it reflects. It
is report-only: nothing in the comment or its score fails the build or blocks a merge. Separately,
this package's test suite runs as a required check, on pull requests touching apps/webapp/app and
on any other package through the internal job, and fails when a symbol the tool references stops
resolving in one of the three trees it reads (apps/webapp/app, packages/plugins/src,
internal-packages/rbac/src), or when the first route with an anticipated sensitive segment lands.
Each failure names the list to update (src/webappSymbols.test.ts).
See .github/workflows/observability-map.yml.
The workflow runs on every pull request and applies the path list as a gate inside the job rather
than as a paths: filter on the trigger. GitHub evaluates one of those per workflow, so a pull
request whose diff stops matching never starts the workflow at all, and the comment an earlier push
left then stands for ever showing findings that are no longer in the diff. The case that matters is
a pull request touching a route and other files whose author reverts the route change and keeps the
rest, which still has a diff and still does not match. So a pull request with a comment and nothing
left to compare gets the comment reconciled to its resolved state without scanning anything, and one
with neither pays for a single cheap job that reads the paths and looks for a comment.
The base is github.event.pull_request.base.sha rather than a merge base, which reviewers have
reported as a bug twice. actions/checkout on a pull_request event checks out GitHub's test merge
commit, whose two parents are base.sha and the PR head, so the head tree being scanned already
contains everything on the base branch up to base.sha. Diffing that against base.sha isolates
the pull request's own work. A real merge base would leave the intervening base-branch commits in
the head tree and out of the base tree, and attribute all of them to the pull request.
It is the mean score of the 412 entry points that had at least one applicable check, where an entry's score is the share of its applicable checks that passed.
One property of that definition will mislead you otherwise. Changing which routes a check applies
to moves the score without anything in the webapp changing. Widening the sensitive cohort from 26
routes to 67 gave auth-boundary 39 more routes to look at, 36 of which already passed it, and the
global went from 15 to 19 without a line of apps/webapp changing. Narrowing a check, or a refactor
that takes routes out of the denominator, runs the same way in reverse. A movement is evidence about
the codebase only once you have checked the CHECKS block below for an applicability change. Compare
fix lists, not scores.
The number is deliberately unflattering, and one platform change would move most of it. Nothing
central attaches a tenant: logger pushes { requestId, path, host, method } onto every line
through AsyncLocalStorage and forwards errors to Sentry, and the route builders log
logBoundaryError(message, error, url). If the auth path pushed environmentId through
trace(...), several hundred entry points would flip at once, and request-context would want
rethinking rather than celebrating.
request-context checks that a failure-path log names a tenant field. It does not check that the
value is real. Adding a synthetic environmentId: "obs-map" field to the first object argument of
all 139 in-catch log calls, with no other change, takes the global from 19 to 29 and the CONTEXT
figure from 11 to 98.
That is the tool verifying presence, not meaning, and it is not a bug to fix. Every check reads
syntax: a field name, a call, a binding reference. None can tell a genuine tenant id from a
hardcoded string with the right key. A reviewer owns whether the value behind the field is real, the
same way a Lighthouse accessibility score checks that an alt attribute exists and not that its
text describes the image. The number tells you where to look. It does not tell you what you will
find there.
src/mutationCorpus.test.ts applies 53 semantics-preserving or handling-deleting rewrites to the
whole route tree in a temp copy and asserts three things for each: the published global does not
rise, the mean over the routes measured in both runs does not rise, and for a semantics-preserving
rewrite no individual route's score rises or drops out of the measured set. Every laundering shape a
reviewer has found is an entry in src/mutations.ts, and each entry says which kind it is.
Two entries are the ones the design turns on. Deleting every catch clause in the tree drops the
score from 19 to 8, so the metric does not pay you for removing error handling. Wrapping every body
in try { ... } catch (e) { throw e } leaves the global at 19 and raises no route, so it does not
pay you for adding error handling that does nothing either.
Two holes are open and the corpus says so. A catch over try { 0; } is refused, but canRaise
accepts any call, so try { String(0); } reads as real error handling: it takes the tree from 19 to
44 and raises 224 routes. Telling an inert call from one that can throw needs types the scanner does
not have. The second, dead-conjunction-instanceof-if, is a dead condition rather than a dead arm:
selectsADistinctPath folds the arm, and literalTruth treats && as always null on purpose so a
live guard is never read as dead, so widening that fold is a different rule needing its own
measurement. Both run as expected failures with the residual written out beside them, so the claim
is "51 rewrites are defended and here are the two that are not", never "unpaddable".
The corpus takes minutes rather than seconds, under two on CI's runner and closer to eight on a
laptop, so it is gated behind OBS_MAP_MUTATION_CORPUS=1
and runs as its own CI job rather than in pnpm test. Run it if you change this package:
OBS_MAP_MUTATION_CORPUS=1 pnpm --filter @internal/observability-map exec vitest run \
src/mutationCorpus.test.ts --disable-console-intercept
authorization or by filtering on the caller's own id. All nine route
builders authenticate the request, but their authorization option is optional and
apiBuilder.server.ts runs the RBAC gate inside if (authorization), so a route can be
authenticated and scoped to nobody. That is the cross-org IDOR class apps/webapp/CLAUDE.md
names. It applies to 19 routes and 17 pass; both failures resolve their target organization from
the URL slug with no membership filter and put nothing but an ability check in front of it
(_app.orgs.$organizationSlug.settings.sso/route.tsx in its loader,
_app.orgs.$organizationSlug.settings.team/route.tsx in its action). The fix in each is
members: { some: { userId } } on the lookup.prisma.impersonationAuditLog.create in
models/admin.server.ts; the other 46 do not.audit-trail is excluded from the score. The other four are in it.
auth-boundary, auth-scope and audit-trail only look at routes src/sensitivity.ts calls
sensitive, and that cohort is the fix list's primary sort key, so what goes in it decides what a
reader sees first. 67 routes are in it today: credentials and tokens, envvars, billing and the two
billing settings the bare billing segment does not match, impersonation, membership and invites
and roles and the team page, the login surface, API keys, and org or project deletion. Calling a
guard never makes a route sensitive, and src/webappSymbols.test.ts fails if a symbol or path
segment in the vocabulary stops resolving in the webapp.
The check list describes a composite the number mostly is not, so the report discloses the shape instead of hiding it behind a weight. Today:
CHECKS
error-classification 166 applicable, 94 pass, 0 sole, global without it 10
auth-boundary 62 applicable, 59 pass, 0 sole, global without it 15
auth-scope 19 applicable, 17 pass, 0 sole, global without it 18
request-context 412 applicable, 11 pass, 223 sole, global without it 65
audit-trail 49 applicable, 3 pass, 0 sole, not in the score
sole says the most: 223 of the 412 measured entry points have exactly one applicable scored check,
so their score is 0 or 100 on a single boolean. Read the family bars with that in mind. They do not
compare families on observability in general; they mostly compare them on whether someone wrote a
tenant field into a catch log.
Weighting was considered and rejected, because a coefficient nobody can explain invites argument
about the number instead of about the finding. The block above is in the terminal report and in the
JSON as checkContributions.
audit-trail fails 46 of 49 and request-context fails 401 of 412. Printing either one per route
would bury the route-specific findings under the same sentence repeated hundreds of times, so both
are reported as a figure: the AUDIT and CONTEXT lines. 328 entry points fail nothing except
request-context and appear only in that figure, which leaves 76 in the fix list. An entry that
fails request-context and another scored check keeps both findings and stays in the list, so
/account/tokens still shows the whole picture. audit-trail does not count as "another" for this
purpose, being already a headline, so a route failing only request-context and audit-trail
collapses too (28 do today, all of them sensitive).
42 of those 328 are sensitive, so the CONTEXT line says how many. Read them out of
observability-map.json, where every entry keeps its full check results, rather than assuming the
fix list is the whole story.
request-context is still scored, unlike audit-trail. The gap it measures is real and the score
is meant to show it. Only the presentation collapses.
The rule every applicability decision follows: would this evidence necessarily be visible in the body if it existed?
A log call inside a catch would be, because the catch is right there in the body being read, so its
absence is evidence of absence and request-context fails the route. A guard on work that happens
inside an imported helper would not be, because neither the work nor the guard is in the body, so
auth-boundary reports not-applicable with a detail saying it could not verify rather than accusing
the route of being unguarded. resources.impersonation.ts is the worked example: it calls
clearImpersonation, which authenticates and writes an audit row in app/models/admin.server.ts, a
file this tool never opens. Five of the 67 sensitive routes sit out for this reason.
The failure mode the rule exists to prevent is a fix list whose top three entries are all wrong, which is what a check asserting more than its evidence supports produces.
An entry with no applicable scored check is measured: false, and it is left out of every mean the
report computes (src/score.test.ts: excludes an unmeasured entry point from the global mean,
excludes an unmeasured entry point from its family mean too). Its score field reads 100, a
placeholder for "nothing was measured here" that nothing averages, because the alternative of
letting unmeasured entries into the mean at 100 would let the tool look better the less it
understood. The header prints every count (412 measured, 15 unmeasured) so the denominator is
never hidden, and a family with nothing measured renders as not measured rather than as a full
green bar.
15 of the 427 routes are unmeasured because isTrivial rules them out before any check runs.
Trivial means a body of three statements or fewer, three or fewer calls, no try/catch, no builder
wrapping it, and nothing in the calls or the source naming a datastore or a service (prisma,
logger, fetch, redis, and the like). Parse the params, build a path, redirect: nothing there
for a check to find evidence in either way. Exclusion is a denominator exit, not a credit.
A route whose body is somewhere else is a different case. export { action } from "./handler.server"
and export const action = handleWebhook are not trivial: a redirect stub genuinely has nothing to
instrument, while a delegating route has work the scanner cannot see, and treating them alike would
delete a route from the metric whenever someone moved a body into a .server.ts file. Delegating
routes are counted apart from the unmeasured ones, listed on a DELEGATED line and carried in the
JSON as delegating, the same treatment a parse failure gets and for the same reason. There are
none in the tree today, which is exactly why the case needed writing down before someone wrote one.
// obs-map-disable auth-boundary -- public by design, see ADR 12
The reason is mandatory: a suppression without one is ignored. The directive is read from comments only, so a string literal quoting it does not switch a check off.
It applies to the whole entry point, not to the line under it. Line scoping is not available,
because a finding is attached to an entry point and carries no line number to match against, which
is why the old obs-map-disable-next-line spelling is not honoured.
A suppression cannot raise a score. The suppressed check leaves the numerator and the denominator,
and the result is capped by what the entry would have scored unsuppressed, so suppressing a failing
check holds the number still rather than improving it (src/score.test.ts: does not raise the score when a failing check is suppressed). What you buy is removal from the worklist with a reason
on the record. The report prints how many suppressions are in force so the practice stays visible.
Read these before trusting a specific verdict.
auth-boundary applies to 62 of the 67 sensitive
entry points; the other 5 hand their work to an imported helper and are reported as unverified
rather than unguarded.SOFT_GUARDS. src/webappSymbols.test.ts proves each one is declared somewhere;
nothing proves the declaration it found is the guard we meant. authenticateAdmin and
authenticatePlainRequest are local helpers inside one route file each, so a second route
declaring its own no-op function of either name would be credited.getUser and getUserId answer with
null instead of throwing, so calling one is not a boundary. They are credited only when the body
binds the result and some condition reads it (EntryPoint.checkedCallees). What that cannot see
is whether the test guards anything: if (!user) { logger.warn("anonymous"); } followed by the
work reads the same as returning.authenticate and isAuthenticated are unresolved on purpose. They are remix-auth's, and
resolving them means reading a path inside apps/webapp/node_modules, which fails confusingly on
an install-layout change. They are listed in EXTERNAL_GUARDS instead, so the resolution test
still rejects a name that is neither first-party nor listed.auth-scope applicable structurally implies auth-boundary pass. The check only applies to a
builder-wrapped route, and auth-boundary passes any builder-wrapped route, so all 19 carry the
same auth-boundary detail, "authenticated by the builder". That free point is a third or a
quarter of each of their scores: the 19 average 59.7 as scored and 44.6 with auth-boundary taken
out, and settings.team, a confirmed cross-org exposure, scores 25 rather than 0. Read the
finding rather than the score.auth-scope cannot tell a caller-id filter from a caller-id actor argument.
presenter.call({ userId: user.id }) narrows the query; generatePortalLink({ organizationId, userId: user.id }) records who asked. Both read as scoping. Separating them means following the
argument into the callee, so the four helpers credited this way (ApiKeysPresenter,
TeamPresenter, regenerateApiKey, DeleteOrganizationService, all of which do
members: { some: { userId } } and throw) were hand-read instead. No route in the tree passes on
an actor argument alone.auth-scope reads property assignments in that export's own handler. A handler that pulls the
id into a local first, const userId = user.id; ... { userId }, or that builds its filter in a
same-file helper, scopes itself and is not seen, so it would be reported as unscoped.auth-scope reads the builder-wrapped exports and says nothing about the rest of the file. A
route whose action is builder-wrapped and whose loader is a plain export async function loader
is judged on the action alone, and the pass detail, "every builder-wrapped export has an
authorization gate", is true of what it read while reading as a claim about the whole route. Ten
routes in the tree mix the two, and the one sensitive enough for the check to run on is
_app.orgs.$organizationSlug.settings._index/route.tsx, whose builder-wrapped action carries the
pass and whose plain loader filters on members: { some: { userId } }. That was hand-read.auth-boundary correctly and unhelpfully. /auth/sso,
/api/v1/authorization-code and /api/v1/token are unauthenticated by design: the caller is
anonymous at that point, which is the whole purpose. The check's statement about them is true and
there is nothing to fix, so they are candidates for a suppression comment with the reason on the
record.logger.* or
log.*. An aliased logger, one wrapped in a helper, or console.error is invisible, so a route
can be reported as recording nothing while it records plenty.error-classification withholds credit rather
than granting it and reopening the free-points path a single logger.error line wide.logger.error("failed", ctx) where ctx is a variable contributes nothing, and neither
does a second object.items.map((item) => { try {...} }) is
a fresh boundary per element, so its clause is not read as the route's own error handling. The
test is the method name, which cannot tell users.map from Result.map. Being wrong there costs
precision rather than points: a refused catch fails the route rather than excusing it.export { action } from "./x" beside a loader written in the file is not counted as delegating,
so half the route is scored and half is invisible.try { String(0); } still buys a pass. The open corpus entry above, and the largest single
hole known in the tool: measured live, it takes the tree from 19 to 44 and raises 224 routes.request-context reads the field name, never the
value, so a codemod writing environmentId: "obs-map" into every in-catch log call takes the
global from 19 to 29. Unlike the entry above this one is not a bug to fix, since no syntactic
check can tell a real tenant id from a constant, but it bounds what the number can mean either
way.scan.ts walks the routes directory and produces an EntryPoint per module, carrying only
body-scoped evidence. checks/ holds the five checks, each a pure function of an EntryPoint.
score.ts turns checks into an entry score and a report, report/ renders it, cli.ts is the
entry point. sensitivity.ts, triviality.ts and suppression.ts are the three inputs the checks
share.
Tests sit next to their subject in src/. Every check has a false-positive fixture, something it
must not flag, alongside the positive one. Keep that: most of the bugs this package has had were
checks that fired on the wrong thing, and a test that only proves the heuristic fires would have
caught none of them.