internal/planning/ai-security-scanner-evaluation.md
Revision history: Use git log -- internal/planning/ai-security-scanner-evaluation.md. Vendor snapshot:
Re-check product names, URLs, plans, and language coverage before running the evaluation, then refresh this snapshot
annually or when a listed vendor is acquired, deprecated, or materially changes scope.
Evaluate whether AI-native security scanners can find actionable vulnerabilities or logic bugs in React on Rails beyond the issues already covered by existing lint, test, dependency, and code review workflows.
This plan tracks Issue 2018. It is an evaluation plan, not a decision to adopt any vendor or CI integration.
Public planning note: This file lives in a public repository even though the path starts with
internal/planning/. Here,internalmeans "not end-user documentation," not restricted access; GitHub search indexes this directory. Keep raw scanner findings, exploit details, private fork URLs, and intentionally vulnerable fixtures outside this repo; commit only sanitized summaries here.
Evaluate the open-source gem and npm package first:
react_on_rails/libreact_on_rails/lib/generatorspackages/react-on-rails/srcEvaluate React on Rails Pro separately after the OSS scan path is understood, because Pro contains separate package and licensing boundaries.
Compare scanner output against the current baseline before counting a finding as net-new signal:
.github/codeql/codeql-config.yml.github/dependabot.ymlNo dedicated bundler-audit (CLI: bundle-audit), npm audit, or pnpm audit CI workflow is configured at this
snapshot. If one is added before an evaluation run, include it in this baseline before counting dependency CVEs as
net-new signal.
Record whether each verified finding is missed by the existing baseline or whether the scanner mainly improves prioritization, explanation, or reachability analysis for a finding that existing tools already surface.
Start with this point-in-time vendor shortlist from the issue if the products still offer an appropriate plan at evaluation time.
| Vendor | Reference | Initial status | Before scheduling |
|---|---|---|---|
| ZeroPath | https://zeropath.com/ | Candidate | Confirm OSS or trial plan and Ruby/TypeScript coverage. |
| Corgea | https://corgea.com/ | Candidate | Confirm OSS or trial plan and Ruby/TypeScript coverage. |
| DryRun Security | https://dryrun.security/ | Candidate | Confirm OSS or trial plan and Ruby/TypeScript coverage. |
Deferred: Almanax was Web3-heavy as of 2026-05-09 with no clear Ruby/TypeScript coverage signal. Re-add it to the shortlist only when both signals are present at the next annual review: (1) vendor documentation explicitly names Ruby, Rails, Node, or TypeScript as supported languages or frameworks, and (2) a public changelog entry, customer case study, or release note shows non-Web3 coverage on a Rails or Node codebase. Record the URL for each signal in this section before promoting the vendor.
If the named vendors are unavailable or unsuitable, look for tools in these categories:
Prefer scanners that can run on a branch without requiring broad organization-level permissions. If a free or trial plan is not available, record that and defer paid adoption until at least one OSS scan produces a concrete finding worth validating.
Use a fixed branch and commit for the first comparison so results are reproducible:
mainBefore running scans, record the exact baselines used:
| Dataset entry | Source | Branch or URL | Commit SHA or version | Date recorded |
|---|---|---|---|---|
main | react_on_rails | main | <fill full SHA> | <fill date> |
| Known-issue fixture | Private fork or training project | <fill branch or URL> | <fill SHA/version> | <fill date> |
| Safe refactor | react_on_rails | <fill branch> | <fill full SHA> | <fill date> |
Do not run any scanner evaluation until every cell in this table is filled in and committed. Undocumented baselines produce results that cannot be reproduced or compared across evaluators. For the known-issue fixture, fill the table with a concrete private repository/branch or public training-project URL, commit or version, fixture owner, and access request path before scheduling any scanner. Until Issue 2018 names a narrower owner, repository maintainers with write access own fixture selection and access approval.
The follow-up issue, #3265 Track baseline prerequisites for AI security scanner evaluation, tracks this prerequisite, but closing that issue is not sufficient by itself. Before Issue 2018 moves from planning to execution, verify that this dataset table has no placeholder cells and that the completed table is committed on the evaluation branch.
The intentionally vulnerable fixture should be small and obvious, such as unsafe template evaluation in a test-only file. Do not commit intentionally vulnerable fixtures, secrets, real credentials, or exploit-ready application behavior to a public branch of this repository. Keep any private positive-control fixture non-indexable, clearly labeled test-only, and inert: no operational code paths, real network calls, or reusable exploit payloads. Limit access to the default triage group unless Issue 2018 assigns a narrower group for the evaluation.
Score each scanner against the same rubric (1 = poor, 3 = acceptable, 5 = excellent). Use a normalized weighted average
to make security signal and operational cost comparable across evaluators while keeping the final total on the 1-5 scale:
weighted score = score x weight; normalized weighted average = sum(weighted scores) / sum(weights).
Example: Actionability score = 4 with weight = 1.0 produces a weighted score of 4.0. If all eight criteria score 3,
the normalized weighted average is
(3 x 1.0 + 3 x 1.0 + 3 x 0.9 + 3 x 0.8 + 3 x 0.8 + 3 x 0.7 + 3 x 0.7 + 3 x 1.0) / 6.9 = 3.0,
where 6.9 is the sum of all weights. If any criterion or weight changes, recompute this denominator in the example and
in the final-score row before merging the change.
Current weight sum: 1.0 + 1.0 + 0.9 + 0.8 + 0.8 + 0.7 + 0.7 + 1.0 = 6.9. Update this line whenever a criterion or
weight changes.
The table below has eight scored criteria followed by a Final score row; that last row is a computed aggregate (its
weight is intentionally N/A) and must not be counted toward sum(weights).
| Criterion | Question | Score (1-5) | Weight (0-1) | Weighted score |
|---|---|---|---|---|
| Actionability | Does the finding name the concrete file, behavior, and reachable path? | 1.0 | ||
| Correctness | Can we reproduce or disprove the finding locally? | 1.0 | ||
| False-positive rate | What fraction of surfaced findings survive local verification as true positives? | 0.9 | ||
| Ruby/Rails coverage | Does it understand Rails generators, helpers, and server rendering paths? | 0.8 | ||
| TypeScript/React coverage | Does it understand package exports, SSR utilities, and browser/runtime boundaries? | 0.8 | ||
| Permission model | Can it run with minimal GitHub permissions? | 0.7 | ||
| CI fit | Can results be advisory first, without failing every PR? | 0.7 | ||
| Maintenance cost | How much config, triage time, and vendor lock-in does it add? | 1.0 | ||
| Final score | Normalized weighted average: sum(weighted scores) / sum(weights); use the current weight sum above; recompute if weights change | N/A | <fill> |
Anchor examples:
main branch of react_on_rails and one candidate scanner from the shortlist.SECURITY.md for the disclosure process and approve timing with maintainers before public disclosure.Owner: Tracked in Issue 2018. If that issue is closed before adoption, repository maintainers with write access own the next triage decision. Default triage group: repository maintainers with write access, unless the issue assigns a narrower group. Default triage SLA: first response within five business days for high or critical alerts.
Do not add a scanner to CI until all of these are true:
main scan after one triage pass, measured among
high/critical findings reviewed for that scanner run, not cumulatively across scanners. This initial bar is
intentionally lenient to support first-pass data collection; tighten it to 10% or lower before moving findings from
advisory to blocking. Revisit this threshold after the second full triage cycle or after 90 days of CI advisory data,
whichever comes first. A scan with fewer than five high/critical findings cannot satisfy this false-positive-rate gate
by itself; manually review every finding, record the raw count and why the sample is too small for a stable rate, and
gather a larger sample before using this rate to justify CI adoption.If no scanner clears this bar, keep the issue as a record of what was tested and revisit later.