What the evaluation shows, and what it cannot.

The rules were written against the dev split. Two holdout sets were written afterwards, with the ruleset frozen, and each was run exactly once before anything changed. Those first runs are the closest thing here to an unbiased estimate. Every fix they prompted is listed, and later numbers on the same split no longer count as clean.

These numbers include the dev split the rules were written against and the holdout cases after their fixes, so they are optimistic by construction. The intervals are 95% percentile bootstrap intervals over cases; with this few cases they are wide, and a perfect point estimate still comes with a lower bound.

The ablation is the most informative row. Without the repository snapshot, the same rules keep their precision but recall falls from 93.8% to 72.3%: the missing share is what cross-file analysis (callers, imports, tests) contributes.

Each miss below is a change whose problem lies in its meaning rather than its shape. Nothing syntactic distinguishes these lines from a correct edit, so a rule that flagged them would also flag a great many correct edits.

This is where a language model could earn its place, and it partly does: qwen2.5-coder:7b found 2 of these 4. The section on language models shows what that cost.

Each model ran locally through ollama on all 67 cases with the same prompt (v1.0.0), greedy decoding, and a fixed seed. Every response is recorded, so these numbers replay exactly in CI without a model. The vote row samples three answers at temperature 0.7 and keeps a proposed risk only if a majority of samples agree.

Two results hold across models. First, grounding works: no system cited evidence that failed the independent re-check, and the verifier rejected the claims that named findings, paths, or symbols the evidence did not support. Second, grounded is not the same as right. The larger model proposed many additional risks that cite real evidence and still match no label. That is why model findings sit in their own lane, are capped at medium confidence, and can never fail a build.

Self-consistency helps, at a price. Voting over three samples dropped 21 of 57 model-proposed risks while keeping every one that matched a label, at roughly three times the tokens. The smaller model is quieter: it rarely proposes risks and mostly attaches explanations to findings the rules already made.

Not measured: whether the explanations are correct or useful. That needs human rating, and none has been done.

Deterministic and heuristic findings are counted separately, as are confidence levels. A finding is acceptable when the case lists it as a reasonable extra observation, or when it is a second finding on a label already matched; acceptable findings count as neither true nor false positives. Most heuristic findings land there: test gaps a reviewer would agree with, but that the labels do not require.

9 cases are safe changes with no label at all, so any finding on them is a false alarm. A review tool that cries wolf gets ignored, which makes this the number reviewers feel most.

Separately, an independent checker re-reads each case’s own files and verifies every evidence item a finding cites: it must exist, point at lines that exist in the stated revision, and quote those lines verbatim. A finding with any evidence that fails counts as unsupported.