Method and system cardengine 0.1.0 · ruleset 2026.10.4 · report schema 1.0
How it works, what it checks, and where it stops.
This page documents the engine the way a system card documents a model: what it is for, how it is built, every rule it can report, how the optional language model is constrained, what it defends against, what it stores, and what it cannot do.
1Intended use
A second reader for code review, not a gate on its own.
Designed for
- Pointing reviewers at the lines of a change most likely to need attention, with the evidence for each.
- Catching mechanical breakage before review: callers that no longer bind, removed symbols still referenced, new lint errors.
- Showing which changed lines no test executed, from a coverage report you already produce.
- Gating CI on deterministic and heuristic findings above a chosen severity.
Not designed for
- Deciding whether a change is safe to ship. An empty report means no rule matched, not that the change is correct.
- Estimating the probability of a production incident. No score here is a probability.
- Replacing tests, type checkers, or human review of meaning: semantic regressions are its weakest category.
- Analysing languages without structural support beyond the text-level rules (secrets, migrations, dependencies).
2Architecture
One pipeline, four ways in, nothing executed.
ChangeGuard is a modular monolith: a Python engine (FastAPI, tree-sitter, Ruff) with a SQLite store, a Next.js web app that proxies the engine on its own origin, and a CLI that runs the same pipeline in CI. Every analysis goes through the same eleven stages, each reporting its status, timing and a structured summary.
untrusted inputs
Patch
Snapshot
Coverage
engine · parsed, never executed
01 Parse inputs
02 Reconstruct revisions
03 Structural diff
04 Cross-file references
05 Static analysis
06 Risk rules
07 Test mapping
08 Coverage
09 Assemble report
optional model zone · claims only
AI synthesis
Grounding verification
surfaces
Report JSON
REST API + SSE
Web app
CLI
Two revisions, rebuilt in memory
The patch is applied to the snapshot to recover both sides. Without a snapshot, analysis falls back to a diff-only mode that parses hunks as fragments and says so in the report.
Structure before text
Python, JavaScript, TypeScript and TSX are parsed with tree-sitter into symbols, signatures, imports and calls. Line-level rules handle everything else.
Differential by default
Ruff and the pattern rules only report what the change introduced. Pre-existing problems are never blamed on it.
Every stage is observable
Stage results, timings and structured summaries are part of the report and stream to the browser as server-sent events.
3Findings and provenance
Three kinds of finding, never blended.
Established by analysis of your inputs: a parse, a resolved call, a measured line. Evidence is the proof.
Inferred by a rule. Worth a reviewer's attention; impact depends on context, so confidence is stated explicitly.
Proposed by a model and kept only after verification. Confidence capped at medium; never fails a build.
Severity (critical, high, medium, low, info) is how bad the problem would be if real. Confidence (high, medium, low) is how sure the rule is that it is real. They are separate categorical judgements defined per rule. Neither is a probability, and they are never multiplied into a score.
Every finding cites evidence by ID (E1, E2, …): a diff hunk, a call site, a diagnostic, a coverage measurement. Finding IDs are derived from rule and location, so the same change yields the same IDs on every run.
Review priority: the first rule that matches, top to bottom
- BlockA critical finding established by deterministic analysis.
- HighAny high-severity deterministic or heuristic finding.
- ElevatedAny medium-severity finding. AI findings alone can raise priority this far and no further.
- RoutineOnly low-severity or informational findings.
4Rule catalogue
91 rules, each with a stated rationale.
40 rules are ChangeGuard’s own; 51 are curated Ruff rules run differentially on Python files. 75 are deterministic and 16 heuristic. Severities shift with context where the rule says so: a changed default value is medium when callers rely on it or the function is public and low otherwise, and a removed symbol is critical when other files still import it. Findings link here from their rule ID.
91 rules shown
- CG-API-001ChangeGuard
Call site incompatible with new signature
A caller still uses the old calling convention. In Python this raises TypeError at call time; in TypeScript it fails compilation.
Breaking changepython, javascript, typescript
High Deterministic - CG-API-002ChangeGuard
Removed symbol is still referenced
Code that imports or calls a removed or renamed symbol fails at import or call time.
Breaking changepython, javascript, typescript
Critical Deterministic - CG-API-003ChangeGuard
Breaking signature change
The signature changed incompatibly. No incompatible caller was found in the analysed code, but callers outside it (other services, packages, scripts) would break.
Breaking changepython, javascript, typescript
Medium Heuristic - CG-API-004ChangeGuard
Function became async but callers do not await it
Calling an async function without awaiting it returns a coroutine/promise instead of the value, silently changing behaviour.
Breaking changepython, javascript, typescript
High Deterministic - CG-API-005ChangeGuard
Default argument value changed
Callers that omit the argument now receive different behaviour without any code change on their side.
Behaviour changepython, javascript, typescript
Medium Heuristic - CG-API-006ChangeGuard
Return type annotation changed
Callers may depend on the previous return type.
Behaviour changepython, typescript
Low Heuristic - CG-API-007ChangeGuard
Public symbol removed
A public symbol was removed. No internal references remain, but external consumers may rely on it.
Breaking changepython, javascript, typescript
Low Heuristic - CG-SYN-001ChangeGuard
Syntax error introduced
The changed file no longer parses. It will fail at import/compile time.
Syntax errorpython, javascript, typescript
Critical Deterministic - CG-SEC-001ChangeGuard
Dynamic code execution
eval/exec/new Function execute strings as code; attacker-influenced input leads to code injection.
Securityjavascript, typescript, python
High Deterministic - CG-SEC-002ChangeGuard
Unsafe HTML sink
Assigning untrusted strings to HTML sinks enables cross-site scripting.
Securityjavascript, typescript
High Deterministic - CG-SEC-003ChangeGuard
Shell command built from dynamic input
Passing data to a shell enables command injection.
Securityjavascript, typescript, python
High Deterministic - CG-SEC-004ChangeGuard
Unsafe deserialisation
pickle and yaml.load can construct arbitrary objects (and run code) from untrusted data.
Securitypython
High Deterministic - CG-SEC-005ChangeGuard
TLS certificate verification disabled
Disabling certificate checks allows man-in-the-middle interception.
Securitypython
High Deterministic - CG-SEC-006ChangeGuard
SQL built from string formatting
Interpolating values into SQL text enables SQL injection; use parameterised queries.
Securitypython
High Deterministic - CG-SEC-010ChangeGuard
Hard-coded secret
Credentials committed to source control are exposed to everyone with repository access and persist in history after removal; they must be rotated.
Secret exposureany file
Critical Deterministic - CG-SEC-011ChangeGuard
Environment file committed
Environment files usually hold deployment secrets and should not be committed.
Secret exposureany file
Medium Deterministic - CG-ERR-001ChangeGuard
Exception silently swallowed
An empty or bare handler hides failures, turning errors into silent data corruption or missing work.
Error handlingjavascript, typescript, python
Medium Deterministic - CG-ERR-002ChangeGuard
Error handling removed
A try/except (try/catch) block was removed from a modified function; failures that were handled now propagate to callers.
Error handlingpython, javascript, typescript
Medium Heuristic - CG-CON-001ChangeGuard
Async callback passed to forEach
Array.prototype.forEach ignores returned promises: iterations run concurrently and errors are unhandled.
Concurrencyjavascript, typescript
Medium Deterministic - CG-CON-002ChangeGuard
Blocking call inside async function
A blocking call inside a coroutine stalls the event loop and every concurrent request.
Concurrencypython
Medium Deterministic - CG-DBG-001ChangeGuard
Debug artifact left in code
Debug statements leak data to logs or pause execution in production.
Debug artifactjavascript, typescript, python
Low Deterministic - CG-TYP-001ChangeGuard
Type checking suppressed
Suppressing the type checker hides the class of errors the compiler would otherwise catch.
Type safetytypescript, python
Low Deterministic - CG-CFG-001ChangeGuard
Risky configuration value
Debug mode or wildcard CORS in application configuration weakens production security.
Configurationany file
Medium Heuristic - CG-TST-001ChangeGuard
No related tests found for changed code
No test in the snapshot imports and references the changed symbol, so a regression may go undetected. Tests that exercise it indirectly are not detected.
Test gapany file
Medium Heuristic - CG-TST-002ChangeGuard
Test disabled
A skip marker removes the test from the suite; the behaviour it covered is no longer verified.
Test integrityany file
Medium Deterministic - CG-TST-003ChangeGuard
Focused test committed
.only restricts the runner to the focused test(s); every other test in the file stops running in CI.
Test integrityjavascript, typescript
High Deterministic - CG-TST-004ChangeGuard
Assertions removed from test
Fewer assertions means the test verifies less behaviour than before.
Test integrityany file
Medium Deterministic - CG-TST-005ChangeGuard
Tests deleted
Deleted tests reduce regression protection; confirm the behaviour they covered is gone or re-tested.
Test integrityany file
Medium Deterministic - CG-COV-001ChangeGuard
Changed lines not covered by tests
The uploaded coverage report marks these changed executable lines as never executed by the test suite.
Coverage gapany file
Medium Deterministic - CG-COV-002ChangeGuard
Changed file absent from coverage report
The coverage report has no data for this changed source file; it may not be imported by any test.
Coverage gapany file
Low Heuristic - CG-MIG-001ChangeGuard
Destructive schema migration
Dropping or rewriting tables/columns destroys data and breaks code still reading it during a rolling deploy.
Data migrationany file
High Deterministic - CG-MIG-002ChangeGuard
Dropped column still referenced in code
Application code still references a column this migration removes or renames.
Data migrationany file
High Heuristic - CG-DEP-001ChangeGuard
Major version change of a dependency
Major releases commonly contain breaking API changes.
Dependencyany file
Medium Heuristic - CG-DEP-002ChangeGuard
New dependency added
New third-party code enters the supply chain and should be reviewed for maintenance and licence.
Dependencyany file
Low Heuristic - CG-DEP-003ChangeGuard
Dependency removed
Code that still imports a removed dependency fails at import time.
Dependencyany file
Medium Heuristic - CG-DEP-004ChangeGuard
Version constraint loosened
A looser constraint lets future installs pull untested versions.
Dependencyany file
Low Heuristic - CG-LOG-001ChangeGuard
Condition logic changed
A comparison/boolean operator or negation changed. Boundary behaviour differs, which is a classic source of off-by-one and inverted-condition bugs.
Logic changeany file
Medium Heuristic - CG-LOG-002ChangeGuard
Boundary constant changed
A numeric constant in a condition changed, moving a threshold.
Logic changeany file
Low Heuristic - CG-CPX-001ChangeGuard
Complexity increased
Higher cyclomatic complexity means more paths to test and more room for untested edge cases.
Complexitypython, javascript, typescript
Low Heuristic - CG-AIS-001ChangeGuard
Prompt-injection text in change
Text addressed to AI reviewers attempts to manipulate automated review. ChangeGuard neutralises it before any model sees the code, and AI output cannot remove deterministic findings.
Prompt injectionany file
Medium Deterministic - RUFF-invalid-syntaxRuff, differential
Ruff invalid-syntax
The file no longer parses.
Syntax errorpython
Critical Deterministic - RUFF-F821Ruff, differential
Ruff F821
Undefined name: raises NameError when the line executes.
Correctnesspython
High Deterministic - RUFF-F823Ruff, differential
Ruff F823
Local variable referenced before assignment: raises UnboundLocalError.
Correctnesspython
High Deterministic - RUFF-F811Ruff, differential
Ruff F811
Redefinition of an unused name shadows the earlier definition.
Correctnesspython
Medium Deterministic - RUFF-F632Ruff, differential
Ruff F632
iscomparison with a literal depends on interning; use ==.Correctnesspython
Medium Deterministic - RUFF-F601Ruff, differential
Ruff F601
Dictionary key repeated; the earlier value is silently discarded.
Correctnesspython
Medium Deterministic - RUFF-F704Ruff, differential
Ruff F704
yield outside a function is a SyntaxError.
Correctnesspython
High Deterministic - RUFF-F706Ruff, differential
Ruff F706
return outside a function is a SyntaxError.
Correctnesspython
High Deterministic - RUFF-F822Ruff, differential
Ruff F822
Name listed in __all__ is not defined.
Correctnesspython
Medium Deterministic - RUFF-B006Ruff, differential
Ruff B006
Mutable default argument is shared between calls.
Correctnesspython
Medium Deterministic - RUFF-B008Ruff, differential
Ruff B008
Function call in default argument runs once at definition time.
Correctnesspython
Low Deterministic - RUFF-B012Ruff, differential
Ruff B012
return/break/continue in finally swallows in-flight exceptions.
Error handlingpython
High Deterministic - RUFF-B015Ruff, differential
Ruff B015
Comparison result is discarded; this was probably meant to be an assert.
Correctnesspython
Medium Deterministic - RUFF-B018Ruff, differential
Ruff B018
Useless expression has no effect.
Correctnesspython
Low Deterministic - RUFF-B020Ruff, differential
Ruff B020
Loop variable overrides the iterable it iterates.
Correctnesspython
Medium Deterministic - RUFF-B023Ruff, differential
Ruff B023
Closure captures the loop variable by reference (late binding).
Correctnesspython
Medium Deterministic - RUFF-B904Ruff, differential
Ruff B904
Exception re-raised without
from, losing the original cause.Error handlingpython
Low Deterministic - RUFF-B011Ruff, differential
Ruff B011
assert Falseis removed under python -O.Correctnesspython
Low Deterministic - RUFF-E722Ruff, differential
Ruff E722
Bare except also catches SystemExit and KeyboardInterrupt.
Error handlingpython
Medium Deterministic - RUFF-BLE001Ruff, differential
Ruff BLE001
Blind
except Exceptionhides unexpected failures.Error handlingpython
Low Deterministic - RUFF-S110Ruff, differential
Ruff S110
try/except/pass silently swallows errors.
Error handlingpython
Medium Deterministic - RUFF-S102Ruff, differential
Ruff S102
exec() runs arbitrary code.
Securitypython
High Deterministic - RUFF-S307Ruff, differential
Ruff S307
eval() runs arbitrary code; use ast.literal_eval for literals.
Securitypython
High Deterministic - RUFF-S301Ruff, differential
Ruff S301
pickle deserialisation of untrusted data executes arbitrary code.
Securitypython
High Deterministic - RUFF-S506Ruff, differential
Ruff S506
yaml.load without SafeLoader can instantiate arbitrary objects.
Securitypython
High Deterministic - RUFF-S602Ruff, differential
Ruff S602
subprocess with shell=True enables command injection.
Securitypython
High Deterministic - RUFF-S604Ruff, differential
Ruff S604
Function call with shell=True enables command injection.
Securitypython
High Deterministic - RUFF-S605Ruff, differential
Ruff S605
Starting a process with a shell enables command injection.
Securitypython
High Deterministic - RUFF-S608Ruff, differential
Ruff S608
SQL built with string formatting enables SQL injection.
Securitypython
High Deterministic - RUFF-S105Ruff, differential
Ruff S105
Hard-coded password string.
Secret exposurepython
Medium Deterministic - RUFF-S106Ruff, differential
Ruff S106
Hard-coded password passed as an argument.
Secret exposurepython
Medium Deterministic - RUFF-S107Ruff, differential
Ruff S107
Hard-coded password default argument.
Secret exposurepython
Medium Deterministic - RUFF-S501Ruff, differential
Ruff S501
TLS certificate verification disabled (verify=False).
Securitypython
High Deterministic - RUFF-S323Ruff, differential
Ruff S323
Unverified SSL context disables certificate checks.
Securitypython
High Deterministic - RUFF-S324Ruff, differential
Ruff S324
Insecure hash function (MD5/SHA1) used.
Securitypython
Medium Deterministic - RUFF-S311Ruff, differential
Ruff S311
Non-cryptographic random generator used.
Securitypython
Low Deterministic - RUFF-S113Ruff, differential
Ruff S113
HTTP request without a timeout can hang indefinitely.
Correctnesspython
Low Deterministic - RUFF-S201Ruff, differential
Ruff S201
Flask debug mode exposes an interactive debugger.
Securitypython
High Deterministic - RUFF-S701Ruff, differential
Ruff S701
Jinja2 autoescape disabled enables XSS.
Securitypython
High Deterministic - RUFF-S103Ruff, differential
Ruff S103
Overly permissive file permissions.
Securitypython
Medium Deterministic - RUFF-S202Ruff, differential
Ruff S202
tarfile.extractall without filtering allows path traversal.
Securitypython
Medium Deterministic - RUFF-S104Ruff, differential
Ruff S104
Binding to all interfaces exposes the service.
Configurationpython
Low Deterministic - RUFF-ASYNC210Ruff, differential
Ruff ASYNC210
Blocking HTTP call inside async function stalls the event loop.
Concurrencypython
Medium Deterministic - RUFF-ASYNC220Ruff, differential
Ruff ASYNC220
Blocking subprocess call inside async function.
Concurrencypython
Medium Deterministic - RUFF-ASYNC221Ruff, differential
Ruff ASYNC221
Blocking subprocess call inside async function.
Concurrencypython
Medium Deterministic - RUFF-ASYNC230Ruff, differential
Ruff ASYNC230
Blocking file I/O inside async function.
Concurrencypython
Low Deterministic - RUFF-ASYNC251Ruff, differential
Ruff ASYNC251
time.sleep inside async function blocks the event loop.
Concurrencypython
Medium Deterministic - RUFF-RUF006Ruff, differential
Ruff RUF006
Task created without keeping a reference may be garbage-collected.
Concurrencypython
Medium Deterministic - RUFF-T100Ruff, differential
Ruff T100
Debugger breakpoint left in code pauses execution.
Debug artifactpython
Medium Deterministic - RUFF-T201Ruff, differential
Ruff T201
print() left in non-test code.
Debug artifactpython
Low Deterministic - RUFF-PLE1142Ruff, differential
Ruff PLE1142
await outside async function is a SyntaxError.
Correctnesspython
High Deterministic
5AI system card
An optional model, held to the evidence.
- Cannot delete, downgrade, or rewrite a rule's finding; notes appear beside it.
- Every claim must cite evidence IDs, and is verified before display.
- Model-proposed risks have no location fields; location comes from the cited evidence.
- Model risks: confidence at most medium, priority at most elevated, never counted by --fail-on.
- Free-text summaries cannot be verified claim by claim and are never displayed as analysis.
- Secrets are masked and injection-like lines withheld before the prompt is built.
- Model output is never executed, and suggested test code is never run.
- llama3.2:3bprecision 98.4% · false alarms 0%
- qwen2.5-coder:7bprecision 53.4% · false alarms 100%
- qwen2.5-coder:7b@3precision 65.0% · false alarms 78%
Grounding held for every model: no unsupported evidence. Larger models proposed more risks, most matching no label. Explanation quality has not been rated by people. Full results.
Known failure modes
- Misattribution in free text: a recorded summary put a condition change in the wrong function. Such text stays in the trace only.
- Plausible but wrong risks: grounded claims that cite real evidence yet describe a problem that is not there.
- Context truncation: large changes are trimmed to the budget, so the model may not see every finding.
- Provider variance: hosted models change over time; recorded responses make evaluation reproducible, not live behaviour.
6Threat model
Everything it reads is treated as hostile.
Patches, archives, coverage reports, and the code inside them are attacker-controlled. The table lists each threat considered and the mitigation in place. Deployment guidance, including putting the web app behind access control when it should not be public, is in docs/SECURITY.md.
- A hostile archive: path traversal, symlinks, decompression bombs, millions of membersRead in memory, never extracted. Absolute paths,
.., drive letters and NUL bytes are rejected; links and devices are ignored; member count, per-file size and decompressed bytes are capped on bytes actually read; suspicious compression ratios are refused. - A malformed or adversarial patchStrict parser with size, line and file limits. Absolute and parent-relative paths are rejected. Stray content after a hunk is an error, not something to skip.
- XML attacks in coverage reports (entity expansion, external entities)Cobertura is parsed with defusedxml under a size limit.
- Code execution through the analysed repository: hooks, build scripts, linter configurationNothing from the repository runs. Ruff reads source through stdin with repository configuration ignored, no cache, a minimal environment and a timeout. The CLI's git mode disables hooks, external diff drivers and fsmonitor.
- Prompt injection hidden in code or commentsA rule flags instruction-like text; flagged lines are withheld from the model; every excerpt is fenced as untrusted data; model output is parsed as data and never triggers an action.
- Secrets in the changeDetected credentials are masked before a report, export or prompt is built, and are never sent to a model provider.
- Model hallucination presented as analysisA closed schema without location fields, a grounding verifier that rejects rather than repairs, and a separate, capped lane for model findings.
- Abuse of an exposed APIOptional API keys compared in constant time, a per-client sliding-window rate limit, request body limits, a per-analysis timeout, and a cap on stored analyses.
- Script injection into the web UI from analysed contentUntrusted text is rendered as React text nodes, never HTML; syntax highlighting produces token spans. The site sends a same-origin Content Security Policy and refuses framing.
- Leaking the engine credentials to browsersThe web app proxies the API on its own origin and adds the engine key server-side; the key never reaches the browser.
7Data handling
What is kept, where, and for how long.
Stored
Each analysis: its report (the diff hunks, cited excerpts, findings and stage results), the patch's SHA-256, options and progress events, in the engine's SQLite database. With AI enabled, model responses are cached there by request fingerprint.
Not stored
The uploaded archive, the coverage file, and the full repository. They are read in memory and discarded when the analysis ends.
Masked
Detected secrets are masked in everything stored, exported, or sent to a model.
Retention
Analyses are deleted on request from the Reports page or the API. The store keeps the 500 most recent by default.
Model providers
With a hosted model enabled, the redacted evidence pack is sent to that provider and its data policy applies. Local models keep everything on the machine.
Telemetry
None at runtime: neither the engine nor the web app calls out except to a model provider you configure. Fonts are bundled at build time, and the provided images disable Next.js build telemetry.
8Limitations
Known limits of the engine.
- 01Static, name-based reference resolution: dynamic dispatch, reflection, monkey-patching, and dependency injection are not followed.
- 02Severity and confidence are categorical judgements defined per rule; they are not calibrated probabilities of a production incident.
- 03Test mapping detects tests that import and reference a changed symbol; tests that exercise it indirectly are not counted.
- 04ChangeGuard never executes code or tests; coverage numbers come only from an uploaded report.
- 05Structural analysis covers Python, JavaScript, TypeScript and TSX. Other languages get line-level rules only.
- 06Coverage numbers are only as current as the uploaded report; a plausibility check lowers confidence when it looks stale, but cannot prove it fresh.
- 07Evaluation uses a small synthetic dataset written by the same author as the rules; see the evaluation for what that implies.