Agentic Code Security Needs Evidence, Not More Findings
On September 10 and 11, 2026, GitHub rolled out two changes that show where AI-assisted code security is heading. Its AI Scan can now be enabled across repositories through organization and repository APIs, while Copilot code review can run builds, tests, targeted scripts, and other shell tools before presenting findings. One day earlier, GitHub also expanded agentic autofix for batches of code-quality findings.
The timing matters. Anthropic's September 10 threat report says the cyber operations it observed increasingly used AI for direct execution or orchestration across reconnaissance, exploitation, and data exfiltration. As attackers automate more of the chain, defenders cannot answer with a larger pile of probabilistic warnings. The useful unit is a verified security change: a scoped finding, a reproducible test, a patch, a failed re-attack, passing regression tests, and an enforceable merge decision.
That is the practical shift for engineering teams: stop measuring how many issues an AI reviewer mentions and start measuring how many findings survive verification.
A Finding Is a Hypothesis, Not a Security Result
Large language models are good at navigating unfamiliar code, connecting distant call paths, and suggesting plausible failure modes. Those abilities make them useful discovery engines. They do not make the first answer true.
Google's open-source Mantis security harness makes this distinction explicit. It warns that models are non-deterministic, can hallucinate findings, and can generate incorrect patches. Its workflow separates research, criticism, reproduction, patching, and re-attack rather than letting one agent declare its own output correct.
That separation matters because code-security false positives are unusually expensive. A noisy reviewer trains developers to ignore comments. A plausible but incorrect patch can remove a symptom while leaving the exploitable path open. An apparently successful reproducer may only trigger a debug-only condition or crash a different sink.
Treat every AI-generated finding as an untrusted hypothesis. Promotion to a security result should require machine-checkable evidence and an independent decision step.
Build a Proof Object for Every Finding
The most useful design pattern is to give each candidate vulnerability a durable record that accumulates evidence as it moves through the pipeline. It does not need to match a vendor's schema, but it should answer five questions:
- Where is the trust boundary? Record the untrusted input, required privileges, reachable assets, and the exact code path to the sensitive operation.
- Can the behavior be reproduced? Preserve a minimal input, build profile, environment, exit status, and evidence that the intended sink was reached.
- Is the behavior viable in production? Distinguish release builds from tests, samples, disabled features, and non-default configurations.
- Did the patch close the original path? Run the same reproducer against a clean patched workspace and verify that it fails for the expected reason.
- Did the patch preserve intended behavior? Run relevant unit, integration, and policy tests before proposing a merge.
The record should move through states such as needs_research, provisionally_valid, verified, patched, and verification_incomplete. A prose confidence score alone is not enough. Evidence must be attached to the transition.
Google's Mantis launch guidance recommends sandboxed reproduction and explicit vulnerability-acceptance criteria. It also describes using repository history and hierarchical summaries to supply architectural context. Those are not cosmetic improvements. They reduce two common failure modes: reporting behavior the project intentionally accepts, and missing a vulnerability because the model never learned the system's actual boundaries.
Separate Discovery, Validation, and Remediation
One model can perform every phase, but one uninterrupted conversation should not own every decision. Discovery agents are rewarded for breadth. Validators should be skeptical and try to disprove a candidate. Patchers should see the accepted evidence and the required invariants, not merely a persuasive vulnerability description.
A practical pipeline looks like this:
changed code
-> broad discovery
-> deduplicate and rank
-> isolated reproduction
-> human/policy acceptance
-> patch in a shadow workspace
-> re-run exploit and regression tests
-> draft pull request
-> merge gate
GitHub's September 11 Copilot code-review update points in this direction. The reviewer now uses a broader tool set to run builds, tests, and targeted scripts, and its Lite mode combines an ensemble of agents. GitHub reports more addressed high- and medium-severity comments in its experiments, but the transferable lesson is not the reported percentage. It is that review quality improves when the system can test claims and combine independent perspectives instead of only generating comments from a diff.
Keep the boundaries visible even when a platform bundles the phases. Log which component discovered the issue, which test reproduced it, which policy accepted it, and which run verified the patch. That provenance is what lets a team audit failures later.
Re-Attack the Patch, Then Test the Product
Re-running the original proof of concept is necessary but insufficient. A patch may overfit one payload while leaving equivalent paths open. After the first check passes, generate boundary variants: alternate encodings, nearby lengths, sign changes, different endpoints reaching the same sink, and reordered multi-step inputs.
The goal is not to prove that no vulnerability exists. That would be impossible. The goal is to demonstrate that the proposed fix survives a defined family of attacks without breaking legitimate behavior.
This is where an isolated shadow workspace is essential. Autonomous security tools execute attacker-controlled inputs and model-generated code. They should not have production credentials, unrestricted egress, shared writable caches, or access to internal control planes. TeqVolt's earlier agent-containment analysis explains why a container alone is not a complete boundary: reachable package mirrors, artifact stores, metadata services, and shared files can inherit broader authority.
Run the exploit and the product tests inside the same restricted environment, but preserve their results outside the agent's write access. If the verification harness can rewrite its own evidence, the pipeline is grading itself.
Turn Verification Into a Merge Policy
AI security becomes operational only when repository policy consumes the result. GitHub's September 10 AI Scan APIs let organizations roll out pull-request scanning programmatically. Its separate secret-scanning ruleset can block a merge until the scan finishes and newly introduced secret alerts are resolved.
Use that pattern for agentic findings, but keep the gate narrow. Block automatically only when the evidence is deterministic enough for the severity: a reproduced exploit, a policy violation, a known vulnerable dependency, an exposed secret, or a static-analysis result with a stable rule. Route uncertain architectural findings to review without stopping every pull request.
Define bypasses before rollout. Emergency overrides should require a named owner, a reason, an expiry, and an audit event. Otherwise teams will either disable the gate after the first incident or normalize permanent exceptions.
Measure Verified Fixes, Not Comment Volume
The wrong dashboard rewards activity: findings generated, comments posted, or patches proposed. A better dashboard follows outcomes:
- verified findings per thousand changed lines;
- false-positive rate after independent validation;
- median time from discovery to reproduced evidence;
- patch acceptance rate after re-attack and regression tests;
- reopened findings caused by bypasses or incomplete fixes;
- engineer time spent per accepted security change;
- compute cost per verified fix.
These metrics expose whether an agent is saving work or merely moving it into triage. They also let teams compare a fast, inexpensive model with a slower specialist on the result that matters.
This is especially important because the attack side is becoming more automated. Anthropic's September threat report says humans in observed campaigns still selected targets and reviewed stolen data, but AI increasingly executed or coordinated the intermediate work. Defensive review therefore needs machine-speed gates, while final authority over material risk remains accountable to people and policy.
Limitations and Tradeoffs
Agentic verification costs more than a single-pass review. Sandboxes take time to create, test suites may be slow or incomplete, and reproducing a vulnerability can itself require dangerous code. Small teams should start with high-risk repositories and trust boundaries rather than applying the deepest pipeline everywhere.
Vendor-reported improvements are not portable benchmarks. GitHub's internal review results reflect its own models, repositories, prompts, and acceptance measurements. Mantis is a demonstration framework, not a supported production product, and its own documentation requires expert verification.
Some vulnerabilities will remain non-reproducible because the environment, timing, hardware, or private dependencies cannot be recreated safely. Mark those cases as incomplete; do not silently convert missing evidence into either “false positive” or “verified.”
The Bottom Line
AI can widen security coverage, but coverage is not confidence. The durable advantage comes from turning each finding into a proof-carrying change: scoped, reproduced, patched, re-attacked, regression-tested, and enforced by repository policy.
Start with one critical service. Separate discovery from validation, run both inside a restricted shadow environment, preserve evidence outside the agent's control, and block merges only for findings that meet explicit criteria. If the system cannot explain why a patch is verified, it has produced more code—not more security.
