AI Fixes Security Flaws Correctly Only 26 Per Cent of the Time, Study Finds

Researchers at 1Password’s Off-by-1 Labs generated 6,080 security patches using two frontier AI models and found that only 26 per cent achieved a clean, fully successful fix. Roughly half the patches failed to close at least one existing exploit path, meaning a “patched” system could still be compromised via the route the patch was supposed to seal.

The research, reported by The Register on August 6, tested ChatGPT 5.5 at “medium” effort and Claude Opus 4.8 at “high” effort across six recently disclosed CVEs. The authors — Axel Mierczuk, Spencer Michaels, and Keith Hoodlet — propose the acronym FLAWED: Fix-Like Artifacts With Embedded Defects.

🔍 THE BOTTOM LINE

AI-generated security patches fail 74 per cent of the time. Half leave the original exploit path open. The researchers conclude that fully autonomous, non-human-reviewed AI patching has a “net-negative expected value by a considerable margin.” This is the first large-scale empirical study quantifying what security teams have suspected: AI can write patches, but it cannot reliably tell whether the patch actually works.

The Numbers

The breakdown of 6,080 generated patches:

26.0 per cent — fully successful, clean fix with no behaviour change
20.1 per cent — fixed the original issue but altered application behaviour (e.g. changing “allow list” logic to “deny list” logic)
2.3 per cent — fixed the issue but introduced new security problems
49.3 per cent — failed to fix at least one existing exploit path
2.2 per cent — failed to fix the vulnerability AND introduced a new exploit path

Even among the patches that technically worked, more than a third were rated “fragile” — the repair guarded against a specific attack vector without addressing the underlying problem. The code might block one way of exploiting a flaw while leaving other routes wide open.

The Guidance Problem

The most striking finding is how sensitive AI patching is to input quality. When models received correct initial guidance about a vulnerability, the fix-success rate reached 65 per cent. With no guidance, it dropped to 50.4 per cent. With incorrect guidance, it collapsed to 15.2 per cent.

The Register notes that human developers, by contrast, have a reasonable chance of catching misleading information as they reason through vulnerable code. The AI models tested did not.

This matters because real-world vulnerability reports are frequently incomplete or imprecise. If a model’s patching success rate drops from 65 per cent to 15 per cent when given bad guidance — and bad guidance is common in practice — then the real-world success rate of autonomous AI patching may be closer to 15 per cent than 26 per cent.

Why This Matters Now

The research lands at a moment when organisations face growing pressure to automate security remediation. Anthropic’s Mythos model identified 231 Microsoft vulnerabilities faster than Microsoft could issue fixes, creating a backlog that demands automated remediation. The temptation to let AI handle the patching pipeline is real.

But the 1Password research suggests that doing so without human validation could increase risk rather than reduce it. A patch that claims to fix a vulnerability but leaves the exploit path open is arguably worse than no patch at all — it creates a false sense of security while the vulnerability remains exploitable.

The AI Governance Institute notes that organisations under EU Cyber Resilience Act obligations that rely on AI-assisted patching pipelines face a material control gap: if AI-generated patches routinely leave exploit paths open, those gaps may count as unaddressed known vulnerabilities under emerging mandatory security standards.

The FLAWED Framework

The researchers have released their patch evaluation harness as an open-source tool called FLAWED, available on GitHub. Organisations can use it to evaluate the effectiveness of AI-generated security fixes before deploying them.

The tool tests whether a patch fully closes the vulnerability, whether it changes application behaviour, and whether it introduces new exploit paths. This is the kind of automated validation layer that would need to sit between AI patch generation and production deployment to make autonomous patching viable.

Without that layer, the research is clear: the math does not work. As ZDNet reported, AI is not ready to patch software without human oversight — and pretending otherwise creates more vulnerabilities than it fixes.

NZ Angle

New Zealand’s growing cybersecurity sector, including government agencies like the National Cyber Security Centre (NCSC), will need to factor this research into any AI-assisted security workflows. The NCSC’s Five Eyes call to action on AI preparedness already flags AI-enabled threats as a priority. This research adds a corollary: AI-enabled defences have their own failure rate, and assuming they work is itself a vulnerability.

For NZ businesses considering AI-assisted DevSecOps tools, the practical takeaway is straightforward: use AI to draft patches, but do not deploy them without human review. The 26 per cent success rate means three out of four AI-generated patches are wrong in some way.

❓ FAQ

What does the 26 per cent figure mean? Only 26 per cent of the 6,080 AI-generated patches fully resolved the vulnerability without changing application behaviour or introducing new issues. The remaining 74 per cent either failed to close the exploit, changed how the application worked, or introduced new security problems.

Which models were tested? ChatGPT 5.5 at “medium” effort and Claude Opus 4.8 at “high” effort. Both are frontier, cyber-capable reasoning models.

What is the FLAWED framework? An open-source patch evaluation harness released by 1Password’s Off-by-1 Labs. It tests whether AI-generated patches actually close vulnerabilities, whether they change application behaviour, and whether they introduce new exploit paths. Available on GitHub.

Does this mean AI is useless for security patching? Not exactly. With correct guidance, the success rate rises to 65 per cent. The researchers’ conclusion is that AI patching without human review has negative expected value — meaning the harm from broken patches outweighs the speed gains. AI-assisted patching with human validation remains viable.