Skip to main content

AI Coding Agent Failure Modes

An AI coding agent failure mode is a recurring way an agent’s work goes wrong that a reviewer, a program or a test could catch: code that calls APIs that don’t exist, tests edited until they pass, changes nobody asked for, a “done” for work that isn’t. SAICA-Bench defines 11 of them, each with a plain definition, the other names it goes by and the signals that point to it. The modes are tags, not exclusive classes: one failure can carry several of them, and each mode’s page names the modes it is most easily confused with.

For 9 of the 11, SAICA-Bench measured what supervision tools and language models flag on labelled samples: see how the benchmark works and the full leaderboard. The table groups the modes by where the bad samples of each set come from, because that decides how a score should be read; inside a group the order is alphabetical. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. Terms are defined in the glossary.

All 11 failure modes

AI coding agent failure modes and what SAICA-Bench measured for each, grouped by where the bad samples come from
Failure mode What it is Also called Best reviewer on our set Best free catalog program Samples
Real agent output The bad samples are what an agent produced on its own; the label comes from running hidden tests.
Incomplete execution
scored as: False 'done' claims
J under 25% on our set
The agent reports the task done while parts are skipped, stubbed or not working. premature completion, false completion, stub left behind no reviewer reaches J 25%
highest J +19.5% (95% interval +13.9% to +24.9%)
none above the floor 1,200
Security vulnerability
J under 25% on our set
Correct-looking code with injection flaws, weak cryptography or leaked secrets. insecure code, code injection, secret leakage no reviewer reaches J 25%
highest J +7.9% (95% interval 0.0% to +15.9%)
none ran 1,702
Model-written code, not agent output The bad samples are what a code model wrote for a textbook task; the label comes from running the task's tests.
Logic error
J 25% to under 80% on our set
Real APIs used correctly, but the code computes the wrong result. semantic bug, off-by-one, inverted condition mimo-v2.6-flash
highest J +74.3% (95% interval +69.4% to +78.9%)
none ran 2,440
Failure planted on purpose The bad samples were written, injected or swapped in on purpose, by a model, a script or by hand.
Dependency blindness
J 80% or more on our set
Hand-written code for what the standard library, a dependency or the codebase already provides. reinvention, ecosystem duplication top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash
highest J +90.0% (95% interval +82.0% to +98.0%)
none ran 100
Fabrication
J 80% or more on our set
Code that calls functions, methods or packages that do not exist. code hallucination, API hallucination, package hallucination top group of 4: deepseek-v4.1-flash, glm-5.3 and 2 more
highest J +94.3% (95% interval +91.6% to +96.7%)
pyright
J +85.3%
636
Obsolescence
J 80% or more on our set
Real APIs that are deprecated, removed or changed, often from stale training data. deprecated API usage, version drift, training-cutoff drift top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash
highest J +89.7% (95% interval +82.9% to +95.1%)
pyright (latest libraries)
J +56.8%
310
Scope creep
J 80% or more on our set
Changes nobody asked for: other files, new packages, config edits, unrequested refactors. task drift, over-action, unrequested changes mimo-v2.6-flash
highest J +97.7% (95% interval +93.6% to +99.1%)
none ran 350
Supply-chain attack
J 25% to under 80% on our set
The agent installs malicious code: a squatted hallucinated name, a lookalike or a compromised tool. slopsquatting, typosquatting, dependency confusion top group of 2: gpt-6-luna and deepseek-v4-flash
highest J +77.0% (95% interval +70.3% to +83.8%)
none ran 296
Test manipulation
scored as: Test edits that hide failures
J 80% or more on our set
Tests edited, weakened or skipped so failures disappear instead of the bug being fixed. test gaming, spec gaming, reward hacking top group of 8: minimax-m3, giskard and 6 more
highest J +91.0% (95% interval +81.3% to +94.9%)
none ran 305
Not benchmarked yet No dataset with confirmed labels exists for these modes, so nothing is scored.
Cascading failure
Not measured
Each fix introduces new errors and the code drifts further from working. recovery spiral, error cascade, fix regression not measured — —
Context pollution
Not measured
The working context degrades over a session: facts dropped, noise added, old mistakes kept. context contamination, stale context, irrelevant context not measured — —

J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence. Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate. Samples = bad plus clean samples scored. Results of 2026-11-22; per-reviewer tables are on each mode’s page and on the leaderboard.

What the scores can and cannot say

On the 2 sets of real agent output (Security vulnerability and False 'done' claims; 2,902 samples) no reviewer reaches J 25%. Best J: +8% on Security vulnerability and +20% on False 'done' claims; 5 of the 23 ranked rows there have a 95% interval for J that excludes 0. On the 6 sets where the failure was planted on purpose the best reviewer reaches J +77% to +98%; on Logic error (model-written code that is not agent output) +74%. Every set of real agent output scores below every planted set, so this benchmark cannot tell 'reviewers catch these failures' from 'reviewers catch planted failures'. Whether the high scores carry over to failures agents make on their own is not tested.

J compares reviewers within one mode. Do not compare J across modes or average it over them: each set was built differently (real agent runs, model-written functions, injected or hand-built failures) and has its own sample unit, and where the bad samples come from explains most of the difference between modes. Tiers describe the best reviewer's J on our set, not how well a failure is caught in use: J 80% or more on our set; J 25% to under 80% on our set; J under 25% on our set.

All findings and caveats → · Threats to validity →

How these map to other taxonomies

Each mode is cross-walked to published agent-failure taxonomies. Bold = the same failure (exact match), plain = partial overlap; looser “related” links are on each mode’s page and on the taxonomy pages. Of the 20 mappings in this table, 0 are exact. Mapping drafted by an AI model from the source text; not yet reviewed by a person.

SAICA-Bench failure modes mapped to DAPLab, MAST, OWASP Agentic Top 10 and Microsoft AIRT categories
SAICA-Bench DAPLab 9 patternsMASTOWASP Agentic Top 10Microsoft AIRT
Incomplete execution —
FM-3.1 Premature termination
— —
Security vulnerability
DAP-06 Security Vulnerabilities
—
ASI02 Tool Misuse and Exploitation
ASI03 Identity and Privilege Abuse
ASI05 Unexpected Code Execution (RCE)
ASI07 Insecure Inter-Agent Communication
—
Logic error
DAP-03 Business Logic Mismatch
FM-1.1 Disobey task specification
— —
Dependency blindness
DAP-08 Codebase Awareness & Refactoring Issues
— — —
Fabrication — — —
MS-EF-06 Hallucinations
Obsolescence — — — —
Scope creep —
FM-1.1 Disobey task specification
ASI01 Agent Goal Hijack
ASI02 Tool Misuse and Exploitation
MS-ES-09 Excessive agency
Supply-chain attack — —
ASI04 Agentic Supply Chain Vulnerabilities
MS-ES-05 Function compromise and malicious functions
Test manipulation —
FM-3.3 Incorrect verification
— —
Cascading failure — — — —
Context pollution —
FM-1.4 Loss of conversation history
ASI06 Memory & Context Poisoning
MS-ES-01 Memory poisoning and theft

Where to go next