Skip to main content

Logic errors in AI-generated code

A logic error is when an AI coding agent uses real, current APIs correctly but writes logic that computes the wrong result for the task, for example an inverted condition, an off-by-one or a faulty algorithm. It is the classic “passes the type check, fails correctness” bug. It differs from code hallucination, where the API does not exist, and from obsolescence, where the API is out of date. A fix that makes things worse is a cascading failure.

Nothing about the code looks wrong to a compiler, type checker or linter, so only tests, execution or a reader who understands the task can catch it. In the Gemini CLI case in the catalog, the agent misread a failed command as success and went on to destroy the user’s files.

At a glance: On SAICA-Bench: Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0. Results ↓

Also called
  • semantic bug
  • off-by-one
  • control-flow bug
  • incorrect implementation
  • faulty algorithm
  • reasoning error
  • inverted condition

How well can logic errors in AI-generated code be caught?

J 25% to under 80% on our set The code runs but computes the wrong result. Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0.

Best reviewer

any ranked row; a tie is shown as a top group

mimo-v2.6-flash

Highest J +74.3% (95% interval +69.4% to +78.9%)

Best catalog tool

a tool of the catalog running its own check

PR-Agent

J +53.2% (95% interval +47.9% to +58.3%)

Shortcut baseline

a rule that understands nothing about the failure

Surface classifier

J +39.4%. Read the reviewers against this, not against 0.

What was tested

Bad samples
Model-written code, not agent output. 11 code models of 2023-24 (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini) answering MBPP and HumanEval tasks; not agent runs. Label: execution: the function fails a test assertion of its task. Clean samples: the reference solutions of the same tasks.
One sample
one function; functions of the same task are grouped (intervals resample tasks).
Samples scored
1,536 wrong functions and 904 clean (63% of the set is bad).
Measure
J 25% to under 80% on our set.
Shortcut baseline
A rule with no understanding of the failure scores J +39.4% here (Surface classifier). Read reviewer scores against it, not against 0.

Threats to validity that apply to this set →

All results on this set

Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

Logic error: ranked reviewers, best first
# Reviewer 95% interval bad samples flagged clean samples flagged No verdictanswers left out False alarmsper 100 clean changes Precisionif 1% of changes are badPrecisionif 5% of changes are badPrecisionif 20% of changes are bad per 1,000 reviews
1 mimo-v2.6-flash, asked directly
Model asked directly · best reviewer
Run card
Model
mimo-v2.6-flash
Provider
Hyperbolic
What it was shown
task description; function source
Counts as flagged when
flagged when the answer carries "logic_error": true
Temperature
0
Output token limit
16384
First run
2026-11-20
Runs
2,440; failed 4 (0.2%): truncated 4
Samples
1,536 bad, 904 clean
Answers with no verdict
4 (1 bad, 3 clean): truncated 4
Cost
$0.23 per 1,000 reviews (api); $0.57 spent on this run. Charge reported by the API with each call (Hyperbolic)
+74.3%+69.4% to +78.9% 89.0%87.3% to 90.5% 14.7%12.5% to 17.1% 4 of 2,440 14.7 6%24%60% $0.23api
2 deepseek-v4-flash, asked directly
Model asked directly
Run card
Model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
task description; function source
Counts as flagged when
flagged when the answer carries "logic_error": true
Temperature
0
Output token limit
16384
First run
2026-11-20
Runs
2,440; failed 8 (0.3%): truncated 8
Samples
1,536 bad, 904 clean
Answers with no verdict
8 (0 bad, 8 clean): truncated 8
Cost
$0.31 per 1,000 reviews (api); $0.75 spent on this run. Charge reported by the API with each call (Hyperbolic)
+69.1%+64.1% to +73.8% 90.8%89.2% to 92.1% 21.7%19.1% to 24.5% 8 of 2,440 21.7 4%18%51% $0.31api
3 gpt-6-luna, asked directly
Model asked directly
Run card
Model
gpt-6-luna
Provider
Azure AI Foundry
What it was shown
task description; function source
Counts as flagged when
flagged when the answer carries "logic_error": true
Output token limit
16384
First run
2026-11-20
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.15 per 1,000 reviews (list); $0.37 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+68.5%+63.1% to +73.5% 89.3%87.6% to 90.7% 20.8%18.3% to 23.6% 0 20.8 4%18%52% $0.15list
4 PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Run card
Version
pr-agent 0.50.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
task description; function source
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
2,440; failed 1 (0.0%): tool error 1
Samples
1,536 bad, 904 clean
Answers with no verdict
1 (0 bad, 1 clean): tool error 1
Cost
$1.29 per 1,000 reviews (api); $3.16 spent on this run. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
+53.2%+47.9% to +58.3% 91.4%89.9% to 92.7% 38.2%35.1% to 41.4% 1 of 2,440 38.2 2%11%37% $1.29api
5 uqlm CodeGenUQ: 5 fresh solutions, functional-equivalence rate < 0.5 (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Run card
Version
uqlm 0.6.8
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
task description; function source
Counts as flagged when
Flagged when fewer than half of 5 fresh solutions are judged functionally equivalent to the reviewed function.
Temperature
1
Output token limit
8192
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
600; failed 23 (3.8%): nan score 23
Samples
a subset: 300 bad and 300 clean of the set's 1,536 and 904
Answers with no verdict
23 (1 bad, 22 clean): nan score 23; under 95% of one class was readable
Cost
~$3.81 per 1,000 reviews (api); $2.28 spent on this run. Estimated: run on the first 300 + 300 samples only (our cost $2.28), scaled to all 2440; rates are on those 600. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • CodeGenUQ's functional-equivalence scorer: 5 fresh solutions written at temperature 1.0 are judged against the reviewed function
  • flagged when fewer than half are judged equivalent (cut-off 0.5 chosen by us)
+52.5%+44.4% to +60.1% 96.0%93.1% to 97.7% 43.5%37.8% to 49.4% 23 of 600 (incomplete) 43.5 2%10%36% ~$3.81api
6 Phi-4-mini, asked directly
Model asked directly
Run card
Model
Phi-4-mini-instruct
Provider
Azure AI Foundry
What it was shown
task description; function source
Counts as flagged when
flagged when the answer carries "logic_error": true
Temperature
0
Output token limit
1024
First run
2026-11-20
Runs
2,440; failed 117 (4.8%): content filter 4, empty 9, no verdict 91, provider error 13
Samples
1,536 bad, 904 clean
Answers with no verdict
117 (69 bad, 48 clean): content filter 4, empty 9, no verdict 91, provider error 13; under 95% of one class was readable
Cost
$0.07 per 1,000 reviews (list); $0.18 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+22.0%+17.2% to +26.7% 61.7%59.2% to 64.1% 39.7%36.5% to 43.0% 117 of 2,440 (incomplete) 39.7 2%8%28% $0.07list
7 grok-4.1-fast (no reasoning), asked directly
Model asked directly
Run card
Model
grok-4-1-fast-non-reasoning
Provider
Azure AI Foundry
What it was shown
task description; function source
Counts as flagged when
flagged when the answer carries "logic_error": true
Temperature
0
Output token limit
8192
First run
2026-11-20
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.14 per 1,000 reviews (list); $0.34 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+11.9%+8.8% to +15.0% 93.5%92.1% to 94.6% 81.6%79.0% to 84.0% 0 81.6 1%6%22% $0.14list

Shown for reference, not ranked

These rows are never counted as best. A shortcut baseline shows what a rule with no understanding of the failure scores on this set; read the reviewers' scores against it.

Logic error: rows shown for reference and not ranked
Row and why it is not ranked J95% interval TPR FPR No verdict Costper 1,000 reviews
pyright
Catalog tool · Free program · own check · outside design scope
Not ranked: outside the tool's design scope: a type checker looks for type errors, not for wrong results; shown as a floor.
Run card
Version
pyright 1.1.414
What it was shown
function source
Counts as flagged when
Flagged when the output has any ' - error:' line.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Design scope
outside design scope: a type checker looks for type errors, not for wrong results; shown as a floor
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • any error line counts; the function is checked alone, without its tests
+0.3%−0.5% to +1.1% 1.5% 1.2% 0 $0
mypy (--check-untyped-defs --ignore-missing-imports)
Catalog tool · Free program · own check · outside design scope
Not ranked: outside the tool's design scope: a type checker looks for type errors, not for wrong results; shown as a floor.
Run card
Version
mypy 2.4.1
What it was shown
function source
Counts as flagged when
Flagged when the output has any ': error:' line.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Design scope
outside design scope: a type checker looks for type errors, not for wrong results; shown as a floor
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • --check-untyped-defs --ignore-missing-imports; any error line counts
−0.7%−2.0% to +0.5% 2.6% 3.3% 0 $0
ruff (pyflakes + bugbear rules)
Catalog tool · Free program · own check · outside design scope
Not ranked: outside the tool's design scope: a linter looks for lint findings, not for wrong results; shown as a floor.
Run card
Version
ruff 0.17.6
What it was shown
function source
Counts as flagged when
Flagged on any finding.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Design scope
outside design scope: a linter looks for lint findings, not for wrong results; shown as a floor
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • pyflakes (F) and bugbear (B) rules; any finding counts
−3.1%−5.4% to −0.9% 4.3% 7.4% 0 $0
Surface classifier: code tokens only, task never seen
Shortcut baseline
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
A logistic regression on which code tokens occur in the function, trained on four fifths of the tasks and applied to the remaining fifth. It never reads the task and runs nothing.
+39.4%+34.6% to +44.1% 69.7% 30.3% 0 $0
Formatting cues: any of four layout habits
Shortcut baseline
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
Flags a function that has a single-quoted string, trailing whitespace, a blank line inside the body or 'return (...)'. Correct functions are tidy reference solutions; wrong ones are raw model output.
+34.6%+30.1% to +39.1% 59.8% 25.2% 0 $0
How to read these numbers
  • J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
  • An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
  • 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
  • Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
  • A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
  • Precision is the share of flagged changes that are really bad, if the stated share of all changes were bad. It is computed from the row's TPR and FPR; the set's own share of bad samples is 63%.

The same answers, scored another way

Parts of the set, or the same set under another labelling. They show how far the headline J depends on how the set was built.

Mbpp · 882 bad, 477 clean

MBPP tasks only: one-line task descriptions, three hidden asserts per task.

RowJ95% intervalTPRFPR
mimo-v2.6-flash+57.8%+50.2% to +65.0%81.6%23.8%
gpt-6-luna+51.7%+44.0% to +59.1%82.1%30.4%
deepseek-v4-flash+51.2%+43.8% to +58.4%84.4%33.1%
PR-Agent+37.9%+31.0% to +44.6%85.3%47.4%
uqlm+36.6%+25.4% to +47.4%92.4%55.8%
Phi-4-mini+20.4%+13.9% to +26.9%64.5%44.2%
grok-4.1-fast+11.3%+7.1% to +15.6%91.4%80.1%
pyright+0.6%−0.3% to +1.6%0.8%0.2%
mypy0.0%−1.5% to +1.6%1.7%1.7%
ruff−4.5%−7.8% to −1.2%5.0%9.4%
Humaneval · 654 bad, 427 clean

HumanEval tasks only: the function signature and docstring are the task.

RowJ95% intervalTPRFPR
mimo-v2.6-flash+94.5%+91.3% to +97.2%98.9%4.5%
deepseek-v4-flash+90.6%+86.7% to +94.1%99.4%8.8%
gpt-6-luna+88.9%+83.6% to +93.4%98.9%10.1%
uqlm+71.8%+62.1% to +80.5%100.0%28.2%
PR-Agent+71.8%+65.0% to +78.4%99.7%27.9%
Phi-4-mini+23.2%+16.0% to +30.3%57.5%34.3%
grok-4.1-fast+13.0%+8.4% to +17.7%96.3%83.4%
pyright+0.1%−1.3% to +1.4%2.5%2.3%
mypy−1.3%−3.4% to +0.5%3.8%5.1%
ruff−1.8%−4.7% to +1.1%3.4%5.1%
Gpt4o mini · 98 bad, 198 clean

Wrong functions written by gpt-4o-mini, the one recent generator in the set, against the correct reference solutions of the same tasks.

RowJ95% intervalTPRFPR
mimo-v2.6-flash+33.2%+18.7% to +46.9%60.2%27.0%
gpt-6-luna+30.9%+16.9% to +44.4%63.3%32.3%
deepseek-v4-flash+27.4%+13.4% to +40.7%62.2%34.9%
PR-Agent+27.0%+14.2% to +39.3%69.4%42.4%
pyright−1.0%−2.7% to 0.0%0.0%1.0%
mypy−2.5%−6.6% to +0.5%2.0%4.5%
ruff−4.5%−10.7% to +1.2%4.1%8.6%
Phi-4-mini−6.1%−17.9% to +5.5%41.2%47.3%
grok-4.1-fast−11.3%−18.5% to −4.5%79.6%90.9%
More tables for this set in results.json
  • generators
  • tpr_by_generator: Share of each generator's wrong functions a reviewer flags, on readable answers. The false-positive rate is shared: every negative is a reference solution.
  • formatting_cues
  • surface_classifier
  • consensus_disagreements: Rows where the three highest-scoring models all give the answer opposite to the label, among the rows all three answered readably. Labels follow the hidden asserts; reviewers were told to judge against the task text only.

They are under modes[].tables of results.json.

How it was tested

  • 1536 functions that run but fail a test assertion vs 904 correct solutions, across 379 tasks. Code by 11 models (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini).
  • Every label re-checked by running the code against its tests (the original asserts of MBPP and HumanEval; MBPP has three per task): crashes, timeouts, 'wrong' outputs that actually pass and reference solutions that fail were dropped.
  • Reviewers see the task description and the function, not the tests. Same prompt for every model; temperature 0 (gpt-6-luna: its default). Run 2026-11-20.

Caveats

  • Short, self-contained functions from two famous public problem sets: models may know the right answers, so expect lower scores on real agent code.
  • Type checkers and linters look for a different kind of bug; they are here as a floor, not as fair competitors.
  • KG tools run through their own built-in check, with deepseek-v4-flash as their model: uqlm re-solves the task 5 times and flags code that disagrees with most re-solves; it was run on the first 300 + 300 functions only (it makes 10 model calls per review).
  • Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
  • Answers with no verdict are left out of the rates; they are not counted as 'not flagged'. Unreadable: Phi-4-mini 117 of 2,440 (69 bad, 48 clean; mostly answers without a verdict); uqlm 23 of 600 (1 bad, 22 clean; mostly no score returned); deepseek-v4-flash 8 of 2,440 (0 bad, 8 clean; mostly answers cut by the token budget); mimo-v2.6-flash 4 of 2,440 (1 bad, 3 clean; mostly answers cut by the token budget); PR-Agent 1 of 2,440 (0 bad, 1 clean; mostly the tool failed to run). Marked incomplete (under 95% readable in one class): Phi-4-mini, uqlm.
  • Run on part of the samples: uqlm (300 bad + 300 clean). Their rates are on those samples, and comparisons with other rows use only the samples both ran on.
  • The headline J blends two problem sets. For the three highest-scoring models, J is +89% to +94% on HumanEval and +51% to +58% on MBPP.
  • 65% of the wrong functions were written by open code models of 8 billion parameters or fewer (2023-24). On the 98 wrong functions of the one recent generator, gpt-4o-mini, the same three models catch 60% to 63% (other generators: 82% to 97%), and their J against the reference solutions of the same tasks is +27% to +33%. The surface classifier (code tokens only) scores +30% on those rows.
  • Every correct function is a reference solution and every wrong one is model-written, so layout gives the label away: a classifier that sees only code tokens, never the task, scores J +39%, and four formatting cues score +35%.
  • Labels follow the hidden asserts, while reviewers are told to judge against the task text only. On MBPP, 111 of 882 'wrong' functions are flagged by none of the three highest-scoring models and 85 of 471 'correct' ones by all three; a hand check of 24 of these rows in the review of 21 November 2026 found the label did not hold under that question in 19.
  • Textbook functions written by pre-agent models: this mode should not be averaged with the modes that score agent patches.

Logic error on the leaderboard → How this set was built → Threats to validity → Build a supervision kit →

How to detect it

Signals that point to logic error, from the SAICA knowledge graph:

  • Unit tests fail with correctness assertions (not import or type errors).
  • Execution in a sandbox produces outputs that contradict the task spec.
  • Mutation testing reveals that substitution of logic primitives does not change test outcomes (test insensitivity is a different signal, but logic-error cases often co-occur).
  • Symbolic or property-based tests find counterexamples.

Real-world incidents

Tools in the catalog that address it

6 tools are tagged with logic error in the catalog, most-starred first. A tag is the catalog's claim about what a tool is for; it is not a test result. Where SAICA-Bench ran the tool on this mode, the measured J is shown: that row is the evidence.

  • Ruff measured J −3.1% (not ranked: outside its design scope) Extremely fast Python linter and formatter, written in Rust.
  • ESLint Pluggable JavaScript/TypeScript linter and code-quality enforcer.
  • pr-agent measured J +53.2% AI-powered tool for automated pull-request analysis, feedback, and suggestions.
  • Pyright measured J +0.3% (not ranked: outside its design scope) Microsoft's fast static type checker for Python.
  • pytest Python test framework — discipline gate for "done means tests pass."
  • uqlm measured J +52.5% Uncertainty Quantification for Language Models.

In other taxonomies

How logic error maps to other failure taxonomies. “Exact” means the same failure, “partial” an overlap, “related” a neighbouring category.

  • 9 Critical Failure Patterns of Coding Agents

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    • partial DAP-03 Business Logic Mismatch — The post's wording, "code that runs without errors, but it produces the wrong output because the underlying logic is flawed", is the definition of logic_error. DAP-03 is the narrower side: it covers business rules, permissions and calculations only.
  • Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    • partial SHAH-CAT-04 Input Interpretation — Code that runs on real APIs but parses, validates or converts its input wrongly is a logic_error when a coding agent writes it. Narrower than logic_error (input handling only) and observed in framework code. Was "exact" before 2026-11-21.

    Related: SHAH-CAT-12 Failure Handling & Implementation Robustness

  • MAST: Multi-Agent System Failure Taxonomy

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    • partial FM-1.1 Disobey task specification — FM-1.1 includes code that runs but misses a stated requirement (the paper's ChatDev Wordle example ships a fixed word list against the spec), which is logic_error. FM-1.1 also covers non-code violations.

    Related: FM-3.2 No or incomplete verification

  • OWASP Top 10 for Agentic Applications 2026

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    Related: ASI02 Tool Misuse and Exploitation; ASI08 Cascading Failures

  • Swiss Cheese Model for AI Safety: Multi-Layered Runtime Guardrail Taxonomy

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    Related: SC-QA-01 Accuracy

  • Taxonomy of Failure Mode in Agentic AI Systems (Microsoft AI Red Team, version 1.0)

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    Related: MS-EF-07 Misinterpretation of instructions

  • Reinventing existing code (dependency blindness) — Hand-written code for what the standard library, a dependency or the codebase already provides.
  • Cascading failures — Each fix introduces new errors and the code drifts further from working.
  • Test manipulation — Tests edited, weakened or skipped so failures disappear instead of the bug being fixed.
  • False “done” claims (incomplete execution) — The agent reports the task done while parts are skipped, stubbed or not working.

All 11 AI coding agent failure modes → · Glossary

Research

Questions and answers

What is a logic error in AI-generated code?

A logic error is when an AI coding agent uses real, current APIs correctly but writes logic that computes the wrong result for the task, for example an inverted condition, an off-by-one or a faulty algorithm. It is the classic “passes the type check, fails correctness” bug. It differs from code hallucination, where the API does not exist, and from obsolescence, where the API is out of date. A fix that makes things worse is a cascading failure.

How do you detect logic errors in AI-generated code?

Signals that point to it: Unit tests fail with correctness assertions (not import or type errors). Execution in a sandbox produces outputs that contradict the task spec. Mutation testing reveals that substitution of logic primitives does not change test outcomes (test insensitivity is a different signal, but logic-error cases often co-occur). Symbolic or property-based tests find counterexamples. On SAICA-Bench: Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0.

Which reviewers catch logic errors in AI-generated code on SAICA-Bench, and how well?

Best reviewer: mimo-v2.6-flash at J +74.3%. Its paired difference from the next row (deepseek-v4-flash) has a 95% interval that excludes 0. The best catalog tool that ran its own check is PR-Agent, at J +53.2%. A rule with no understanding of the failure scores J +39.4% on the same set (Surface classifier), so read the scores against that, not against 0. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

How was detection of logic errors in AI-generated code measured?

On 2,440 scored samples: 1,536 wrong functions and 904 clean ones. Where the bad samples come from: model-written code, not agent output (11 code models of 2023-24 (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini) answering MBPP and HumanEval tasks; not agent runs). Label: execution: the function fails a test assertion of its task. Data: Collu-Bench (lt-asset, BSD-3-Clause-Clear): MBPP and HumanEval functions written by 11 code models. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Benchmark results from 2026-11-22. The definition, detection signals and tags come from the SAICA knowledge graph.

View raw YAML on GitHub →