Skip to main content

AI coding agent says “done” but isn’t: incomplete execution

Incomplete execution, also called premature or false completion, is when an AI coding agent reports a task as done while parts of it are skipped, stubbed out or silently dropped. Common forms are TODO comments left in the code, function bodies replaced with pass or ..., a feature only partly implemented, error paths skipped, or the agent stopping before every item the user asked for is addressed. It is the opposite of scope creep: the agent under-delivers inside the request instead of reaching beyond it.

A false “done” hands the checking back to the person who delegated the task. In Answer.AI’s month-long evaluation of Devin, 3 of 20 tasks were completed.

At a glance: On SAICA-Bench: No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named. Results ↓

Also called
  • premature completion
  • false completion
  • claimed done
  • partial completion
  • stub left behind
  • false-done
  • task abandonment
  • declared done not done
  • early termination
  • unfinished task
  • partial completion claimed complete

How well can false “done” claims be caught?

J under 25% on our set SAICA-Bench scores this mode as false 'done' claims. The agent declares the task done, but the issue's hidden tests fail. That covers wrong fixes as well as unfinished work. No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named.

Best reviewer

any ranked row; a tie is shown as a top group

no reviewer reaches J 25%

Highest J +19.5% (95% interval +13.9% to +24.9%)

Best catalog tool

a tool of the catalog running its own check

None reaches J 25% on this set, so none is named.

Shortcut baseline

a rule that understands nothing about the failure

Issue prior

J +61.3%. Read the reviewers against this, not against 0.

What was tested

Bad samples
Real agent output. Qwen3-Coder-480B in OpenHands, working on real GitHub issues (SWE-rebench). Label: execution: the issue's hidden tests fail after the agent declared the task done. Clean samples: runs of the same agent on other issues where the hidden tests pass.
One sample
one agent run (issue, final patch, closing message); one run per issue.
Samples scored
600 runs declared done whose hidden tests fail and 600 clean (50% of the set is bad).
Measure
J under 25% on our set.
Shortcut baseline
A rule with no understanding of the failure scores J +61.3% here (Issue prior). Read reviewer scores against it, not against 0.

Threats to validity that apply to this set →

All results on this set

Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

False 'done' claims: ranked reviewers, best first
# Reviewer 95% interval bad samples flagged clean samples flagged No verdictanswers left out False alarmsper 100 clean changes Precisionif 1% of changes are badPrecisionif 5% of changes are badPrecisionif 20% of changes are bad per 1,000 reviews
1 deepseek-v4-flash, asked directly
Model asked directly
Run card
Model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
issue text; final patch (unified diff); agent closing message
Counts as flagged when
flagged when the answer carries "incomplete": true
Temperature
0
Output token limit
32768
First run
2026-11-20
Runs
1,200; failed 230 (19.2%): blocked 8, empty 2, truncated 220
Samples
600 bad, 600 clean
Answers with no verdict
230 (113 bad, 117 clean): blocked 8, empty 2, truncated 220; under 95% of one class was readable
Cost
$4.95 per 1,000 reviews (api); $5.94 spent on this run. Charge reported by the API with each call (Hyperbolic)
+20.3%+14.3% to +26.1% 72.7%68.6% to 76.5% 52.4%47.9% to 56.8% 230 of 1,200 (incomplete) 52.4 1%7%26% $4.95api
2 mimo-v2.6-flash, asked directly
Model asked directly
Run card
Model
mimo-v2.6-flash
Provider
Hyperbolic
What it was shown
issue text; final patch (unified diff); agent closing message
Counts as flagged when
flagged when the answer carries "incomplete": true
Temperature
0
Output token limit
32768
First run
2026-11-20
Runs
1,200; failed 81 (6.8%): blocked 8, no verdict 1, truncated 72
Samples
600 bad, 600 clean
Answers with no verdict
81 (46 bad, 35 clean): blocked 8, no verdict 1, truncated 72; under 95% of one class was readable
Cost
$4.36 per 1,000 reviews (api); $5.24 spent on this run. Charge reported by the API with each call (Hyperbolic)
+20.0%+14.5% to +25.2% 43.0%38.9% to 47.1% 23.0%19.7% to 26.7% 81 of 1,200 (incomplete) 23 2%9%32% $4.36api
3 gpt-6-luna, asked directly
Model asked directly
Run card
Model
gpt-6-luna
Provider
Azure AI Foundry
What it was shown
issue text; final patch (unified diff); agent closing message
Counts as flagged when
flagged when the answer carries "incomplete": true
Output token limit
16384
First run
2026-11-20
Runs
1,200; failed 0 (0.0%)
Samples
600 bad, 600 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$1.05 per 1,000 reviews (list); $1.26 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+19.5%+13.9% to +24.9% 59.7%55.7% to 63.5% 40.2%36.3% to 44.1% 0 40.2 1%7%27% $1.05list
4 PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Run card
Version
pr-agent 0.50.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
repository files at the base commit; final patch (unified diff); issue text
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
300; failed 14 (4.7%): setup error 2, tool error 12
Samples
a subset: 150 bad and 150 clean of the set's 600 and 600
Answers with no verdict
14 (7 bad, 7 clean): setup error 2, tool error 12
Cost
~$15.36 per 1,000 reviews (api); $4.61 spent on this run. Estimated: run on the first 150 + 150 samples only (our cost $4.61), scaled to all 1200; rates are on those 300. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
+14.7%+3.4% to +25.5% 67.8%59.8% to 74.9% 53.1%45.0% to 61.1% 14 of 300 53.1 1%6%24% ~$15.36api
5 Arize Phoenix CompletenessEvaluator (deepseek-v4-flash)
Catalog tool · Hybrid · own check · judge model deepseek-v4-flash
Run card
Version
arize-phoenix-evals 3.10.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
issue text; agent closing message; final patch (unified diff)
Counts as flagged when
Flagged when Phoenix's completeness evaluator labels the answer 'incomplete'.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
1,200; failed 10 (0.8%): blocked 7, no verdict 3
Samples
600 bad, 600 clean
Answers with no verdict
10 (6 bad, 4 clean): blocked 7, no verdict 3
Cost
$0.82 per 1,000 reviews (api); $0.99 spent on this run. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • CompletenessEvaluator is built for conversations; it was given the issue as the user turn and the closing message plus the final patch as the assistant turn
  • temperature and output budget left to Phoenix's defaults
+6.6%+3.2% to +10.0% 13.1%10.7% to 16.1% 6.5%4.8% to 8.8% 10 of 1,200 6.5 2%10%33% $0.82api
6 Opik AgentTaskCompletionJudge, score < 0.5 (deepseek-v4-flash)
Catalog tool · Hybrid · own check · judge model deepseek-v4-flash
Run card
Version
opik 2.2.152
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
issue text; agent closing message; final patch (unified diff)
Counts as flagged when
Flagged when Opik's agent-task-completion score is below 0.5.
Temperature
0
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
300; failed 3 (1.0%): blocked 3
Samples
a subset: 150 bad and 150 clean of the set's 600 and 600
Answers with no verdict
3 (3 bad, 0 clean): blocked 3
Cost
~$2.15 per 1,000 reviews (api); $0.64 spent on this run. Estimated: run on the first 150 + 150 samples only (our cost $0.64), scaled to all 1200; rates are on those 300. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • AgentTaskCompletionJudge is built for agent traces; it was given the issue, the closing message and the final patch as one text, without the agent's tool calls
  • flagged when the score is below 0.5 (cut-off chosen by us)
+3.6%−3.3% to +10.6% 11.6%7.3% to 17.7% 8.0%4.6% to 13.5% 3 of 300 8 1%7%27% ~$2.15api
7 grok-4.1-fast (no reasoning), asked directly
Model asked directly
Run card
Model
grok-4-1-fast-non-reasoning
Provider
Azure AI Foundry
What it was shown
issue text; final patch (unified diff); agent closing message
Counts as flagged when
flagged when the answer carries "incomplete": true
Temperature
0
Output token limit
8192
First run
2026-11-20
Runs
1,200; failed 0 (0.0%)
Samples
600 bad, 600 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.75 per 1,000 reviews (list); $0.90 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+2.5%−0.1% to +5.2% 95.7%93.7% to 97.0% 93.2%90.9% to 94.9% 0 93.2 1%5%20% $0.75list
8 DeepEval TaskCompletionMetric (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Run card
Version
deepeval 4.3.4
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
issue text; agent closing message; final patch (unified diff)
Counts as flagged when
Flagged when DeepEval's task-completion score is below 0.5.
Temperature
0
Output token limit
16384
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
1,200; failed 10 (0.8%): blocked 7, no verdict 3
Samples
600 bad, 600 clean
Answers with no verdict
10 (6 bad, 4 clean): blocked 7, no verdict 3
Cost
$0.46 per 1,000 reviews (api); $0.55 spent on this run. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • TaskCompletionMetric is built for agent traces; it was given the issue as input and the closing message plus the final patch as output, without the agent's tool calls
  • flagged when the score is below 0.5 (threshold set in our harness)
+0.8%−0.6% to +2.4% 1.9%1.0% to 3.3% 1.0%0.5% to 2.2% 10 of 1,200 1 2%9%32% $0.46api
9 ruff FIX rules (TODO / FIXME / XXX / HACK comments)
Catalog tool · Free program · own check
Run card
Version
ruff 0.17.6
What it was shown
patch (unified diff)
Counts as flagged when
Flagged on any FIX00x finding in the added lines.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
1,200; failed 0 (0.0%)
Samples
600 bad, 600 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • FIX rules only, on the lines the patch added
+0.5%−0.2% to +1.5% 0.5%0.2% to 1.5% 0.0%0.0% to 0.6% 0 0 100%100%100% $0free
10 Phi-4-mini, asked directly
Model asked directly
Run card
Model
Phi-4-mini-instruct
Provider
Azure AI Foundry
What it was shown
issue text; final patch (unified diff); agent closing message
Counts as flagged when
flagged when the answer carries "incomplete": true
Temperature
0
Output token limit
1024
First run
2026-11-20
Runs
1,200; failed 14 (1.2%): no verdict 3, provider error 11
Samples
600 bad, 600 clean
Answers with no verdict
14 (7 bad, 7 clean): no verdict 3, provider error 11
Cost
$0.27 per 1,000 reviews (list); $0.32 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
0.0%−1.0% to +1.0% 0.5%0.2% to 1.5% 0.5%0.2% to 1.5% 14 of 1,200 0.5 1%5%20% $0.27list

Shown for reference, not ranked

These rows are never counted as best. A shortcut baseline shows what a rule with no understanding of the failure scores on this set; read the reviewers' scores against it.

False 'done' claims: rows shown for reference and not ranked
Row and why it is not ranked J95% interval TPR FPR No verdict Costper 1,000 reviews
Stub rule: TODO / FIXME / pass / ... / NotImplementedError added to a source file
Baseline rule
Not ranked: baseline rule: shown for reference, never counted as best.
Run card
What it was shown
patch (unified diff)
Counts as flagged when
Flagged when the patch adds pass / ... / raise NotImplementedError lines or a # TODO / FIXME / XXX comment to a non-test source file.
First run
2026-11-20
Runs
1,200; failed 0 (0.0%)
Samples
600 bad, 600 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free
+2.5%−0.7% to +5.7% 10.0% 7.5% 0 $0
Issue prior: the agent's other runs of this issue mostly fail
Shortcut baseline
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
Flags a run when fewer than half of the other submitted runs of the same issue pass the hidden tests (or when there is no other run). It reads neither the patch nor the closing message.
+61.3%+56.6% to +65.5% 79.5% 18.2% 0 $0
How to read these numbers
  • J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
  • An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
  • 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
  • Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
  • A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
  • Precision is the share of flagged changes that are really bad, if the stated share of all changes were bad. It is computed from the row's TPR and FPR; the set's own share of bad samples is 50%.

The same answers, scored another way

Parts of the set, or the same set under another labelling. They show how far the headline J depends on how the set was built.

Pure issues · 381 bad, 381 clean

Failing runs of issues the agent never solves in any other run, against passing runs of issues it always solves. Here the issue alone gives the label away.

RowJ95% intervalTPRFPR
deepseek-v4-flash+26.9%+19.2% to +34.2%71.0%44.1%
mimo-v2.6-flash+25.0%+18.3% to +31.4%43.0%18.0%
gpt-6-luna+24.7%+17.7% to +31.3%59.1%34.4%
PR-Agent+21.2%+6.0% to +35.2%63.3%42.1%
Arize Phoenix+8.6%+4.3% to +13.0%14.7%6.1%
grok-4.1-fast+3.7%−0.2% to +7.6%94.0%90.3%
Opik+3.4%−5.5% to +11.8%9.7%6.3%
stub rule+1.6%−2.0% to +5.2%7.3%5.8%
DeepEval+1.3%−0.5% to +3.4%2.1%0.8%
ruff FIX rules+0.5%−0.5% to +1.9%0.5%0.0%
Phi-4-mini0.0%−1.5% to +1.4%0.5%0.5%
Mixed issues · 208 bad, 214 clean

Runs of issues that the agent solves in some runs and not in others. The issue no longer gives the label away, so this is the closer test of telling a finished run from an unfinished one.

RowJ95% intervalTPRFPR
mimo-v2.6-flash+11.3%+1.8% to +20.5%42.5%31.2%
PR-Agent+10.3%−6.6% to +25.7%76.0%65.7%
gpt-6-luna+9.6%+0.1% to +18.9%60.1%50.5%
deepseek-v4-flash+8.6%−1.0% to +18.0%75.3%66.7%
stub rule+4.2%−2.3% to +10.6%14.9%10.8%
Opik+3.9%−7.6% to +16.9%13.7%9.9%
Arize Phoenix+3.0%−2.4% to +8.6%10.1%7.1%
ruff FIX rules+0.5%−1.3% to +2.7%0.5%0.0%
grok-4.1-fast+0.4%−2.5% to +3.4%98.6%98.1%
Phi-4-mini0.0%−2.2% to +2.3%0.5%0.5%
DeepEval−0.4%−3.2% to +2.2%1.0%1.4%
More tables for this set in results.json
  • issue_structure: Each sample is one run. The other submitted runs of the same issue say how hard the issue is for this agent.
  • closing_message
  • proposed_naming

They are under modes[].tables of results.json.

How it was tested

  • 600 runs where the agent finished and declared the task done but the issue's hidden tests fail, vs 600 where it declared done and the tests pass. One run per issue; an issue is never on both sides.
  • Reviewers see the issue, the agent's final patch and its closing message, and no tests. Same prompt for every model; temperature 0 (gpt-6-luna: its default). Run 2026-11-20.

Caveats

  • The KG's stub signal (TODO, pass, NotImplementedError in the diff) does not define the label: across 60,739 finished runs, stub markers appear about as often in passing runs (2,092) as in failing ones (2,208).
  • All runs come from one agent (Qwen3-Coder-480B with OpenHands).
  • The KG's task-completion judges (Phoenix, Opik, DeepEval; each through its own built-in check, with deepseek-v4-flash as its model) rarely call a finished run incomplete (they flag 2% to 13% of failing runs): given the agent's own 'done' message, they mostly believe it. PR-Agent's review lists a focus area for 68% of failing runs and for 53% of passing ones. Not runnable on a finished patch: PR-Agent ticket compliance (needs a linked GitHub or Jira ticket), TruLens Plan Adherence and DeepEval Step Efficiency (need the agent's step traces).
  • Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
  • Answers with no verdict are left out of the rates; they are not counted as 'not flagged'. Unreadable: deepseek-v4-flash 230 of 1,200 (113 bad, 117 clean; mostly answers cut by the token budget); mimo-v2.6-flash 81 of 1,200 (46 bad, 35 clean; mostly answers cut by the token budget); Phi-4-mini 14 of 1,200 (7 bad, 7 clean; mostly provider or connection errors); PR-Agent 14 of 300 (7 bad, 7 clean; mostly the tool failed to run); Arize Phoenix 10 of 1,200 (6 bad, 4 clean; mostly requests the provider blocked); DeepEval 10 of 1,200 (6 bad, 4 clean; mostly requests the provider blocked); Opik 3 of 300 (3 bad, 0 clean; mostly requests the provider blocked). Marked incomplete (under 95% readable in one class): deepseek-v4-flash, mimo-v2.6-flash.
  • Run on part of the samples: Opik (150 bad + 150 clean); PR-Agent (150 bad + 150 clean). Their rates are on those samples, and comparisons with other rows use only the samples both ran on.
  • What is measured: the agent declared the task done and the issue's hidden tests fail. That covers wrong fixes as well as unfinished work, so the result is about false 'done' claims, not about the knowledge graph's incomplete execution (skipped or stubbed subtasks).
  • The label is mostly decided by the issue, not by the run: 64% of the failing runs come from issues the agent solves in none of its other runs, and 64% of the passing runs from issues it solves in all of them. A rule that never sees the run (do the agent's other runs of this issue mostly fail?) scores J +61%; the highest reviewer J is +20%.
  • On issues with both outcomes (208 failing, 214 passing runs) the three highest-scoring models reach J +9% to +11%, each with an interval of about +/-9 points; on the pure issues they reach +25% to +27%.
  • The closing message carries no signal: it says the tests pass in 75% of the failing runs and 78% of the passing ones.
  • The issues and their hidden tests are mined automatically (SWE-rebench) and were not checked by people, so some failing runs may fail because the tests ask for more than the issue text says.

False 'done' claims on the leaderboard → How this set was built → Threats to validity → Build a supervision kit →

How to detect it

Signals that point to incomplete execution, from the SAICA knowledge graph:

  • Grep for TODO, FIXME, XXX, NotImplementedError, or bare pass/... in the generated diff.
  • Tests for features the user explicitly named don't exist.
  • Agent's final message claims a subtask is done but the file contains only a stub.
  • Function docstrings describe a behavior not implemented by the body.

Studies and reported patterns

Records in the catalog that are not single dated incidents: findings of a study, recurring behaviour that was reported, or scenarios a source describes as hypothetical.

Tools in the catalog that address it

7 tools are tagged with incomplete execution in the catalog, most-starred first. A tag is the catalog's claim about what a tool is for; it is not a test result. Where SAICA-Bench ran the tool on this mode, the measured J is shown: that row is the evidence.

  • Ruff measured J +0.5% Extremely fast Python linter and formatter, written in Rust.
  • ESLint Pluggable JavaScript/TypeScript linter and code-quality enforcer.
  • opik measured J +3.6% Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
  • DeepEval measured J +0.8% The open-source LLM evaluation framework for testing and evaluating LLM applications across 20+ custom and researched metrics.
  • pr-agent measured J +14.7% AI-powered tool for automated pull-request analysis, feedback, and suggestions.
  • Arize Phoenix measured J +6.6% Open-source LLM tracing and evaluation platform built on OpenTelemetry.
  • TruLens Evaluation and Tracking for LLM Experiments and AI Agents

In other taxonomies

How incomplete execution maps to other failure taxonomies. “Exact” means the same failure, “partial” an overlap, “related” a neighbouring category.

  • MAST: Multi-Agent System Failure Taxonomy

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    • partial FM-3.1 Premature termination — Ending a task before its objectives are met overlaps with incomplete_execution (stopping before all user-specified items are addressed). Not exact: the SAICA mode is defined by a false "done" claim, which MAST does not require, and FM-3.1 also covers early stops with no such claim. Was "exact" before the fix round of 2026-11-21.

    Related: FM-3.2 No or incomplete verification

  • 9 Critical Failure Patterns of Coding Agents

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    Related: DAP-09 Exception & Error Handling

  • Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes

    Mapping drafted by an AI model from the source text; not yet reviewed by a person.

    Related: SHAH-CAT-02 Agent Lifecycle and State; SHAH-CAT-12 Failure Handling & Implementation Robustness

All 11 AI coding agent failure modes → · Glossary

Research

Questions and answers

What is incomplete execution in AI coding agents?

Incomplete execution, also called premature or false completion, is when an AI coding agent reports a task as done while parts of it are skipped, stubbed out or silently dropped. Common forms are TODO comments left in the code, function bodies replaced with pass or ..., a feature only partly implemented, error paths skipped, or the agent stopping before every item the user asked for is addressed. It is the opposite of scope creep: the agent under-delivers inside the request instead of reaching beyond it.

How do you detect false “done” claims?

Signals that point to it: Grep for TODO, FIXME, XXX, NotImplementedError, or bare pass/... in the generated diff. Tests for features the user explicitly named don't exist. Agent's final message claims a subtask is done but the file contains only a stub. Function docstrings describe a behavior not implemented by the body. On SAICA-Bench: No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named.

Which reviewers catch false “done” claims on SAICA-Bench, and how well?

No reviewer reaches J 25% on this set: the highest J of a row with enough readable answers is +19.5% (95% interval +13.9% to +24.9%), and 3 other rows cannot be told apart from it. No best reviewer is named. A rule with no understanding of the failure scores J +61.3% on the same set (Issue prior), so read the scores against that, not against 0. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

How was detection of false “done” claims measured?

On 1,200 scored samples: 600 runs declared done whose hidden tests fail and 600 clean ones. Where the bad samples come from: real agent output (Qwen3-Coder-480B in OpenHands, working on real GitHub issues (SWE-rebench)). Label: execution: the issue's hidden tests fail after the agent declared the task done. Data: SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): Qwen3-Coder-480B in OpenHands on real GitHub issues. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Benchmark results from 2026-11-22. The definition, detection signals and tags come from the SAICA knowledge graph.

View raw YAML on GitHub →