Skip to main content

Leaderboard

What do reviewers flag in the work of AI coding agents?

J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

The setting: Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

Real agent output: Model-written code, not agent output: Failure planted on purpose: Not benchmarked:

On the 2 sets of real agent output (Security vulnerability and False 'done' claims; 2,902 samples) no reviewer reaches J 25%. Best J: +8% on Security vulnerability and +20% on False 'done' claims; 5 of the 23 ranked rows there have a 95% interval for J that excludes 0. On the 6 sets where the failure was planted on purpose the best reviewer reaches J +77% to +98%; on Logic error (model-written code that is not agent output) +74%. Every set of real agent output scores below every planted set, so this benchmark cannot tell 'reviewers catch these failures' from 'reviewers catch planted failures'. Whether the high scores carry over to failures agents make on their own is not tested.

Reviewer by failure mode, one block per kind of set

J for each reviewer that ran on several failure modes, with its 95% interval. The blocks are not comparable with each other and no average over modes is shown: where the bad samples of a set come from explains most of the difference between sets. A mode's name opens its tab.

Real agent output

The bad samples are what an agent produced on its own; the label comes from running hidden tests.

Reviewer False 'done' claims Incomplete execution Security vulnerability
gpt-6-luna +20% +14% to +25% +5% −6% to +16%
deepseek-v4-flash +20% +14% to +26% +3% −8% to +13%
mimo-v2.6-flash +20% +14% to +25% +4% −4% to +12%
grok-4.1-fast +3% 0% to +5% −3% −13% to +6%
Phi-4-mini 0% −1% to +1% not run
ruff +1% 0% to +1% not run
PR-Agent +15% +3% to +25% 0% −7% to +7%
promptfoo not run +2% −7% to +11%
giskard not run +4% −7% to +15%
Best reviewer no reviewer reaches J 25%no reviewer reaches J 25%
For reference, never counted as best
Best simple rule +3% −1% to +6% not ranked not run
Shortcut baseline +61% +57% to +66% not ranked +49% +36% to +59% not ranked

Model-written code, not agent output

The bad samples are what a code model wrote for a textbook task; the label comes from running the task's tests.

Reviewer Logic error
gpt-6-luna +68% +63% to +73%
deepseek-v4-flash +69% +64% to +74%
mimo-v2.6-flash +74% +69% to +79%
grok-4.1-fast +12% +9% to +15%
Phi-4-mini +22% +17% to +27%
pyright 0% −1% to +1% not ranked
mypy −1% −2% to +1% not ranked
ruff −3% −5% to −1% not ranked
PR-Agent +53% +48% to +58%
Best reviewer mimo-v2.6-flash
For reference, never counted as best
Shortcut baseline +39% +35% to +44% not ranked

Failure planted on purpose

The bad samples were written, injected or swapped in on purpose, by a model, a script or by hand.

Reviewer Dependency blindness Fabrication Obsolescence Scope creep Supply chain Test edits that hide failures Test manipulation
gpt-6-luna +90% +82% to +98% +92% +89% to +95% +90% +83% to +95% +91% +86% to +95% +77% +70% to +84% +79% +70% to +84%
deepseek-v4-flash +86% +76% to +94% +92% +89% to +96% +88% +82% to +94% +93% +87% to +96% +72% +65% to +79% +89% +79% to +93%
mimo-v2.6-flash +86% +76% to +94% +90% +86% to +93% +88% +80% to +95% +98% +94% to +99% +65% +57% to +73% +89% +78% to +94%
grok-4.1-fast +64% +50% to +76% +70% +65% to +76% +53% +41% to +64% +53% +44% to +61% +18% +9% to +26% +86% +75% to +91%
Phi-4-mini −6% −24% to +11% +12% +6% to +18% +14% +6% to +24% −4% −14% to +7% 0% −3% to +3% +35% +23% to +49%
pyright not run +85% +81% to +89% +57% +42% to +71% not runnot runnot run
mypy not run +77% +72% to +82% +14% +7% to +23% not runnot runnot run
ruff not run +54% +47% to +60% +8% 0% to +17% not runnot runnot run
PR-Agent not run +69% +64% to +74% not runnot runnot runnot run
promptfoo not runnot runnot runnot runnot run +86% +76% to +91%
giskard not runnot runnot runnot runnot run +89% +80% to +93%
Best reviewer top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flashtop group of 4: deepseek-v4.1-flash, glm-5.3 and 2 moretop group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flashmimo-v2.6-flashtop group of 2: gpt-6-luna and deepseek-v4-flashtop group of 8: minimax-m3, giskard and 6 more
For reference, never counted as best
Best simple rule +12% +4% to +22% not ranked not runnot run 0% −9% to +9% not ranked +84% +78% to +89% not ranked +44% +33% to +52% not ranked
Shortcut baseline +38% +24% to +52% not ranked +60% +54% to +65% not ranked +97% +94% to +99% not ranked +47% +39% to +54% not ranked +100% +96% to +100% not ranked +60% +49% to +67% not ranked

AI model (baseline)Free programAI-basedHybridSimple rule (baseline) Each cell: J and its 95% interval · darker = higher J · “·” = not run

If the prompt of every mode ran on every change

Each cell above is the answer to one question about one failure. In use every question would run on every change. This table estimates what that does to false alarms; it was not measured.

estimate under independence, not measured Share of clean changes that get at least one flag when the prompt of every mode is run on every change.

Model Prompts run on each change Clean changes with at least one flag Expected false alarms per clean change Cost per 1,000 changes, all prompts Only the prompts of modes above the floor
gpt-6-luna 9 89% 1.8 $4.74 61% with 7 prompts
deepseek-v4-flash 9 87% 1.67 $10.39 55% with 7 prompts
mimo-v2.6-flash 9 69% 1.05 $8.78 49% with 7 prompts
grok-4.1-fast 9 100% 4.21 $5.47 99% with 7 prompts

The prompts' false alarms are taken as independent, and each prompt's false-alarm rate comes from its own mode's clean samples. No run with all prompts on the same changes has been made.

What the results say

  • On the 2 sets of real agent output (Security vulnerability and False 'done' claims; 2,902 samples) no reviewer reaches J 25%. Best J: +8% on Security vulnerability and +20% on False 'done' claims; 5 of the 23 ranked rows there have a 95% interval for J that excludes 0. On the 6 sets where the failure was planted on purpose the best reviewer reaches J +77% to +98%; on Logic error (model-written code that is not agent output) +74%. Every set of real agent output scores below every planted set, so this benchmark cannot tell 'reviewers catch these failures' from 'reviewers catch planted failures'. Whether the high scores carry over to failures agents make on their own is not tested.
  • 6 of the 9 sets have constructed positives (Fabrication, Test edits that hide failures, Scope creep, Obsolescence, Supply chain and Dependency blindness). 3 use naturally produced output, 2 of them real agent output. A model-free shortcut baseline is published for 9 of the 9 sets.
  • A single best reviewer can be named in 2 of 9 modes. Logic error: mimo-v2.6-flash (J +74%); Scope creep: mimo-v2.6-flash (J +98%). In the others the leading rows cannot be told apart at 95% on the samples they share: Fabrication (4 rows, J +92% to +94%); Test edits that hide failures (8 rows, J +86% to +91%); Security vulnerability (11 rows, J -0% to +8%); False 'done' claims (4 rows, J +15% to +20%); Obsolescence (3 rows, J +88% to +90%); Supply chain (2 rows, J +72% to +77%); Dependency blindness (3 rows, J +86% to +90%).
  • Where a catalog tool ran its own check (5 of 9 modes), a model asked the failure-specific question was clearly ahead of the best such tool in 3; in 2 no row reaches J 25%. Ahead: Fabrication (deepseek-v4.1-flash +94% vs pyright +85%, difference +9 points on shared samples, 95% interval +5 to +13); Logic error (mimo-v2.6-flash +74% vs PR-Agent +53%, difference +21 points on shared samples, 95% interval +17 to +25); Obsolescence (gpt-6-luna +90% vs pyright (latest libraries) +57%, difference +33 points on shared samples, 95% interval +17 to +49). No row at J 25%: Security vulnerability and False 'done' claims. No catalog tool with a check of its own for a finished change was run on Test edits that hide failures, Scope creep, Supply chain and Dependency blindness.
  • No packaged tool beat the same model asked the failure-specific question (12 comparisons on identical samples: 5 worse, 7 indistinguishable). Every tool ran with deepseek-v4-flash as its judge model, set by us. Worse: PR-Agent 0.50.0 on Fabrication (-24 points, 95% interval -28 to -19); uqlm 0.6.8 on Logic error (-16 points, 95% interval -23 to -10); PR-Agent 0.50.0 on Logic error (-16 points, 95% interval -20 to -11); Arize Phoenix 3.10.0 on False 'done' claims (-13 points, 95% interval -19 to -6); DeepEval 4.3.4 on False 'done' claims (-19 points, 95% interval -25 to -13). Each of them ran its own general-purpose check, not the failure-specific question. The frameworks that were given our rule text (giskard and promptfoo) were indistinguishable from the bare model in 4 of 4 comparisons.
  • Free programs reach J 50% or more where the failure is a checkable fact: pyright on Fabrication (+85%); pyright (latest libraries) on Obsolescence (+57%). No free catalog program within its design scope was run on Test edits that hide failures, Security vulnerability, Logic error, Scope creep, Supply chain and Dependency blindness.
  • Phi-4-mini scores J under 20% in 6 of the 8 modes it ran in and grok-4.1-fast flags more than 30% of clean samples in 6 of 9 modes. The other models that ran in at least 8 modes (gpt-6-luna, deepseek-v4-flash and mimo-v2.6-flash) score J under 20% in at most 2 modes and flag more than 30% of clean samples in at most 2.

Read before citing

  • J compares reviewers within one mode. Do not compare J across modes or average it over them: each set was built differently (real agent runs, model-written functions, injected or hand-built failures) and has its own sample unit, and where the bad samples come from explains most of the difference between modes.
  • Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. In use nobody knows which failure a change has, so the prompt of every mode would run on every change and their false alarms add up; the any-flag estimate next to the grid shows by how much. No run with all prompts on the same changes has been made.
  • Samples are grouped in 6 of 9 modes (Fabrication: 163 files; Security vulnerability: 166 tasks; Logic error: 379 tasks; Obsolescence: 69 API pairs; Supply chain: 148 pairs; Dependency blindness: 50 pairs). Intervals there resample whole groups (100,000 draws). A reviewer is named best only when its paired difference from the runner-up, on the samples both answered, has a 95% interval that excludes 0; otherwise the rows that cannot be told apart are listed as a top group. The best row is still a pick on the test set, so its J is a little optimistic.
  • Contamination was not assessed. Every set draws on public material (problem sets, GitHub issues and their fixes, CVE fixes, library release notes, package lists), so a model may have seen a task or its answer. Sources: Delulu (Microsoft) (public since 2026-03); Patronus TRACE (public since 2026-01); SusVibes (LeiLiLab) (public since 2025-12); Collu-Bench (lt-asset) (public since 2024-10); SWE-rebench-OpenHands-Trajectories (Nebius) (public since 2025-09); Wang et al. deprecated-API map (ICSE'25) (public since 2024-06); OpenSSF malicious-packages (public since 2023-03). The reviewers' training cut-off dates are not recorded here.
  • Rates are on answers that carry a verdict. 37 of 116 rows have unreadable answers (the count is beside each row, per label); 10 of them have fewer than 95% readable answers in one class and are marked incomplete.
  • Same prompt for every model within a mode, temperature 0 (gpt-6-luna: its default), one run each; run-to-run variation was not measured. Prompts and tool settings were fixed before scoring. mypy, ruff and the simple rules are shown in the best of the configurations we ran in each mode, which is a pick on the test set; pyright and mypy on Obsolescence ran with the latest libraries installed.
  • Baseline rules and shortcut baselines are shown for reference. They are never counted as best and never enter a finding. Rows outside a tool's design scope are listed apart and are not ranked.
  • A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. Checking a change for every failure costs the sum over the modes.
  • The findings above are computed from the rows, so they change with them.
  • A rule that needs no understanding of the failure reaches J 25% or more on 9 of 9 sets: Fabrication (+60%: context-only rule); Test edits that hide failures (+60%: first user message mentions tests); Security vulnerability (+49%: task prior); Logic error (+39%: surface classifier); False 'done' claims (+61%: issue prior); Scope creep (+47%: two or more hunks in files that already existed); Obsolescence (+97%: the code uses an API listed in the 2024 deprecation map); Supply chain (+100%: an added package is in the OpenSSF malicious-packages list); Dependency blindness (+38%: no name from the shown file appears in the added lines). High reviewer scores on those sets may partly reflect the same shortcut; J 0 is not the right zero there.

Threats to validity, with what was measured and what was not →

Free programAI-basedHybridAI model (baseline)Simple rule (baseline)

How each failure mode is tested → · Findings report → · Knowledge graph of supervision tools →