Skip to main content

Supervision tools for AI coding agents

49 tools, most of them open source, that can supervise an AI coding agent: linters and type checkers, AI code reviewers, LLM eval frameworks, sandboxes and guardrails. Each is classified by control paradigm (prevent, detect, correct or recover), when it acts (before, during or after generation), how much human review it needs and how its check is carried out (a program, a model, or both), and tagged with the failure modes it addresses. A tag is the catalog's claim, not a test result: 11 of the tools have rows on the SAICA-Bench leaderboard, and their pages show those rows with the configuration that was run. Filters and sorting are keyboard-accessible and client-side.

What was measured, per failure mode

For each measured failure mode: the best reviewer of any kind on the SAICA-Bench leaderboard, and, as a separate thing, the highest-scoring tool of this catalog that ran a check of its own. The sets are grouped by where their bad samples come from, because scores are not comparable across groups; the findings report explains what the numbers mean. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

Real agent output

The bad samples are what an agent produced on its own; the label comes from running hidden tests.

What the set scores Best reviewer Highest J Best catalog tool, own check Shortcut baseline All rows
False 'done' claims failure mode: Incomplete execution J under 25% on our set no reviewer reaches J 25% +19.5% (95% interval +13.9% to +24.9%) none reaches J 25% +61.3% Issue prior All rows →
Security vulnerability J under 25% on our set no reviewer reaches J 25% +7.9% (95% interval 0.0% to +15.9%) none reaches J 25% +48.5% Task prior All rows →

Model-written code, not agent output

The bad samples are what a code model wrote for a textbook task; the label comes from running the task's tests.

What the set scores Best reviewer Highest J Best catalog tool, own check Shortcut baseline All rows
Logic error J 25% to under 80% on our set mimo-v2.6-flash +74.3% (95% interval +69.4% to +78.9%) PR-Agent +53.2% +39.4% Surface classifier All rows →

Failure planted on purpose

The bad samples were written, injected or swapped in on purpose, by a model, a script or by hand.

What the set scores Best reviewer Highest J Best catalog tool, own check Shortcut baseline All rows
Dependency blindness J 80% or more on our set top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash +90.0% (95% interval +82.0% to +98.0%) none ran a check of its own +38.0% No name from the shown file appears in the added lines All rows →
Fabrication J 80% or more on our set top group of 4: deepseek-v4.1-flash, glm-5.3 and 2 more +94.3% (95% interval +91.6% to +96.7%) pyright +85.3% +59.7% Context-only rule All rows →
Obsolescence J 80% or more on our set top group of 3: gpt-6-luna, mimo-v2.6-flash and deepseek-v4-flash +89.7% (95% interval +82.9% to +95.1%) pyright (latest libraries) +56.8% +97.4% The code uses an API listed in the 2024 deprecation map All rows →
Scope creep J 80% or more on our set mimo-v2.6-flash +97.7% (95% interval +93.6% to +99.1%) none ran a check of its own +46.9% Two or more hunks in files that already existed All rows →
Supply chain J 25% to under 80% on our set top group of 2: gpt-6-luna and deepseek-v4-flash +77.0% (95% interval +70.3% to +83.8%) none ran a check of its own +100.0% An added package is in the OpenSSF malicious-packages list All rows →
Test edits that hide failures failure mode: Test manipulation J 80% or more on our set top group of 8: minimax-m3, giskard and 6 more +91.0% (95% interval +81.3% to +94.9%) none ran a check of its own +60.3% First user message mentions tests All rows →

Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate. SAICA-Bench results v2.0.0, built 22 November 2026.

All tools