About
SAICA-Bench measures what reviewers flag when they are shown the work of AI coding agents, on 9 failure modes: fabricated code, test manipulation, obsolescence, scope creep and more. It runs free programs (type checkers, linters), packaged AI tools and plain language models on labelled samples and reports what each flagged, with a 95% interval and the cost. It is built by Vasyl Rakivnenko (Stanford University).
Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product. 6 of the 9 sets have bad samples that were planted on purpose and 2 are real agent output; the benchmark page says how each set was made and lists the threats to validity, and the findings report says what the results do and do not show.
The failure-mode taxonomy and the tool catalog come from our knowledge graph of supervision tools (graph). A failure-mode tag on a tool is the catalog's claim about what the tool is built for; a benchmark row is a measurement of one configuration on one set. A tool that was benchmarked but is not in the catalog is marked as such. Code and results are on GitHub; these pages show SAICA-Bench results v2.0.0, built 22 November 2026.
Data sources
The samples come from SWE-rebench-OpenHands-Trajectories (Nebius) (public since 2025-09), SusVibes (LeiLiLab) (public since 2025-12), Collu-Bench (lt-asset) (public since 2024-10), Delulu (Microsoft) (public since 2026-03), Wang et al. deprecated-API map (ICSE'25) (public since 2024-06), OpenSSF malicious-packages (public since 2023-03) and Patronus TRACE (public since 2026-01). Per failure mode:
- False 'done' claims (Incomplete execution): SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): Qwen3-Coder-480B in OpenHands on real GitHub issues. Bad samples: Qwen3-Coder-480B in OpenHands, working on real GitHub issues (SWE-rebench).
- Security vulnerability: SusVibes (LeiLiLab, MIT): AI agents' patches for 166 real feature tasks, with hidden CVE tests. Bad samples: 26 agent runs (Claude, Gemini, GLM, Kimi, Qwen with SWE-agent, OpenHands and vendor CLIs), collected by the SusVibes authors.
- Logic error: Collu-Bench (lt-asset, BSD-3-Clause-Clear): MBPP and HumanEval functions written by 11 code models. Bad samples: 11 code models of 2023-24 (CodeLlama, DeepSeek-Coder, StarCoder2, Llama 3, gpt-4o-mini) answering MBPP and HumanEval tasks; not agent runs.
- Dependency blindness: Hand-built on SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): correct agent patches, one call re-implemented by hand. Bad samples: written by hand by Claude (the coding assistant building this benchmark): one call replaced by a re-implementation.
- Fabrication: Delulu (microsoft/delulu), Python rows. Bad samples: Claude Sonnet 4.5, asked to write fake completions (by the Delulu authors).
- Obsolescence: Wang et al. (ICSE'25) deprecated-API map (65 pairs, 8 libraries) + solutions written by 5 models, deprecated call swapped in. Bad samples: an automatic code edit that swaps the deprecated call into solutions written by five models; four LLM-SDK pairs written by hand.
- Scope creep: Synthetic, built on SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0): correct agent patches + one change injected by Claude Opus. Bad samples: claude-opus-4-8, asked to inject one unrequested change into a correct agent patch.
- Supply chain: Synthetic, hand-designed on SWE-rebench-OpenHands-Trajectories (Nebius, CC-BY-4.0) + OpenSSF malicious-packages names. Bad samples: hand-designed by Claude (the coding assistant building this benchmark): one dependency line replaced with a known-malicious package name.
- Test edits that hide failures (Test manipulation): Patronus TRACE (PatronusAI/trace-dataset). Bad samples: Claude Opus 4.5, told to write the sessions (by the TRACE authors).
Code is Apache-2.0 and the results data CC-BY-4.0; the source datasets keep their own licenses.
Contact
Questions, corrections or a tool to add: open an issue or email Vasyl Rakivnenko at vasyl@stanford.edu. You can also find him on LinkedIn.