Skip to main content

Code hallucination (fabrication) in AI coding agents

Code hallucination, which SAICA-Bench calls fabrication, is when an AI coding agent writes code that refers to a function, method, class or package that does not exist in any installed dependency, the standard library or the runtime. The code is syntactically plausible and factually false. It differs from obsolescence, where the API is real but deprecated or removed, and from dependency blindness, where the right function exists but the agent ignores it. A hallucinated package name is also the opening for slopsquatting.

At best a hallucinated import crashes at run time; at worst someone registers the made-up name and the agent installs their code. Spracklen et al. (USENIX Security 2025) found that on average 5.2% of the packages recommended by commercial models, and 21.7% of those recommended by open-source models, do not exist. One hallucinated name, huggingface-cli, drew between 15,000 and 30,000 downloads (the two sources differ) after a researcher registered it as an empty package.

At a glance: On SAICA-Bench: No single best reviewer: 4 rows cannot be told apart at 95% on the samples they share (deepseek-v4.1-flash, glm-5.3, deepseek-v4-flash and gpt-6-luna); the highest J is +94.3% (95% interval +91.6% to +96.7%). Results ↓

Also called
  • hallucination
  • API hallucination
  • package hallucination
  • confabulation
  • made-up reference
  • non-existent API
  • phantom import
  • imagined dependency

How well can code hallucination be caught?

J 80% or more on our set Code that calls functions, methods or packages that don't exist. No single best reviewer: 4 rows cannot be told apart at 95% on the samples they share (deepseek-v4.1-flash, glm-5.3, deepseek-v4-flash and gpt-6-luna); the highest J is +94.3% (95% interval +91.6% to +96.7%).

Best reviewer

any ranked row; a tie is shown as a top group

top group of 4: deepseek-v4.1-flash, glm-5.3, deepseek-v4-flash and gpt-6-luna

Highest J +94.3% (95% interval +91.6% to +96.7%)

Best catalog tool

a tool of the catalog running its own check

pyright

J +85.3% (95% interval +81.0% to +89.3%)

Shortcut baseline

a rule that understands nothing about the failure

Context-only rule

J +59.7%. Read the reviewers against this, not against 0.

What was tested

Bad samples
Failure planted on purpose. Claude Sonnet 4.5, asked to write fake completions (by the Delulu authors). Label: construction; each fake was verified to fail when run. Clean samples: the original code the fake replaces.
One sample
one completion (a few lines inside a source file).
Samples scored
348 fabricated completions and 288 clean (55% of the set is bad); 22 more are not scored.
Measure
J 80% or more on our set.
Shortcut baseline
A rule with no understanding of the failure scores J +59.7% here (Context-only rule). Read reviewer scores against it, not against 0.

Threats to validity that apply to this set →

All results on this set

Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

Fabrication: ranked reviewers, best first
# Reviewer 95% interval bad samples flagged clean samples flagged No verdictanswers left out False alarmsper 100 clean changes Precisionif 1% of changes are badPrecisionif 5% of changes are badPrecisionif 20% of changes are bad per 1,000 reviews
1 deepseek-v4.1-flash, asked directly
Model asked directly · in the top group
Run card
Model
deepseek-v4.1-flash
Provider
Hyperbolic
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
16384
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$1.11 per 1,000 reviews (api); $0.73 spent on this run. Charge reported by the API with each call (Hyperbolic)
+94.3%+91.6% to +96.7% 95.7%93.0% to 97.4% 1.4%0.5% to 3.5% 0 1.4 41%78%95% $1.11api
2 glm-5.3, asked directly
Model asked directly · in the top group
Run card
Model
glm-5.3
Provider
Hyperbolic
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
16384
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$6.23 per 1,000 reviews (api); $4.10 spent on this run. Charge reported by the API with each call (Hyperbolic)
+93.0%+89.5% to +95.9% 95.4%92.7% to 97.2% 2.4%1.2% to 4.9% 0 2.4 28%67%91% $6.23api
3 deepseek-v4-flash, asked directly
Model asked directly · in the top group
Run card
Model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
4096
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.38 per 1,000 reviews (api); $0.25 spent on this run. Charge reported by the API with each call (Hyperbolic)
+92.5%+88.8% to +95.6% 94.5%91.6% to 96.5% 2.1%1.0% to 4.5% 0 2.1 31%70%92% $0.38api
4 gpt-6-luna, asked directly
Model asked directly · in the top group
Run card
Model
gpt-6-luna
Provider
Azure AI Foundry
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Output token limit
16384
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.20 per 1,000 reviews (list); $0.13 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+92.2%+88.8% to +95.1% 94.3%91.3% to 96.2% 2.1%1.0% to 4.5% 0 2.1 31%70%92% $0.20list
5 mimo-v2.6-flash, asked directly
Model asked directly
Run card
Model
mimo-v2.6-flash
Provider
Hyperbolic
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
16384
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.13 per 1,000 reviews (api); $0.08 spent on this run. Charge reported by the API with each call (Hyperbolic)
+89.6%+85.9% to +93.0% 91.4%88.0% to 93.9% 1.7%0.7% to 4.0% 0 1.7 35%73%93% $0.13api
6 pyright
Catalog tool · Free program · own check
Run card
Version
pyright 1.1.414, python 3.12 env with the 99 packages the sample files import
What it was shown
changed file; the project's third-party packages installed; added line numbers
Counts as flagged when
Flagged when an error-severity diagnostic with one of the counted rules lands on a line of the completion (the whole file is checked).
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • only diagnostics that say a symbol does not exist are counted (4 rules, error severity)
  • only findings on the lines under review are counted
  • run with the file's third-party packages installed
+85.3%+81.0% to +89.3% 88.8%85.0% to 91.7% 3.5%1.9% to 6.3% 0 3.5 21%57%86% $0free
7 minimax-m3, asked directly
Model asked directly
Run card
Model
minimax-m3
Provider
Hyperbolic
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
16384
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.80 per 1,000 reviews (api); $0.53 spent on this run. Charge reported by the API with each call (Hyperbolic)
+83.8%+79.6% to +87.7% 85.9%81.9% to 89.2% 2.1%1.0% to 4.5% 0 2.1 29%68%91% $0.80api
8 gpt-oss-20b on one RTX 4090, asked directly
Model asked directly
Run card
Model
openai/gpt-oss-20b
Provider
Self-hosted GPU
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
16384
First run
2026-11-19
Runs
636; failed 2 (0.3%): empty 2
Samples
348 bad, 288 clean
Answers with no verdict
2 (1 bad, 1 clean): empty 2
Cost
~$0.14 per 1,000 reviews (gpu); $0.09 spent on this run. Estimated: 372 seconds of GPU time taken from the run log, not from per-call records. GPU time while requests ran, on a rented RTX 4090 at $0.89 per hour (setup and model loading not counted)
+80.6%+75.6% to +85.4% 94.2%91.3% to 96.2% 13.6%10.1% to 18.0% 2 of 636 13.6 7%27%63% ~$0.14gpu
9 Qwen3.8-27B (4-bit) on one RTX 4090, thinking off
Model asked directly
Run card
Model
qwen3.8-27b
Provider
Self-hosted GPU
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
16384
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
~$0.11 per 1,000 reviews (gpu); $0.07 spent on this run. Estimated: 287 seconds of GPU time taken from the run log, not from per-call records. GPU time while requests ran, on a rented RTX 4090 at $0.89 per hour (setup and model loading not counted)
+79.9%+75.2% to +84.4% 83.0%78.7% to 86.6% 3.1%1.7% to 5.8% 0 3.1 21%58%87% ~$0.11gpu
10 mypy --check-untyped-defs
Catalog tool · Free program · own check
Run card
Version
mypy 2.4.1
What it was shown
changed file; the project's third-party packages installed; added line numbers
Counts as flagged when
Flagged when an error with one of the counted codes lands on a line of the completion.
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • --check-untyped-defs switched on
  • only errors that say a symbol does not exist are counted (5 codes)
  • only findings on the lines under review are counted
  • run with the file's third-party packages installed
+77.3%+72.0% to +82.2% 78.7%74.1% to 82.7% 1.4%0.5% to 3.5% 0 1.4 36%75%93% $0free
11 mypy (default)
Catalog tool · Free program · own check
Run card
Version
mypy 2.4.1
What it was shown
changed file; the project's third-party packages installed; added line numbers
Counts as flagged when
Flagged when an error with one of the counted codes lands on a line of the completion.
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • only errors that say a symbol does not exist are counted (5 codes)
  • only findings on the lines under review are counted
  • run with the file's third-party packages installed
+72.5%+66.7% to +77.9% 73.9%69.0% to 78.2% 1.4%0.5% to 3.5% 0 1.4 35%74%93% $0free
12 grok-4.1-fast (no reasoning), asked directly
Model asked directly
Run card
Model
grok-4-1-fast-non-reasoning
Provider
Azure AI Foundry
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
8192
First run
2026-11-19
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
$0.14 per 1,000 reviews (list); $0.09 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+70.4%+64.6% to +75.8% 80.5%76.0% to 84.3% 10.1%7.1% to 14.1% 0 10.1 7%30%67% $0.14list
13 PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Run card
Version
pr-agent 0.50.0, litellm 1.105.1
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
git repo with a base and a change branch
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 1 (0.2%): tool error 1
Samples
348 bad, 288 clean
Answers with no verdict
1 (0 bad, 1 clean): tool error 1
Cost
~$0.23 per 1,000 reviews (api); $0.15 spent on this run. Estimated: the cost ledger holds 285 calls for 657 completed reviews ($0.07 logged; the rest ran before logging started), so the cost per review is taken from the logged calls. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
  • it sees the change only, not a task description
+68.9%+63.9% to +73.8% 75.9%71.1% to 80.1% 7.0%4.6% to 10.5% 1 of 636 7 10%36%73% ~$0.23api
14 ruff
Catalog tool · Free program · own check
Run card
Version
ruff 0.17.6
What it was shown
changed file; added line numbers
Counts as flagged when
Flagged when a finding with one of the counted codes lands on a line of the completion.
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • only F821, F822, F823 and F401 are counted
  • only findings on the lines under review are counted
+54.0%+47.3% to +60.2% 57.5%52.2% to 62.6% 3.5%1.9% to 6.3% 0 3.5 14%47%81% $0free
15 Phi-4-mini, asked directly
Model asked directly
Run card
Model
Phi-4-mini-instruct
Provider
Azure AI Foundry
What it was shown
unified diff of the changed file with 15 lines of context
Counts as flagged when
flagged when the answer carries "fabrication": true
Temperature
0
Output token limit
1024
First run
2026-11-19
Runs
636; failed 26 (4.1%): no verdict 26
Samples
348 bad, 288 clean
Answers with no verdict
26 (16 bad, 10 clean): no verdict 26
Cost
$0.05 per 1,000 reviews (list); $0.04 spent on this run. List price x tokens (the provider reports no per-call charge) (Azure AI Foundry)
+12.1%+6.0% to +18.4% 82.2%77.8% to 86.0% 70.1%64.5% to 75.2% 26 of 636 70.1 1%6%23% $0.05list

Shown for reference, not ranked

These rows are never counted as best. A shortcut baseline shows what a rule with no understanding of the failure scores on this set; read the reviewers' scores against it.

Fabrication: rows shown for reference and not ranked
Row and why it is not ranked J95% interval TPR FPR No verdict Costper 1,000 reviews
Context-only rule: the name appears nowhere else in the file
Shortcut baseline
Not ranked: shortcut baseline: shows what a rule with no understanding of the failure scores on this set.
No knowledge of any library. Import line: flags when an imported name is used nowhere else in the file. Other code: flags when a bare name (not a Python builtin) or an attribute name occurs nowhere else in the file.
+59.7%+53.6% to +65.1% 68.7% 9.0% 0 $0
How to read these numbers
  • J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
  • An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. j_strict is J under the old rule that counted such answers as 'not flagged'; j_bounds is J if every unreadable answer had been wrong, and right.
  • 95% intervals. TPR and FPR: Wilson. J: Newcombe where samples are independent; where samples are grouped (task, pair, API pair, file) whole groups are resampled (100,000 draws, fixed seed).
  • Best reviewer: the best ranked row of a mode that is not marked incomplete. A row with under 95% readable answers in one class is listed and ranked, with its J bounds, but is never named best reviewer or sole winner; if its J is higher than the best reviewer's it stays in the top group. Not ranked: baseline rules, shortcut baselines and rows outside a tool's design scope. Top group: the best reviewer and every ranked row whose paired difference from it includes 0. Best catalog tool: the best tool of the SAICA catalog that ran its own check; a framework that ran our rule text is not a candidate.
  • A row's cost covers every call made to get its scored answers: the current answer of each sample plus earlier attempts that failed or were cut off and had to be re-run. Smoke tests, runs on earlier versions of a set and accidental duplicate answers are not included. Cost per 1,000 reviews is that cost divided by the reviews made, for one mode-specific question per review. 'Cheap' means at most $5.00 per 1,000 reviews.
  • Precision is the share of flagged changes that are really bad, if the stated share of all changes were bad. It is computed from the row's TPR and FPR; the set's own share of bad samples is 55%.

The same answers, scored another way

Parts of the set, or the same set under another labelling. They show how far the headline J depends on how the set was built.

Fake import · 164 bad, 164 clean

Fake imports: a module or name that does not exist is imported. Telling them from real imports needs knowledge of the libraries. Negatives: the clean originals these fakes were made from.

RowJ95% intervalTPRFPR
deepseek-v4.1-flash+90.8%+86.1% to +95.2%91.5%0.6%
glm-5.3+87.8%+82.1% to +93.0%90.2%2.4%
deepseek-v4-flash+87.2%+81.4% to +92.5%88.4%1.2%
gpt-6-luna+87.2%+81.3% to +92.5%89.0%1.8%
mimo-v2.6-flash+84.8%+78.7% to +90.3%85.4%0.6%
gpt-oss-20b (self-hosted)+81.1%+74.6% to +87.2%89.0%7.9%
pyright+78.0%+71.1% to +84.7%81.7%3.7%
PR-Agent+73.8%+66.5% to +80.8%74.4%0.6%
minimax-m3+72.0%+64.9% to +78.8%74.4%2.4%
Qwen3.8-27B (self-hosted)+67.1%+59.3% to +74.7%70.7%3.7%
mypy (untyped defs)+64.0%+56.2% to +71.7%65.2%1.2%
mypy+63.4%+55.6% to +71.1%64.6%1.2%
grok-4.1-fast+57.3%+49.4% to +65.2%66.5%9.2%
ruff+29.9%+22.6% to +37.4%34.8%4.9%
Phi-4-mini+12.7%+4.3% to +21.2%83.5%70.9%
Fake method · 77 bad, 77 clean

Fake methods and functions: a call to something that does not exist. Negatives: the clean originals these fakes were made from.

RowJ95% intervalTPRFPR
deepseek-v4-flash+97.4%+93.2% to +100.0%100.0%2.6%
glm-5.3+97.4%+93.2% to +100.0%100.0%2.6%
gpt-6-luna+96.1%+91.0% to +100.0%98.7%2.6%
deepseek-v4.1-flash+94.8%+89.2% to +98.8%98.7%3.9%
minimax-m3+92.2%+85.3% to +97.5%94.8%2.6%
mimo-v2.6-flash+90.9%+83.6% to +96.5%93.5%2.6%
pyright+89.6%+81.1% to +96.3%93.5%3.9%
Qwen3.8-27B (self-hosted)+89.6%+81.9% to +96.1%92.2%2.6%
mypy (untyped defs)+84.4%+75.0% to +92.5%85.7%1.3%
mypy+75.3%+64.4% to +85.1%76.6%1.3%
gpt-oss-20b (self-hosted)+75.0%+63.6% to +85.3%98.7%23.7%
grok-4.1-fast+74.0%+62.3% to +84.3%84.4%10.4%
PR-Agent+71.4%+60.0% to +81.9%92.2%20.8%
ruff+58.4%+45.7% to +70.2%59.7%1.3%
Phi-4-mini+27.8%+14.8% to +41.0%93.2%65.3%
Undefined name · 107 bad, 107 clean

Undefined names: a variable that is defined nowhere. Checking this needs the file, not knowledge of any library. Negatives: the clean originals these fakes were made from.

RowJ95% intervalTPRFPR
glm-5.3+99.1%+96.9% to +100.0%100.0%0.9%
deepseek-v4.1-flash+99.1%+96.9% to +100.0%100.0%0.9%
deepseek-v4-flash+98.1%+95.2% to +100.0%100.0%1.9%
minimax-m3+97.2%+93.5% to +100.0%97.2%0.0%
mimo-v2.6-flash+97.2%+93.6% to +100.0%99.1%1.9%
gpt-6-luna+97.2%+93.7% to +100.0%99.1%1.9%
pyright+95.3%+90.8% to +99.1%96.3%0.9%
mypy (untyped defs)+93.5%+88.2% to +97.6%94.4%0.9%
Qwen3.8-27B (self-hosted)+93.5%+87.5% to +98.2%95.3%1.9%
ruff+89.7%+83.2% to +95.1%90.6%0.9%
grok-4.1-fast+89.7%+83.3% to +95.1%99.1%9.3%
mypy+85.0%+76.4% to +92.5%86.0%0.9%
gpt-oss-20b (self-hosted)+79.4%+69.9% to +87.8%99.1%19.6%
PR-Agent+53.1%+42.6% to +63.2%66.4%13.2%
Phi-4-mini+3.4%−6.8% to +14.2%72.3%68.9%
Noisy rows dropped · 338 bad, 283 clean

The scored set without the 15 noisy rows listed in the table noisy_rows.

RowJ95% intervalTPRFPR
deepseek-v4.1-flash+96.3%+94.2% to +98.2%97.0%0.7%
deepseek-v4-flash+94.8%+91.5% to +97.4%95.9%1.1%
glm-5.3+94.7%+91.4% to +97.3%96.5%1.8%
gpt-6-luna+93.5%+90.3% to +96.2%95.3%1.8%
mimo-v2.6-flash+91.9%+88.6% to +94.8%92.6%0.7%
pyright+87.5%+83.3% to +91.3%89.6%2.1%
minimax-m3+85.9%+81.8% to +89.6%87.0%1.1%
gpt-oss-20b (self-hosted)+82.9%+77.8% to +87.6%95.0%12.1%
Qwen3.8-27B (self-hosted)+81.8%+77.1% to +86.2%84.3%2.5%
mypy (untyped defs)+79.5%+74.2% to +84.2%80.2%0.7%
mypy+74.4%+68.6% to +79.8%75.1%0.7%
grok-4.1-fast+71.9%+66.1% to +77.3%81.1%9.2%
PR-Agent+71.0%+66.1% to +75.8%76.3%5.3%
ruff+54.3%+47.5% to +60.5%57.1%2.8%
Phi-4-mini+12.4%+6.3% to +18.8%82.0%69.6%
Duplicates dropped · 308 bad, 222 clean

One row per distinct completion text: the first uid of each (label, completion) group is kept.

RowJ95% intervalTPRFPR
deepseek-v4.1-flash+95.3%+92.5% to +97.6%97.1%1.8%
glm-5.3+93.0%+88.9% to +96.3%96.1%3.1%
deepseek-v4-flash+92.1%+87.9% to +95.6%94.8%2.7%
gpt-6-luna+92.1%+88.4% to +95.3%94.8%2.7%
mimo-v2.6-flash+89.6%+85.7% to +93.2%91.9%2.3%
pyright+85.9%+81.4% to +90.0%89.9%4.0%
minimax-m3+85.0%+80.4% to +89.0%87.7%2.7%
Qwen3.8-27B (self-hosted)+81.7%+76.7% to +86.3%85.7%4.0%
gpt-oss-20b (self-hosted)+79.7%+73.7% to +85.3%95.1%15.4%
mypy (untyped defs)+78.4%+73.0% to +83.3%80.2%1.8%
mypy+73.2%+67.1% to +78.9%75.0%1.8%
grok-4.1-fast+72.1%+66.1% to +77.8%84.7%12.6%
PR-Agent+66.2%+60.6% to +71.7%74.4%8.1%
ruff+57.3%+50.3% to +63.7%61.4%4.0%
Phi-4-mini+11.5%+4.6% to +18.5%81.7%70.2%
Noisy and duplicates dropped · 301 bad, 218 clean

Both of the above: noisy rows left out, then one row per distinct completion.

RowJ95% intervalTPRFPR
deepseek-v4.1-flash+96.8%+94.5% to +98.7%97.7%0.9%
glm-5.3+94.0%+90.2% to +97.1%96.4%2.3%
deepseek-v4-flash+94.0%+90.1% to +97.0%95.3%1.4%
gpt-6-luna+92.7%+89.1% to +95.9%95.0%2.3%
mimo-v2.6-flash+91.4%+87.9% to +94.6%92.4%0.9%
pyright+87.3%+82.9% to +91.3%90.0%2.8%
minimax-m3+86.7%+82.3% to +90.6%88.0%1.4%
Qwen3.8-27B (self-hosted)+83.2%+78.2% to +87.7%86.4%3.2%
gpt-oss-20b (self-hosted)+81.5%+75.5% to +87.1%95.3%13.8%
mypy (untyped defs)+80.2%+74.8% to +84.9%81.1%0.9%
mypy+74.8%+68.7% to +80.4%75.8%0.9%
grok-4.1-fast+72.8%+66.7% to +78.5%84.7%11.9%
PR-Agent+67.6%+62.1% to +73.1%74.1%6.5%
ruff+58.3%+51.2% to +64.6%61.5%3.2%
Phi-4-mini+12.0%+5.3% to +19.1%81.7%69.7%
More tables for this set in results.json
  • noisy_rows: Left out of the noisy_rows_dropped stratum; still in the headline set.
  • doubtful_rows: Listed, not dropped: the label is arguable.
  • duplicates
  • ruff_unused_import: ruff's counted codes include F401 (imported but unused). A fake that renames an import leaves the rest of the human-written file using the old name, so F401 fires; code an agent writes itself would use the made-up name throughout.
  • j_change_when_dropped: J in the stratum minus the headline J of the same row, in J points (0.01 = one point).
  • delulu_error_by_type

They are under modes[].tables of results.json.

How it was tested

  • 348 fabricated completions (fake imports, fake methods, undefined names), written by Claude Sonnet 4.5 and verified to fail; 288 clean originals of the same code. Another 22 'wrong parameter' completions were reviewed too (658 reviews per tool, the basis of the cost) but are not scored.
  • Static tools see the whole file with its 99 packages installed. AI tools see a diff with 15 lines of context.
  • Static tools count only findings that a symbol doesn't exist. A flag on a clean original is a false positive. The list of finding codes was chosen after a first run on these samples.
  • Temperature 0 (gpt-6-luna: its default, no temperature sent), one run each. Models on Hyperbolic, except gpt-6-luna, grok-4.1-fast and Phi-4-mini on Azure AI Foundry, and gpt-oss-20b and Qwen3.8-27B self-hosted with vLLM on one RTX 4090. Run 2026-11-19.

Caveats

  • The fakes were written on purpose by a model; real agent mistakes may be easier or harder.
  • Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.
  • Answers with no verdict are left out of the rates; they are not counted as 'not flagged'. Unreadable: Phi-4-mini 26 of 636 (16 bad, 10 clean; mostly answers without a verdict); gpt-oss-20b (self-hosted) 2 of 636 (1 bad, 1 clean; mostly empty answers); PR-Agent 1 of 636 (0 bad, 1 clean; mostly the tool failed to run).
  • The reviewers did not see the same input: models and PR-Agent get a diff with 15 lines of context, the static tools get the whole file with its packages installed. Rows of the two kinds are not a like-for-like comparison.
  • A rule with no knowledge of any library (does the name appear anywhere else in the file?) scores J +60%. It catches 98% of the undefined names, 99% of the fake methods and 35% of the fake imports: the fakes are one-token changes spliced into human code.
  • 144 of the 348 fakes fail in Delulu's own check with a plain 'name is not defined': a scope error, not a made-up library API. Fake imports are the part that needs library knowledge; there the models catch 66% to 91%.
  • 15 scored rows are noisy (a 'fake' import that did not fail, clean originals that are broken files) and the 348 fakes hold 308 distinct completions. Leaving both out changes a J by -1.3 to +4.2 points, while the four highest-scoring models are only 2.1 points apart.
  • The highest J has a 95% interval of +92% to +97% and 3 other rows have a J inside it: the models at the top are tied.
  • The list of finding codes that count as 'symbol does not exist' for ruff, mypy and pyright was fixed after a first full run on these same samples. For ruff it includes 'imported but unused', which alone gives 57 of its catches: the rest of the human-written file still uses the old name.

Fabrication on the leaderboard → How this set was built → Threats to validity → Build a supervision kit →

How to detect it

Signals that point to fabrication, from the SAICA knowledge graph:

  • Agent emits an import or function call whose identifier is absent from all declared dependencies and stdlib.
  • Static analyzer (e.g., pyflakes, tsc) flags the reference as undefined at pre-commit.
  • Execution-grounded check (sandboxed run) raises ImportError, AttributeError, or analogous resolution failure.
  • LSP-based symbol lookup returns no candidate.

Real-world incidents

Studies and reported patterns

Records in the catalog that are not single dated incidents: findings of a study, recurring behaviour that was reported, or scenarios a source describes as hypothetical.

Tools in the catalog that address it

9 tools are tagged with fabrication in the catalog, most-starred first. A tag is the catalog's claim about what a tool is for; it is not a test result. Where SAICA-Bench ran the tool on this mode, the measured J is shown: that row is the evidence.

  • Context7 Up-to-date, version-pinned library documentation for coding agents — delivered as an MCP server.
  • Ruff measured J +54.0% Extremely fast Python linter and formatter, written in Rust.
  • ESLint Pluggable JavaScript/TypeScript linter and code-quality enforcer.
  • mypy measured J +77.3% Static type checker for Python. Catches mismatched types before runtime.
  • Pyright measured J +85.3% Microsoft's fast static type checker for Python.
  • Context Hub Up-to-date, curated API docs for coding agents.
  • garak The LLM vulnerability scanner.
  • Guardrails AI Output validation framework with a catalog of validators and reask-on-failure loops.
  • llm-guard The Security Toolkit for LLM Interactions.

In other taxonomies

How fabrication maps to other failure taxonomies. “Exact” means the same failure, “partial” an overlap, “related” a neighbouring category.

  • Deprecated API usage (obsolescence) — Real APIs that are deprecated, removed or changed, often from stale training data.
  • Reinventing existing code (dependency blindness) — Hand-written code for what the standard library, a dependency or the codebase already provides.
  • Slopsquatting (supply-chain attack) — The agent installs malicious code: a squatted hallucinated name, a lookalike or a compromised tool.

All 11 AI coding agent failure modes → · Glossary

Research

Questions and answers

What is code hallucination in AI coding agents?

Code hallucination, which SAICA-Bench calls fabrication, is when an AI coding agent writes code that refers to a function, method, class or package that does not exist in any installed dependency, the standard library or the runtime. The code is syntactically plausible and factually false. It differs from obsolescence, where the API is real but deprecated or removed, and from dependency blindness, where the right function exists but the agent ignores it. A hallucinated package name is also the opening for slopsquatting.

How do you detect code hallucination?

Signals that point to it: Agent emits an import or function call whose identifier is absent from all declared dependencies and stdlib. Static analyzer (e.g., pyflakes, tsc) flags the reference as undefined at pre-commit. Execution-grounded check (sandboxed run) raises ImportError, AttributeError, or analogous resolution failure. LSP-based symbol lookup returns no candidate. On SAICA-Bench: No single best reviewer: 4 rows cannot be told apart at 95% on the samples they share (deepseek-v4.1-flash, glm-5.3, deepseek-v4-flash and gpt-6-luna); the highest J is +94.3% (95% interval +91.6% to +96.7%).

Which reviewers catch code hallucination on SAICA-Bench, and how well?

No single best reviewer: 4 rows cannot be told apart at 95% on the samples they share (deepseek-v4.1-flash, glm-5.3, deepseek-v4-flash and gpt-6-luna); the highest J is +94.3% (95% interval +91.6% to +96.7%). The best catalog tool that ran its own check is pyright, at J +85.3%. A rule with no understanding of the failure scores J +59.7% on the same set (Context-only rule), so read the scores against that, not against 0. Each model asked directly was told which failure to look for and was tested on samples that have that failure or none. Its score is a single-prompt, mode-primed estimate, not a product.

How was detection of code hallucination measured?

On 636 scored samples: 348 fabricated completions and 288 clean ones. Where the bad samples come from: failure planted on purpose (Claude Sonnet 4.5, asked to write fake completions (by the Delulu authors)). Label: construction; each fake was verified to fail when run. Data: Delulu (microsoft/delulu), Python rows. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Benchmark results from 2026-11-22. The definition, detection signals and tags come from the SAICA knowledge graph.

View raw YAML on GitHub →