Skip to main content

Tools

Static type checker for Python. Catches mismatched types before runtime.

Detection Post-generation Fully autonomous Program Stable Activity band A Environment
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: A (raw score 7.23). Shown band: A .

Facets: documented_in=0evaluated_on=2integration_surfaces=1.5paper_citations=0recent_push=2stars=1.73

Source of truth: pipeline/maturity/score.py

Measured on SAICA-Bench in this configuration

Each row is one run of mypy, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good mypy is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Set Configuration and run card J (95% interval) TPR FPR No verdict Cost per 1,000 reviews
Logic error
Model-written code, not agent output
mypy (--check-untyped-defs --ignore-missing-imports)
Catalog tool · Free program · own check · outside design scope
Not ranked: outside the tool's design scope: a type checker looks for type errors, not for wrong results; shown as a floor.
Version
mypy 2.4.1
What it was shown
function source
Counts as flagged when
Flagged when the output has any ': error:' line.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Design scope
outside design scope: a type checker looks for type errors, not for wrong results; shown as a floor
Runs
2,440; failed 0 (0.0%)
Samples
1,536 bad, 904 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • --check-untyped-defs --ignore-missing-imports; any error line counts
−0.7%
−2.0% to +0.5%
2.6% 3.3% 0 $0
Fabrication
Failure planted on purpose
mypy --check-untyped-defs
Catalog tool · Free program · own check
Rank 10 of 15 ranked rows.
Version
mypy 2.4.1
What it was shown
changed file; the project's third-party packages installed; added line numbers
Counts as flagged when
Flagged when an error with one of the counted codes lands on a line of the completion.
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • --check-untyped-defs switched on
  • only errors that say a symbol does not exist are counted (5 codes)
  • only findings on the lines under review are counted
  • run with the file's third-party packages installed
+77.3%
+72.0% to +82.2%
78.7% 1.4% 0 $0
Fabrication
Failure planted on purpose
mypy (default)
Catalog tool · Free program · own check
Rank 11 of 15 ranked rows.
Version
mypy 2.4.1
What it was shown
changed file; the project's third-party packages installed; added line numbers
Counts as flagged when
Flagged when an error with one of the counted codes lands on a line of the completion.
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 0 (0.0%)
Samples
348 bad, 288 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • only errors that say a symbol does not exist are counted (5 codes)
  • only findings on the lines under review are counted
  • run with the file's third-party packages installed
+72.5%
+66.7% to +77.9%
73.9% 1.4% 0 $0
Obsolescence
Failure planted on purpose
mypy against the latest libraries (deprecated, missing attribute or import)
Catalog tool · Free program · own check
Rank 6 of 9 ranked rows.
Version
mypy 2.4.1
What it was shown
code; latest libraries installed
Counts as flagged when
Flagged when mypy reports an error coded deprecated, attr-defined or import-not-found with the latest libraries installed.
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
310; failed 0 (0.0%)
Samples
155 bad, 155 clean
Answers with no verdict
0 (0 bad, 0 clean)
Cost
free

How this run differs from the tool's defaults

  • error code 'deprecated' switched on; counted: deprecated, attr-defined, import-not-found
  • run with the latest release of each library installed
+14.2%
+6.6% to +23.4%
15.5% 1.3% 0 $0

An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.

Description

mypy is the original gradual-typing checker for Python, reading PEP 484 type hints across a project and reporting type-incompatibility, missing-attribute, and contract-mismatch errors before code runs. It flags fabricated APIs (attributes, names and modules that do not exist) and, with the opt-in deprecated error code (PEP 702), use of deprecated APIs. Configuration lives in pyproject.toml [tool.mypy], which the audit detector now recognises.

Why it's included

Occupies detection × post_generation × fully_autonomous for fabrication and obsolescence in the Python lane. The 2026-11-20 tag audit removed logic_error: that mode is code that passes the type check and is still wrong, and mypy finds type errors (research/fm_tag_audit/final_table.csv). The reference implementation of PEP 484; missing it leaves the audit detector unable to recognise the most widely-deployed Python type-checking gate.

Security notes

Standard dev-tool security posture; no special considerations. mypy runs locally and does not execute target code, only parses and walks its AST.

Addresses failure modes

These tags are the catalog's claim about what mypy is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran.

  • Fabrication — Generating reference to non-existent APIs, packages, or symbols.
  • Obsolescence — Using deprecated APIs, retired packages, or outdated library versions — often because the model's training data pre-dates current releases.

Techniques

static-type-checkinggradual-typingtype-inferencestub-resolution

View raw YAML on GitHub →