Skip to main content

Tools

pr-agent

16,537

AI-powered tool for automated pull-request analysis, feedback, and suggestions.

Detection Post-generation Full HITL AI-based Stable Activity band B EnvironmentHuman
Activity band details

Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.

Computed band: B (raw score 6.69). Shown band: B .

Facets: documented_in=0evaluated_on=2integration_surfaces=1paper_citations=0recent_push=2stars=1.69

Source of truth: pipeline/maturity/score.py

Measured on SAICA-Bench in this configuration

Each row is one run of pr-agent, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good pr-agent is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.

Set Configuration and run card J (95% interval) TPR FPR No verdict Cost per 1,000 reviews
False 'done' claims
failure mode: Incomplete execution
Real agent output
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 4 of 10 ranked rows; no reviewer reaches J 25% on this set.
Version
pr-agent 0.50.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
repository files at the base commit; final patch (unified diff); issue text
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
300; failed 14 (4.7%): setup error 2, tool error 12
Samples
a subset: 150 bad and 150 clean of the set's 600 and 600
Answers with no verdict
14 (7 bad, 7 clean): setup error 2, tool error 12
Cost
~$15.36 per 1,000 reviews (api); $4.61 spent on this run. Estimated: run on the first 150 + 150 samples only (our cost $4.61), scaled to all 1200; rates are on those 300. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
+14.7%
+3.4% to +25.5%
67.8% 53.1% 14 ~$15.36
Security vulnerability
Real agent output
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 10 of 13 ranked rows; no reviewer reaches J 25% on this set.
Version
pr-agent 0.50.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
git repo with the touched files before and after the change; task text
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-19/20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
1,702; failed 157 (9.2%): setup error 51, tool error 106
Samples
1,243 bad, 459 clean
Answers with no verdict
157 (88 bad, 69 clean): setup error 51, tool error 106; under 95% of one class was readable
Cost
$2.90 per 1,000 reviews (api); $4.93 spent on this run. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
  • reviews that failed on provider errors are counted as failed runs, not as 'nothing found'
0.0%
−6.8% to +7.1%
83.6% 83.6% 157
incomplete
$2.90
Logic error
Model-written code, not agent output
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 4 of 7 ranked rows.
Version
pr-agent 0.50.0
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
task description; function source
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-20
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
2,440; failed 1 (0.0%): tool error 1
Samples
1,536 bad, 904 clean
Answers with no verdict
1 (0 bad, 1 clean): tool error 1
Cost
$1.29 per 1,000 reviews (api); $3.16 spent on this run. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
+53.2%
+47.9% to +58.3%
91.4% 38.2% 1 $1.29
Fabrication
Failure planted on purpose
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 13 of 15 ranked rows.
Version
pr-agent 0.50.0, litellm 1.105.1
Judge model
deepseek-v4-flash-0731
Provider
Hyperbolic
What it was shown
git repo with a base and a change branch
Counts as flagged when
the review has a non-empty "Recommended focus areas" section
Temperature
0
First run
2026-11-19
Check
own check: the tool ran its own built-in check
Catalog
in the catalog
Runs
636; failed 1 (0.2%): tool error 1
Samples
348 bad, 288 clean
Answers with no verdict
1 (0 bad, 1 clean): tool error 1
Cost
~$0.23 per 1,000 reviews (api); $0.15 spent on this run. Estimated: the cost ledger holds 285 calls for 657 completed reviews ($0.07 logged; the rest ran before logging started), so the cost per review is taken from the logged calls. Charge reported by the API with each call (Hyperbolic)

How this run differs from the tool's defaults

  • run through its local git provider on a temporary repository, not on a hosted pull request
  • its general review prompt, unchanged; no extra instructions naming the failure
  • the score, tests and effort sections of the review were switched off
  • our rule turns the review text into a verdict: flagged when the review lists any focus area
  • it sees the change only, not a task description
+68.9%
+63.9% to +73.8%
75.9% 7.0% 1 ~$0.23

An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.

Description

pr-agent automates PR review using LLMs. It was published by CodiumAI (now Qodo) and today lives under the GitHub organisation The-PR-Agent. Produces structured code review, security hints, and test suggestions gated on a maintainer who approves diffs. Canonical exemplar for human-in-the-loop code-review supervision.

Why it's included

Code-review supervisor cell — distinct from generic linters / static analyzers by reasoning over diff intent and leaving humans as the final authority.

Addresses failure modes

These tags are the catalog's claim about what pr-agent is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran.

Techniques

pr-reviewsecurity-hintstest-suggestionsmaintainer-gate

View raw YAML on GitHub →