pr-agent
16,537AI-powered tool for automated pull-request analysis, feedback, and suggestions.
Activity band details
Activity band A to F: a bin of a score that adds GitHub stars (log scale), a push to the upstream repository in the 12 months before `as_of`, the number of declared integration surfaces and, where the catalogue records them, citing papers and benchmark evaluations. It measures popularity and activity. It is not a level of evidence: no band means a paper cites the tool or that it was measured. Recency is judged as of 2026-11-22.
Computed band: B
(raw score 6.69).
Shown band: B
.
Facets:
documented_in=0evaluated_on=2integration_surfaces=1paper_citations=0recent_push=2stars=1.69
Source of truth: pipeline/maturity/score.py
Measured on SAICA-Bench in this configuration
Each row is one run of pr-agent, set up by us as its run card says, on one labelled set of faulty and clean samples (how we test). It says what that configuration flagged on that set, not how good pr-agent is in general: another model, another input or the tool's own defaults could score differently. J = TPR - FPR, on the answers that carry a verdict. 0% is chance, 100% flags every bad sample and no clean one. J is the sort key within a mode; each row also has false alarms per 100 clean changes and precision at 1%, 5% and 20% prevalence.
| Set | Configuration and run card | J (95% interval) | TPR | FPR | No verdict | Cost per 1,000 reviews |
|---|---|---|---|---|---|---|
|
False 'done' claims
failure mode: Incomplete execution
Real agent output
|
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 4 of 10 ranked rows; no reviewer reaches J 25% on this set.
How this run differs from the tool's defaults
|
+14.7%
+3.4% to +25.5%
|
67.8% | 53.1% | 14 | ~$15.36 |
|
Security vulnerability
Real agent output
|
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 10 of 13 ranked rows; no reviewer reaches J 25% on this set.
How this run differs from the tool's defaults
|
0.0%
−6.8% to +7.1%
|
83.6% | 83.6% | 157 incomplete |
$2.90 |
|
Logic error
Model-written code, not agent output
|
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 4 of 7 ranked rows.
How this run differs from the tool's defaults
|
+53.2%
+47.9% to +58.3%
|
91.4% | 38.2% | 1 | $1.29 |
|
Fabrication
Failure planted on purpose
|
PR-Agent review, any focus area listed (deepseek-v4-flash)
Catalog tool · AI-based · own check · judge model deepseek-v4-flash
Rank 13 of 15 ranked rows.
How this run differs from the tool's defaults
|
+68.9%
+63.9% to +73.8%
|
75.9% | 7.0% | 1 | ~$0.23 |
An answer with no verdict (failed call, blocked request, text cut by the token budget, no verdict in the text) is unreadable: it is left out of the rates and counted beside them, per label. “~” marks an estimated cost. SAICA-Bench results v2.0.0, built 2026-11-22.
Description
pr-agent automates PR review using LLMs. It was published by CodiumAI (now Qodo) and today lives under the GitHub organisation The-PR-Agent. Produces structured code review, security hints, and test suggestions gated on a maintainer who approves diffs. Canonical exemplar for human-in-the-loop code-review supervision.
Why it's included
Code-review supervisor cell — distinct from generic linters / static analyzers by reasoning over diff intent and leaving humans as the final authority.
Addresses failure modes
These tags are the catalog's claim about what pr-agent is built to catch or prevent. A tag is not a test result. The rows under Measured on SAICA-Bench are the evidence, for the sets and the configuration we ran.
- Logic error — Correct API use, incorrect logic.
- Security vulnerability — Insecure patterns, injection flaws, weak cryptography, secret leakage.
- Incomplete execution — Agent claims completion while subtasks are skipped, stubbed, or silently dropped.