Auditing AI peer reviewers: model choice dominates by ~50 points, and citation retrieval only wins where the model's knowledge is thin
A released benchmark injects 357 errors into 30 real arXiv papers to score 12 reviewer systems — and finds tools help only when they supply what the model can't.

A Cambridge/Flatiron/UAB team released a benchmark that audits AI peer reviewers by injecting 357 objective errors into 30 real arXiv preprints. Its headline finding is blunt: which frontier model you pick matters far more than which reviewer architecture you build. Given the identical one-sentence prompt, three frontier LLMs differ by about 50 percentage points in total error detection on both corpora.
What shipped
The paper (arXiv:2611.17052, accepted at NeurIPS 2026) injects errors directly into a paper's LaTeX source at three severity levels — subtle, clear, blatant — across four error types: general issues, mathematics, numerics, and citations. Each of the 30 papers (15 astrophysics preprints sampled from 2026-04-22 arXiv submissions, 15 ML preprints from cs.LG) gets three error-injected copies plus a clean copy used to measure false positives. In total the authors injected 357 errors (177 in astrophysics, 180 in ML). Detection is adjudicated by a cross-family LLM panel (Anthropic + OpenAI + Google), with mechanical verifiers for mathematics and numerics.
Twelve reviewer systems were scored: six single-LLM reviewers (three models × free-form vs. structured prompt), two Claude Code configurations, three public external reviewers (Stanford Agentic Reviewer, OpenReviewer-8B, PaperAudit-Bench), and the paper's own system, Skepthical. Everything is released — the evaluation pipeline, the injection catalogues, and the full adjudication logs — at github.com/inigozubeldia/skepthical-benchmark (code MIT, data CC-BY-4.0).
Skepthical is a multi-agent reviewer with four parallel specialist modules: a general-issues module that merges independent reports from two cross-family LLMs; a mathematics module that verifies claims with wolframscript (symbolic equality) or pint (dimensional analysis); a numerics module that checks numerical statements with executed Python; and a citations module that verifies factual claims against the retrieved PDFs of the cited papers. It is publicly accessible at skepthical-ai.org, and the evaluated config was frozen on 2026-04-21, one day before the astrophysics corpus was sampled — the authors state no prompt or model was tuned in response to the papers.
Why it matters
The results undercut the assumption that "agents always help." Three findings stand out for anyone building or buying review tooling:

Model choice dominates architecture. Under the same free-form prompt, the three frontier LLMs spread by ~50 percentage points in total detection. A structured prompt — Skepthical's full operating instructions compiled into a 978-line prompt, with no tools — roughly halves that gap by lifting the weaker models. Within Claude Code, changing only the prompt from free-form to structured raised detection by 32.2 points. The lesson: before adding tools or scaffolding, check whether a better model or a better prompt gets you most of the way.
Tools help only when they supply what the model can't. Skepthical's enforced citation retrieval gives it the highest citation-error detection on astrophysics by a large margin (46.7% vs. no other system above 26.7%). But on ML that lead vanishes: two structured single-LLM reviewers reach 57.8% on citations while Skepthical gets 53.3% — statistically indistinguishable. The authors' explanation is that the models already "know the literature" on ML. Retrieval is decisive where the model's own knowledge is thin, and noise where it isn't.
Math flags are where automated reviewers are least trustworthy. False-positive rates stay below 10% for most capable systems, but mathematics flags account for roughly half of all false alarms. Skepthical is the only reviewer with no false-positive failure mode jointly across mathematics, numerics, and citations (2.8% overall), precisely because it mechanically verifies math and numerics instead of trusting the LLM's judgment. If you're building review tooling, this is the strongest argument for mechanical verification over pure LLM flagging.
Caveats
Several caveats matter before treating these numbers as ground truth.
LLM-as-judge circularity. Detection rates are adjudicated by LLMs, with mechanical checks only for math and numerics. The pipeline was validated against a human annotator on just one paper. So the "detection rate" numbers partly measure LLM-judge agreement, not ground truth.
Scope limits. The benchmark only tests detection of objective, localisable errors. It deliberately excludes significance, novelty, and overall review quality — the things editors most need from a reviewer.
Handicapped baselines. Stanford Agentic Reviewer is closed-source, undocumented, and ran only on astrophysics; OpenReviewer-8B and PaperAudit-Bench were not retrained or re-prompted, and their sub-3% detection results are presented as-is. The open-weights fine-tuned baselines failing badly is real, but it isn't a fair fight.
No independent replication found. All detection and false-positive numbers come from the authors' own paper and released logs; we found no independent third-party reproduction of the benchmark results. The "first AI reviewer to combine mechanical math/numerics verification with retrieval-based citation verification" claim is the authors' own.
The most useful artifact here is the benchmark itself: a reproducible, fully-released harness (357 injected errors, adjudication logs, cost ledger) that lets you score any new reviewer against the same injected errors. That is worth more than any single headline number.