AI in Peer Review: Provocation (slides)

AIME-Con 2026 · Educational Measurement as an AI-Native Profession Peer review should use AI twice Once for the reviewer, once for the editor, both run by the journal Derek Briggs · University of Colorado Boulder Why review needs help Review was strained before generative AI 8M+ scientific publications in 2025, double the count five years earlier 4.5 invitations per completed review in 2025, twice the 2018 figure 53% of reviewers used AI for review tasks in 2025, up from 29% a year earlier The right comparator for AI-assisted review is the median human review, not the best one. (Stephen Turner, UVA, in Science) Sources: Brainard, Science 393 (2026); Silverchair report (2026); Frontiers reviewer survey (2025) What 14 journals say now No journal sets its own reviewer rules 13 of 14 journals require authors to disclose generative AI use 12 of 14 bar reviewers from putting manuscript text into an AI tool 0 of 14 set their own reviewer or editor rules. Twelve inherit them from seven publishers. The stated reason is confidentiality, not AI quality. Yet publishers already run their own AI on these manuscripts. Source: Briggs, policy scan of 14 journals (3 NCME, 9 other measurement, 2 comparisons), 4 Oct 2026 The proposal Two uses, both run by the journal Use 1 · Reviewer Review first, then audit the AI's review - Write an independent review. - Read the journal's AI review of code, math, statistics, and reporting. - Revise or not, and tell the editor which. Use 2 · Editor Ask AI how the human reviews fit together - Receive the human reviews. - Ask AI where reviewers agree, conflict, and what no one checked. - Decide, with the AI report as one input. In both uses, humans write every review and make every decision. The evidence · ICLR 2025 randomized trial Reviewers revise after AI feedback 20K+ human reviews got optional AI feedback on their drafts 27% of reviewers who got feedback revised, using 12,000+ AI suggestions 2/3 of revised reviews were judged better by an independent panel

The evidence · ICLR 2025 randomized trial Reviewers revise after AI feedback 20K+ human reviews got optional AI feedback on their drafts 27% of reviewers who got feedback revised, using 12,000+ AI suggestions 2/3 of revised reviews were judged better by an independent panel Caveat: the AI commented on reviewers' drafts, not the manuscript. At AAAI-26, all 22,977 main-track papers got an AI review in under 24 hours, at about $1 each; it was weakest on novelty and significance. Sources: Thakkar et al. 2025; Nature Machine Intelligence 2026, via Science; AAAI-26 pilot report, 2026 Unresolved Whose judgment is it? Accountability When a reviewer revises after reading the AI's critique, who answers for the review? No journal policy in our field covers a review the journal itself supplies. Independence If every reviewer reads the same AI review, the reviews are no longer independent ratings. The editor may see more agreement but have less evidence. Precedent: COPE and NCME let a reviewer bring in a colleague if the journal knows who helped. Does that rule extend to AI?