AI vs. Human Peer Review: Harvard Finds AI Wins on Every Measure

Every researcher knows the “Reviewer 2” 😈 of academic folklore: skims the abstract, asks why you did not cite a 2009 paper that turns out to be their own, and rejects in four sentences. If AI only had to beat that reviewer, nobody would write a paper about it. This study set a much higher bar.

On October 22, 2026, Gary King, a University Professor at Harvard and director of its Institute for Quantitative Social Science (IQSS), and Steven Worthington, the institute’s director of data science, posted a paper whose title sounds like a challenge thrown across a bar: “AI Article Reviews Dominate Human Peer Reviews at Their Best” [1]. The word doing the work is best. They did not pit AI against Reviewer 2, or against a tired, unpaid reviewer squeezing in a report between grant deadlines. They pitted it against the strongest human reviews they could find.

By their own account, they went in expecting AI to win “some in some areas, some of the time.” It won in every area they measured, by wide margins. Within a day, Wharton’s Ethan Mollick was telling academics to work out quickly how research practice should adapt [2], and Stanford political scientist Yiqing Xu argued that the human expert’s job is “no longer about writing 1,500-word essays to summarize findings, check proofs, and so on” [3].

Our work starts from one question: how can review become faster, fairer, and more trustworthy for everyone who depends on it? A result like this matters for that question, and we read it with real interest. But the study is more interesting than its scoreboard, and its limits matter as much as its headline. Here is the whole thing, with our notes in the margin.

The timing is no accident. Peer review in computer science, and increasingly everywhere else, is buckling under volume.

• ICLR 2026 received 19,525 valid submissions. Handling them took 18,054 reviewers and 76,139 reviews [4].
• ICLR 2027 registrations reportedly passed 60,000 at the abstract deadline in September, roughly as many as all previous ICLRs combined. That figure comes from a researcher’s post, not from ICLR [5]. The program chairs had already capped authors at 20 submissions and limited papers with no qualified reciprocal reviewer to one per author [6]. Reviewers who use an LLM must now hand over their original, self-written assessment and their LLM interactions, and misuse can get every paper they authored desk-rejected [7].
• arXiv took in a record 40,363 submissions in September 2026, twice the 20,569 of September 2024. From October 1, each submitter gets two submissions a month [8].

Overloaded reviewers did what overloaded people always do: they found a shortcut. Weixin Liang, James Zou and colleagues at Stanford estimated that 6.5% to 16.9% of the review text submitted to ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023 could have been substantially modified by an LLM. The share was higher for reviews filed close to the deadline, for reviewers reporting low confidence, and for reviewers who did not answer author rebuttals [9]. The fingerprints are easy to spot. In ICLR 2024 reviews, the word “meticulous” became 34.7 times more likely to appear, “intricate” 11.2 times, and “commendable” 9.8 times.

It has not slowed down. A detection vendor flagged about 21% of ICLR 2026 reviews as fully AI-generated, an estimate rather than a count [10]. A Frontiers survey of about 1,600 researchers, reported by Nature, found that more than half had used AI while reviewing, often against guidance [11]. At ICML 2026, which randomly assigned reviewers to AI policies, 22.5% of surveyed reviewers under the no-AI policy said they used an LLM anyway [12].

So the AI is already in the room. It came in through the side door, undisclosed and unevaluated. The Harvard paper asks the more useful question: what happens if you let it in through the front door and measure it?

Before treating human reviews as ground truth, recall how human peer review has done when people actually tested it. The results have been sitting in the literature for decades.

Richard Smith, a former editor of The BMJ, summed up the evidence two decades ago: peer review sits at the heart of science, yet it is slow, expensive, inconsistent, prone to bias and abuse, and backed by little proof that it works [17]. A 2025 review in PNAS of the ideas, interventions, and evidence around peer review finds the same problems still with us, from weak incentives to low agreement between reviewers [18].

None of this means human reviewers are lazy or careless. It means the system asks too much of too few people for too little. That is exactly why the Harvard comparison is fair to the humans: it removes most of those pressures.

In the human corner

The human reviews come from rapidPeer, a service IQSS built (and later opened to other institutions) that gives faculty double-blind reviews of their drafts before journal submission. The reviewers are picked by experts in the field from other institutions. They are paid well, and told that good work means more work. They are asked to read the entire paper and write at least 1.5 to 2 pages of constructive suggestions, not to decide whether the paper deserves a particular journal.

By every measure available, the reviews were excellent. Of 74 authors surveyed a week after delivery, 88% rated their review high or very high quality (60% very high), and the median willingness to recommend the service was 10 out of 10. When the authors informally consulted former editors of major journals, all of them said these reviews were among the best they had seen. The corpus is every eligible submission from 2019 to 2026: 124 papers and 134 human reviews, spanning statistical, mathematical, and qualitative work [1].

In the AI corner

Claude Opus 4.8, at its maximum reasoning setting, was entered twice. The matched version got the anonymized manuscript and the human reviewers’ exact instructions, with no tools at all. The tuned version got a purpose-built prompt and web search, which it used 12.4 times per review on average. Each review came from a single isolated call. If anything, the humans had a small information edge: they could read supplementary files, which the AI never saw.

The judges

There were two kinds of judge, one human and one machine. The machine was GPT-5.5, from a different company and model family, which judged every pair with formatting stripped and authorship hidden. Each comparison ran twice, once in each order, and a win counted only if the same review won both times. The judge first decided which of seven review areas applied to each paper, locked that list in, and then compared and scored the reviews area by area.

The humans were three former editors of leading journals. They judged a random sample of pairs under the same instructions, without being told that either review came from an AI. Their verdict gets its own section below.

The tuned prompt, printed in full in the supplement, rewards a read even if you never touch an AI. It tells the model it is “advisory, not a gatekeeper,” orders it to verify every source by search and “never cite from memory,” and to lead with the two or three issues that would most change the paper. Then it says the thing every reviewer should tape to their monitor:

“Three references that genuinely bear on the argument beat fifteen that merely touch the topic. The expertise is in the filter, not the find.”

Start with the overall verdict. Reading only the two reviews, the judge preferred the AI review in all 134 pairs for the no-web version and in all 132 decided pairs for the web version. Given the full manuscript and told to check each review’s claims against it, the judge still picked the AI in 133 of 133 and 132 of 132 decided pairs. The human review never won both presentation orders, not once [1].

Area by area, the pattern holds. Figure 4 shows the mean 0 to 10 rating in each area. The widest gaps are in additional sources (4.73 for humans, 9.55 for the AI) and in figures and tables (2.95 against 7.57). Part of that is coverage: many human reviews say nothing about the figures. But even among the 90 pairs where both reviews discussed them, the AI led by 3.4 points.

The head-to-head win rates in Table 2 add one more twist. Without web access, the AI won the additional sources area only 62% of the time. With search, it won 100%. The model alone is not the whole trick. Give it a library card and a rule against citing from memory, and it stops resembling the hallucination machine many people remember from 2023 and starts behaving like a well-read colleague.

An AI judging AI is the obvious weak point, so the authors brought in people. Three former editors of leading journals each compared ten randomly chosen pairs, presented side by side in random order, under the same instructions the AI judge got. Nobody told them one review in each pair was written by a machine; they found out at the debrief.

The editors chose the AI review 8, 8, and 10 times out of 10. One of them never picked the human review at all. And here is the detail we keep coming back to: the editors agreed with each other 80% of the time, but with the GPT-5.5 judge 87% of the time. The ten papers they saw were typical of the corpus, with human reviews averaging 6.03 on the AI judge’s scale against 6.02 for the rest.

The standard objection to AI review is that it makes things up. A review can be persuasive and wrong, so the authors checked accuracy three ways.

1. Do the cited works exist? The web-enabled AI cited 651 works with a DOI or arXiv identifier. 648 matched a record in Crossref or arXiv automatically, and the other 3 were confirmed by hand. Fabricated identifiers: zero.
2. Are they used correctly? In a random sample of 20 reviews, the cited work was on topic in 99 of the 100 cases a judge could decide. Where a review attributed a specific finding to a paper, the abstract supported it in 23 of 24 cases.
3. Do the reviews describe the manuscript correctly? Every individual claim in the no-web AI reviews and in the human reviews was shuffled, stripped of its source, and checked against the manuscript. The bar for “wrong” was strict: the paper had to directly contradict the claim.

The experts were wrong about the manuscript in 15.9% of their claims (152 of 956). The AI was wrong in 5.5% (57 of 1,039), about a third as often. Nearly two-thirds of human reviews contained at least one such error, against just over a third of AI reviews. A person double-checked 90 of the audited claims, including every one flagged against the AI, and agreed with the audit on 87. All three disagreements were cases where the audit had wrongly faulted the AI.

Could the audit model simply be bad at spotting errors? The authors tested that too. They planted one false sentence in a copy of every review, each a minimal edit that contradicted the paper, alongside a matching true sentence in another copy. The checker caught all 258 false sentences and flagged none of the 258 true ones.

Many authors will recognize the pattern: a confident “the authors never test robustness” about a paper whose Section 5 is titled “Robustness.” It is not carelessness so much as time pressure. A busy expert reading quickly can end up reviewing the paper they expected rather than the one in front of them. The AI, with no deadline to meet, kept closer to the paper that is actually there.

Still, read the numbers in both directions. More than a third of AI reviews contained at least one claim the manuscript contradicted. “Fewer mistakes than an expert” is not “no mistakes.” Someone still has to read the review with the paper open.

A result this lopsided deserves suspicion, and the authors went looking for ways to break it. They ran ten families of robustness checks. Here are the ones a reviewer would ask for first.

The style test matters most, because LLM judges are known to favor LLM-written text [19]. Dressing human arguments in AI clothing did not help them at all. Whatever the judge rewarded, it was not the prose.

It would be easy to turn this paper into a headline that says “AI replaces peer review.” The paper does not say that, and the authors are careful about it. Before you forward it to your department chair, keep these limits in view.

• Advice, not gatekeeping. Reviews exist to help authors improve papers and to help editors decide. The study speaks to the first job only. Nobody measured whether AI reviews pick the right papers to accept.
• One field cluster. 114 of the 134 pairs were quantitative-statistical social science; only 9 were formal and 11 qualitative. Earlier, smaller comparisons in medicine point both ways: human reports at BMJ Open raised more detailed points than ChatGPT [20], while cardiology editors preferred AI reviews on five of seven criteria [21]. Machine learning and biomedicine need their own test.
• A small human jury. The human verdict rests on 30 picks by three editors who read the reviews, not the papers. It agrees with the AI judge, but most of the evidence still comes from an AI judging AI.
• No downstream outcome. The study did not measure whether papers revised with AI advice became better papers, or what happens to a field when everyone revises against the same models.
• Sameness. AI tools raise individual scientists’ output while narrowing what science as a whole looks at [22], and LLM creativity is homogeneous across models [23]. Reviews written by the same few models can show a “hivemind” of excess agreement, and rewriting a paper with an LLM can raise AI reviewers’ scores without changing the science [24]. When three reviewers lean on one model, their mistakes stop being independent.
• Gaming. Hidden prompts in manuscripts already try to sweet-talk AI reviewers [25]. Any public AI reviewer becomes a target.
• Deep errors are still hard. In an independent evaluation of four AI reviewers for TMLR, no system reliably caught subtle errors planted into published papers [26]. That includes ours, the one TMLR went on to select. Google’s Paper Assistant Tool found 89.7% of known math errors on a 29-error benchmark, against 55.2% for a single model call [27], yet pilot users also reported it challenging proofs that were correct [28].

The second half of the paper is a menu for editors. It is the part most likely to be quoted in editorial board meetings, so we report it carefully.

1. Self-review before submission. Journals recommend or require that authors run repeated AI reviews and revise until they have fixed everything fixable: in effect, a self-supervised revise-and-resubmit.
2. An official prompt. Each journal publishes the AI prompt it wants authors to use, built from its own instructions to contributors. Fewer surprises, fewer desk rejections.
3. The AI review joins the dossier. Authors submit the official AI review with a letter explaining where they disagree. The letter matters, the authors argue, so that model makers do not end up deciding what papers say, and because a well-revised paper eventually draws pedantry like “yes 2+2 = 4, but you never defined the + sign.”
4. Editor-led, humans on demand. Skip routine human review. The editor reads the dossier, runs a second AI review from a prompt authors never see (with manipulation checks [29]), and sends hard cases to a board member or an outside expert. The editor decides.
5. AI reviews only. The authors raise the obvious problem themselves: if the only review is one authors can run in advance, they will revise until it is favorable, and it will tell the editor nothing new.

Done well, the authors argue, publication decisions could take days instead of months, with less of the adversarial theater that makes peer review miserable for authors. Done badly, authors end up optimizing for the reward model instead of the science. That is one of the oldest traps in measurement.

The authors’ safeguard is unglamorous: study the cases where authors disagree with the AI and the editor sides with the author, and keep updating the prompts and models. A published prompt is a measure. The day authors start writing for it, it needs changing.

Large venues are already halfway there. A randomized trial at ICLR 2025 sent AI feedback to reviewers on more than 20,000 reviews; 27% of the reviewers who got it updated their reviews [32]. AAAI-26 attached one clearly labeled AI review to each of 22,977 papers sent for full review [33]. TMLR is piloting one soundness-focused AI review per submission, after an independent evaluation of four candidate systems [26].

Not for writing 1,500-word essays that summarize findings, a job Yiqing Xu says is no longer the expert’s [3]. Not for hunting down the literature the author missed, tidying the prose, or noticing that a figure does not make its point. On this evidence, a machine now does much of that better, faster, and with fewer mistakes about what the paper says.

What is left is the part that was always the point. Is this question worth asking? Does the result change what anyone in the field believes? Is this the right venue, and is this the right moment? Who is accountable if we get it wrong? These are judgments, and the paper is explicit that they belong to a human editor with the authority and the responsibility to make them.

We would add three practical rules.

1. Bring the AI in through the front door. One disclosed, standardized, evaluated AI reviewer per venue is better than a patchwork of undisclosed tools that nobody can audit.
2. Benchmark before you deploy. Test the reviewer on your own past decisions and on papers with planted errors, the way TMLR did, before it touches a live submission.
3. Keep a human on every verdict. Let the AI read everything. Let a person decide what it means.

“AI does not replace the human reviewer. It frees them for the part of the job that only people can do.”

The Harvard results make the next question more urgent: what should increasingly capable AI reviewers be designed to optimize?

Earlier this year, we set out a verification-first vision for AI-assisted peer review: the objective should be to expand the capacity to check scientific claims, not simply the capacity to produce reviews [34]. Our model of the problem shows why this matters. When claims outpace the capacity to verify them, effort can shift from seeking the truth to optimizing the proxy, even while decisions still look reliable. It is Goodhart’s law, applied to peer review. The Harvard study gives us an important occasion to revisit that distinction.

Its findings deserve attention, including the checks on whether reviews accurately describe the manuscripts. But a review that faithfully describes a paper is not, by itself, an independent verification of the paper’s central result. The study also does not establish what happens when researchers begin optimizing their submissions against widely deployed AI reviewers.

For CSPaper, we propose a practical standard: consequential criticisms should come with inspectable support.

This is not a reason to discount better AI reviews. It is a reason to build on them. Human editors should retain responsibility for decisions, while the tools they use make those decisions easier to examine and correct. The ambition is not merely to produce another opinion at lower cost. It is to make the scrutiny behind that opinion more reliable.

The most practical suggestion in the Harvard paper is also the least dramatic: get an AI review before you submit, read it with the paper open, and revise. For venues, the lesson from this study and from TMLR’s evaluation is the same: test an AI reviewer against your own past decisions and against papers with planted errors before you trust it with a live submission.

This is the problem CSPaper was started to work on: making careful verification a normal, open part of research instead of a private shortcut. Two things we can offer, if they help.