TMLR Selects CSPaper as the AI Reviewer for Its Soundness Pilot

C

CSPaper

A part of Scholar7 AB · July 2026

TL;DR

  • Transactions on Machine Learning Research (TMLR) will experiment with one AI-generated review per submission, alongside its regular human reviews.
  • After an independent evaluation of four systems, including GPT-5.5 and Claude Opus 4.7, TMLR selected CSPaper's review agent for the pilot.
  • The AI review does one job: assess whether the paper's claims are supported by accurate, convincing, and clear evidence. No accept/reject recommendation.
  • It is advisory only. Human reviewers still review, authors can respond, and Action Editors decide.
  • Deployment via OpenReview is expected in August 2026.

One of machine learning's most respected journals tested four AI reviewers on real submissions and on papers with planted errors.

Then it picked ours.

We are proud, a little stunned, and very aware of the responsibility. Here is the whole story, numbers included.

TMLR x CSPaper pilot partnership: one AI review per submission, focused on soundness, selected on evidence, deploying August 2026

The news, in one breath

TMLR announced that it will experimentally deploy AI-generated reviews: every new submission will receive one concise AI review next to its regular human reviews. The AI reviewer for this pilot is CSPaper. The announcement is also making the rounds on LinkedIn.

The scope is deliberately narrow. TMLR's primary acceptance criterion is a single question, and it is the question the AI review is asked to answer:

Are the claims made in the submission supported by accurate, convincing and clear evidence?

That is it. No novelty scoring, no excitement rating, no "I wish the authors had cited my 2019 paper". Just soundness: do the claims hold up against the evidence in the paper? It is the part of reviewing that is closest to verification, and verification is the thing we have been building CSPaper around from day one.

How the selection worked

This was not a marketing partnership, and nobody filled in a form. Vishisht Rao and Nihar B. Shah of Carnegie Mellon University (Shah is also a co-Editor-in-Chief of TMLR) ran an independent evaluation of four candidate AI reviewers: CSPaper, ReviewerToo, OpenAI's GPT-5.5, and Anthropic's Claude Opus 4.7. Our part was simple: we built TMLR review agents according to the journal's official rubric and review format, provided API access, and had no further involvement. No special prompts, no tuning for the evaluation, no visibility into the test set.

The evaluation used two very different test beds:

  • Real TMLR submissions with known outcomes: recent accepted papers, papers rejected because the claims were not supported, and papers rejected on both the claims and audience criteria. The Action Editor's judgment on the claims-and-evidence question served as ground truth.
  • Papers with planted errors: a freshly built version of the FLAWS benchmark, where a subtle, claim-breaking error is inserted into a published paper. New papers, never seen by any AI reviewer, specifically constructed so nobody could have memorized the answers.

The full methodology and results are in the technical report, "Evaluating AI Reviewers' Ability to Assess Soundness for Deployment in the TMLR Journal" (also available as a PDF from CMU). We recommend reading it. Not only because it is about us, but because it is a careful piece of work on a question the whole community is wrestling with: how do you evaluate an AI reviewer without fooling yourself?

The numbers behind the decision

Figure 1 puts all five evaluation sets in one place. On the left, real TMLR submissions where ground truth is the Action Editor's answer to the claims-and-evidence question; CSPaper led in all three categories. On the right, the two planted-error sets, where the correct verdict is "not sound" by construction and every system scored low:

Bar chart of correct verdicts on the claims-and-evidence question across all five evaluation sets and all four systems: CSPaper 92.7% on accepted papers, 90.9% on papers rejected for unsupported claims, and 100% on papers rejected on both criteria, ahead of GPT-5.5, Claude Opus 4.7, and ReviewerToo; on the two FLAWS planted-error sets all systems score low (CSPaper 17.9% and 30.0%)
Figure 1. Correct verdicts on the claims-and-evidence criterion, across all five evaluation sets and all four systems. Derived from Table 1 of the report, which lists the percentage of papers where each AI answered "yes"; for the four categories whose ground truth is "no", we plot 100 − yes% so every bar reads as "share of correct verdicts, higher is better". ReviewerToo was run only on a subset of categories (a partial evaluation, per the report) due to reliability and runtime constraints, shown as "n/a" elsewhere.

The number we are proudest of is the quiet 100 in the third group: across 46 papers rejected on both criteria, CSPaper never once called the claims supported. Zero false passes. For a journal deciding whether an AI review can be trusted next to human ones, the failure mode that matters most is an AI cheerfully waving through a paper whose claims do not hold. On that axis, CSPaper was the conservative one in the room.

The planted-error papers were a much harsher test, by design: the FLAWS methodology filters out every error that is easy, trivial, or superficial. No system reliably caught these errors outright, and the report is refreshingly blunt about that. But when the annotators looked at what the reviews actually said, a pattern emerged:

Horizontal bar chart: among sampled reviews that judged a planted-error paper sound, the share that still pointed at the planted error was 70% for CSPaper, 40% for GPT-5.5, and 25% for Claude Opus 4.7
Figure 2. Pointing at the planted error, even while misjudging its severity. From Table 2 of the report: among 20 sampled reviews per system that answered "yes" to the claims-and-evidence question on planted-error papers, the share whose text still identified the inserted error (often dismissing it as trivial). For a reviewer-assistance tool, a review that says "this part looks off, though maybe minor" still routes human attention to the right paragraph.

In other words: even when CSPaper got the verdict wrong, its review usually had its finger on the right paragraph. It found the error and filed it under "minor", which is a very human mistake to make. And when CSPaper answered "no" on these papers, its stated reason pointed at the inserted error over 80% of the time, rather than at something unrelated.

We want to be as honest as the report is, so three caveats, stated plainly. First, no system was an unambiguous winner across every metric; under the strictest reading of the planted-error test, GPT-5.5 caught more errors outright. Second, CSPaper's claims-and-evidence comments run longer (a median of 400 words versus 254 for GPT-5.5), and length alone is not precision. Third, our pipeline has been tuned on reviewing rubrics by design, which the report rightly asks readers to keep in mind. The selection came from the whole picture: consistent agreement with TMLR outcomes, useful pointers even when wrong, and a manual read-through of CSPaper's reviews by a TMLR co-Editor-in-Chief who found them of sufficient quality to move forward.

What the pilot looks like at TMLR

From roughly August 2026, integrated into OpenReview:

  • Every new TMLR submission receives one concise AI review, alongside the usual human reviews.
  • The AI review assesses soundness only and makes no accept/reject recommendation.
  • It is visible to Action Editors, reviewers, and authors, and authors get the opportunity to respond.
  • The AI does not argue back. One review, no rebuttals from the machine.
  • Action Editors make the final call using the human reviews, the AI review, and the author responses. The AI review is advisory, never binding.
  • TMLR will collect feedback from authors, reviewers, and Action Editors throughout, and the experiment will evolve accordingly.

Nobody is being replaced. Reviewer 2 is simply getting a colleague who has read the appendix. Twice.

If you run a venue, this can be yours too

What made this work is that CSPaper's reviews are rubric-faithful: the agent is built to answer a venue's actual acceptance criteria, in the venue's actual review format, rather than producing generic paper commentary (that is also the core idea of our INLG 2025 system paper). TMLR's soundness criterion mapped cleanly onto it. So would yours.

If you help run a conference, journal, or workshop and are curious what a TMLR-style experiment would look like for your review process, we would genuinely love to explore it with you. Concretely, we are happy to:

  • Build a verification agent tailored to your rubric, whether that is soundness, reproducibility, ethics compliance, or your own review form. This is not a bespoke one-off each time: our self-improving agent factory is how we create and refine new agents for partners, with experts defining the rubric and every automated improvement kept auditable and versioned.
  • Support independent benchmarking of the agent before any deployment, TMLR-style: real decisions, planted errors, no marketing numbers.
  • Run a scoped pilot with your editors or program chairs, with humans in charge at every step.

One email starts the conversation: partner@scholar7.com. It is read by the founders, and we answer quickly.

If you write papers, you can use this today

Here is the part we find genuinely useful for individual researchers: the system TMLR evaluated is not a private enterprise build. It is the same engine running on CSPaper right now, and you can point it at your own draft before anyone with decision power does:

  • Correctness Check asks your paper the same question TMLR's pilot asks: are the claims supported by the evidence? Your first three runs are free.
  • Venue-specific review agents review your draft against the rubric of your actual target venue, from NeurIPS to ACL, so the surprises happen in private. That now includes a TMLR review agent: the very reviewer joining the pilot.
  • Reference Check, Idea Check, and Rebutly cover the rest of the submission lifecycle, from citations to rebuttal.
  • OpenPrint lets you share your work with the verification attached, through a fully self-service process: no moderator, no human reviewer, no waiting for anyone's approval. Publish, collect community feedback on the page, and get discovered: OpenPrint papers are indexed by Google Scholar. It is exactly where the TMLR report itself lives.

The math is simple. If a soundness reviewer is going to read your paper anyway, you want the first one to be the one that reports to you. Maximize the chance your work lands the way you intended; minimize the chance a critical flaw makes it to the version reviewers see.

The best compliment an AI reviewer can receive is not a benchmark score. It is a rigorous journal looking at the evidence and saying: alright, you may sit with the humans.

Thank you to TMLR, and to Vishisht Rao and Nihar B. Shah for an evaluation worth being measured by. Now we get to earn it, one submission at a time.

Try the TMLR review agentExplore Correctness Check

The TMLR review agent is powered by the same Correctness Check engine that was evaluated in the report.

TMLR Selects CSPaper as the AI Reviewer for Its Soundness Pilot | CSPaper — CSPaper