Abstract
We evaluate candidate AI reviewers for a planned experimental deployment at the Transactions on Machine Learning Research (TMLR) journal, in which an AI-generated review will be included alongside standard human reviews. We focus on whether AI reviewers can assess the soundness of submitted papers, because checking soundness is well suited for AI reviewers and is also directly aligned with TMLR’s primary acceptance criterion. We evaluate four candidate AI reviewers on recent TMLR submissions with Action Editor decisions and on papers containing inserted claim-breaking errors. We find that soundness-focused AI reviews can provide useful assistance in real peer-review workflows. Based on these evaluations and manual inspection of generated reviews, we identify a candidate system suitable for experimental deployment at TMLR