CSPaper: sidekick of peer review

334 222 234 124 335 223 344 445 233 122

See here for a crowd-sourced score distribution (biased ofc): https://papercopilot.com/statistics/acl-statistics/acl-2025-statistics/ [image: 1743634054428-screenshot-2025-04-03-at-00.47.22.png]

The ICBINB workshop webpage had a section about this - "AI-Generated Papers" in the reviewer guidelines https://sites.google.com/view/icbinb-2025/reviewer-guidelines [image: 1742515641656-screenshot-2025-03-21-at-01.07.09.png]

And ... Mamba2 was accepted by ICML2024 apparently

@xiaolong yeah, right? Hopefully we do not hear such news in 2025 and onwards…

Zochi's achievement is NOT the first AI-driven scientific research platform. Last year, Llion Jones, one of the original creators behind the Transformer architecture, founded Sakana AI and launched an automated research platform straightforwardly named "AI Scientist", which has already evolved to its 2nd generation. Interestingly, a paper produced by AI Scientist v2 also passed peer review at this year's ICLR workshop on ICBINB, receiving scores of 6/7/6. However, it's important to consider that workshop acceptance criteria typically differ from those of the main ICLR conference, with acceptance rates for workshops being roughly two to three times higher. Despite their acceptance, controversy around AI-driven research persists. Even successful AI-generated papers face the risk of being withdrawn before formal publication due to ongoing academic debate. For example, Intology (the creators behind Zochi) acknowledged that "AI should not be credited as an author in academic work and are currently discussing with workshop organizers whether and how these results should be presented to the research community". Furthermore, according to internal assessments by Sakana using main-conference-level standards, the AI Scientist-v2 paper failed to meet acceptance criteria. This aligns with Intology’s own NeurIPS-based automated evaluation, which gave AI Scientist-v2 an average score below 4 that is actually worse than its predecessor. [image: 1742380427079-screenshot-2025-03-19-at-11.33.19.png] Zochi's performance clearly outshines that of AI Scientist-v2, yet whether its research would succeed at the main conference level remains to be seen, I believe. Due to ongoing controversies surrounding AI-driven research within academia, even if accepted, research teams might withdraw their papers before formal publication. Intology has explicitly stated that, "in the interest of preserving academic integrity, they agree AI should not be listed as an author on scholarly works". Currently, they are in discussions with workshop organizers to determine whether these AI-generated findings should be presented publicly.

Interesting research that got accepted by EMNLP 2023 findings.

Really interesting thought experiment! Compared to other fields such as medicine, I think it is a very good thing that there are usually no or very low processing feels for getting an article published. This really opens up the research to everyone. Compensating the reviewers would make it difficult to keep the fees low. Plus, the improvements in review quality seem to be rather marginal! What do you thing universities/research departments could do to incentivise better reviews?

It is heating up. The scale and tooling for peer review will have to catch up.

Thanks for raising this important topic! Reproducibility is indeed crucial, especially in ML, and something our community needs to actively improve. Encouraging clearer standards during peer review could definitely help. Platforms like "Papers Without Code" are great initiatives to highlight issues. The pushing for open code, better documentation, and transparency is an important effort. Curious to hear more thoughts from everyone: What practical steps could reviewers or conferences take to boost reproducibility?

Thanks for sharing these thinking! I totally resonate with your points, especially about incremental research still being valuable. Not every paper can be paradigm-shifting, and recognizing solid, incremental progress helps keep science moving forward. Plus, the emphasis on methodological rigor and ethical considerations is spot-on. Peer review isn’t easy, but clear guidelines like these definitely make the process smoother ...