A NeurIPS2022 reviewer's quick view on when to reject a paper
-
Have you ever wonder, as a reviewer, under what circumstances would you choose to reject a paper?
To provide clarity, let's look at insights shared by an experienced reviewer, recognized with a Top Reviewer Award at NeurIPS. They were initially posted in 2022 and I find still very relevant to nowadays peer reviews.
Types of papers and how they influence rejection decisions:
Paradigm-shifting contributions
These papers fundamentally change your understanding or introduce revolutionary concepts, prompting an enthusiastic reaction. Such papers easily deserve a rating of "strong accept" or higher, and reviewers usually actively defend these submissions during discussions.
For example, Stanford’s paper on differential privacy drastically reducing computation costs from tens of times down to twice the normal workload reshaped perceptions of privacy-computation complexity. Similarly, transformative works like the invention of Transformer models or adaptive optimization techniques (e.g., AdaGrad) fall into this category.
Incremental, but valuable confirmations
Approximately 70% of academic work fits here. These papers might not be groundbreaking but provide valuable confirmations and thorough examinations of existing ideas. They help reduce uncertainty about a research path by validating and refining current methodologies.
For instance, after Neural Tangent Kernel (NTK) theory initially explored two-layer neural networks, subsequent work by researchers like Allen Zhu expanded these results, covering cross-entropy loss and SGD settings. Although incremental, such solid contributions are typically accepted.
Minimal impact, repetitive, or superficial innovations
These papers often repackage existing knowledge or slightly adjust known models, datasets, or tasks without genuine novelty. They usually receive "borderline reject" scores, particularly in prestigious conferences like NeurIPS, unless impeccably executed and thorough.
Fundamental flaws and serious errors
Increasingly common are submissions containing fundamental methodological errors, misrepresented data, inadequate baselines, or unclear and contradictory explanations. Examples include using test sets as training sets or drastically misrepresenting performance metrics. Such papers typically receive immediate "borderline reject" ratings, with potential adjustments only if authors clarify concerns convincingly.
Beyond paper quality itself, ethical violations such as plagiarism or simultaneous submissions to multiple journals or conferences are grounds for automatic rejection, regardless of manuscript quality.
Summary Advice for New Reviewers:
- Clearly differentiate between groundbreaking and incremental contributions.
- Ensure thoroughness and accuracy, but stay open-minded about incremental advances.
- Remain vigilant for methodological rigor and clarity.
- Prioritize ethical compliance and rigorously check submissions against potential misconduct.
In the end, clear classification and objective judgment ensure both fair assessments and constructive feedback, benefiting the broader scientific community.
-
Thanks for sharing these thinking! I totally resonate with your points, especially about incremental research still being valuable. Not every paper can be paradigm-shifting, and recognizing solid, incremental progress helps keep science moving forward. Plus, the emphasis on methodological rigor and ethical considerations is spot-on. Peer review isn’t easy, but clear guidelines like these definitely make the process smoother ...
