- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the reliability of LLM-based scientific reviewers under subtle, semantics-preserving changes to paper abstracts. The authors introduce TREAP, a framework covering six perturbation strategies across form, reasoning, and judgment levels, and show that static single-template evaluation can miss important vulnerabilities. They further propose SCOPE, an adaptive fuzzing method that searches for paper-specific perturbations which increase review scores, and demonstrate that it uncovers reviewer vulnerabilities more effectively than static and non-adaptive baselines.
Critical strengths
-
Broader perturbation coverage. TREAP evaluates six perturbation strategies across form, reasoning, and judgment levels rather than relying on a single perturbation type.
-
Adaptive evaluation. SCOPE turns reviewer robustness testing into an iterative fuzzing problem and consistently finds stronger perturbations than static or non-adaptive baselines.
Critical weaknesses
Limited evaluation scope. The multi-probe and fuzzing experiments use only 93 ICLR 2024 papers, which limits how broadly the findings can be generalized.
Questions for authors
Can you extend the evaluation to the full ICLR 2024 and 2025 paper sets to better assess whether the observed vulnerabilities generalize beyond the current subset?