- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper examines whether LLMs can function as reliable reviewers in academic peer review. To explore this, the authors develop a multi-round simulation framework called LLM-REVal, which models the interaction between a Research Agent—responsible for generating both human-like and LLM-authored papers—and a Review Agent that evaluates those submissions. The simulation reproduces the full review process, including initial assessment, rebuttals, revisions, and resubmission cycles, with a focus on fairness and bias in LLM-based reviewing.
The results reveal that LLM reviewers systematically favor the writing style characteristic of LLM-generated papers, leading to inflated scores for LLM-authored work. In contrast, some human-written papers—particularly those that discuss risks, fairness, or critical perspectives on AI—consistently receive lower scores, even after multiple revisions.
Critical strengths
- Novel Problem Formulation: Addresses a timely and underexplored issue—what happens when LLMs act as both researchers and reviewers, creating feedback loops in scientific workflows.
- Comprehensive Simulation Framework: The paper develops a realistic multi-agent system encompassing literature search, paper creation, feedback, rebuttal, revision, and meta-review. This end-to-end simulation is technically impressive.
- Clear Empirical Evidence of Bias: The study rigorously demonstrates two forms of bias in LLM reviewers:
- Linguistic bias toward LLM-style writing.
- Topic/framing bias against papers emphasizing risks or fairness.
Critical weaknesses
No critical weaknesses from my perspective. I reviewed this paper last year, and I am still supportive of it. I think the authors updated the paper following the guidance of other reviewers, so I believe we can consider accepting it.
Questions for authors
NA