- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper asks when an AI reviewer can reliably replace a human reviewer in peer review. The authors observe that AI–human disagreement is concentrated near the accept/reject decision boundary, while judgments farther from the boundary are generally more aligned with human recommendations. Based on this observation, they propose HESR, a selective review rule that lets AI occupy a reviewer slot only for judgments sufficiently far from the boundary and defers ambiguous cases to humans.
Critical strengths
The paper highlights an interesting empirical pattern in the studied setting: AI–human recommendation agreement is lower near the accept/reject boundary than in adjacent score ranges.
Critical weaknesses
- The evaluation is quite narrow, focusing primarily on recommendation agreement while leaving other important aspects of review quality unassessed. It remains unclear whether the selected AI reviews provide technically sound and well-supported critiques, reliably identify substantive flaws, or offer genuinely useful feedback to authors.
- The experimental protocol lacks sufficient transparency. Key details about the relationships among evaluation stages and the comparison procedures remain unclear, making it difficult to assess comparison fairness and the robustness of HESR’s reported transfer.
- The evidence for HESR’s generalizability remains insufficient. The fixed margin is an empirically chosen heuristic, and the overall evidence covers only 99 papers, including just 40 frozen-test papers, providing limited support for its reliability across broader submission populations and different review settings.
Questions for authors
- Please provide: (i) how each data block was sampled and used in method development and evaluation, any overlap between blocks, and the LOO procedure; (ii) implementation details, comparison settings, and per-block results for the alternatives in Sec. 4.1; and (iii) reviewer prompts and inference settings.
- Could the authors assess HESR’s sensitivity to the margin choice and validate the fixed rule on a larger, more diverse independent sample beyond the 99 papers studied, including the 40 frozen-test papers?
- For the subset selected by HESR, could the authors assess review quality beyond recommendation agreement and provide representative case studies of both successful and unsuccessful AI reviews, comparing them with human reviews in terms of factual correctness, technical soundness, and identification of substantive flaws?