- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies rhetorical robustness in AI peer review, combining within-paper stability under content-preserving rewrites with between-paper discrimination to identify “false robustness” caused by score collapse. RobustReview comprises 1,260 versions of 60 ICLR 2026 submissions evaluated across 30 reviewer configurations. SciCore averages judgments from the full manuscript and an extracted, structured science core. In the primary GPT-5.5 evaluation, it improves the stability–discrimination profile while maintaining competitive human alignment. Validation includes content-fidelity audits, manuscript-averaging controls, repeated-review baselines, and paper-family bootstrap analyses.
Critical strengths
-
The paper identifies an important finding about AI peer review. AI review scores can change substantially with manuscript wording, even when rewrites are intended to preserve scientific content. This finding highlights sensitivity to presentation and has important implications for the reliability of AI reviewing.
-
The empirical study includes substantial coverage and useful controls. The benchmark spans multiple rewrite conditions, two rewrite producers, and several reviewer families. Human fidelity checks, paper-family bootstrap analyses, same-PDF repeatability tests, and manuscript-averaging baselines help assess alternative explanations for the observed results.
Critical weaknesses
-
Scientific-content equivalence and science-core fidelity are not sufficiently established.
Rewrites of novelty stance, scope, and evidence framing may change claim strength, qualifications, or argumentative meaning, making some score changes legitimate. Human audits are valuable but focus on tables, numbers, experiments, methods, and results, offering less direct validation of semantic equivalence. Comparisons with detected discrepancies remain included; analyses restricted to fidelity-verified samples would strengthen the conclusions. Similarly, embedding similarity does not establish whether science cores preserve critical assumptions, derivations, visual evidence, negative results, and qualifications. Direct factual and coverage audits could distinguish content normalization from stability caused by omitted information.
-
The study of rhetorical sensitivity is valuable, but specific rewrite effects remain underexplored.
Understanding how wording changes affect scores helps identify susceptibility to rhetorical manipulation. However, aggregate robustness results do not sufficiently reveal which rewrite types matter most, their direction, or how effects vary across models and paper-quality levels. Condition-specific score changes and confidence intervals for novelty emphasis, evidence framing, contribution salience, and linguistic changes, supported by representative cases, would make the findings more explanatory.
-
Deployment cost and compatibility with other review methods are insufficiently evaluated.
Scientific-judgment accuracy and review quality remain central to practical AI reviewing. As a safety and reliability intervention, SciCore should justify its additional cost and demonstrate compatibility with methods that improve reviewing performance. SciCore requires three calls: manuscript review, science-core extraction, and core review. Aggregate expenditure does not isolate per-manuscript cost, token consumption, or latency, and the useful three-call manuscript-averaging control does not ensure equal actual costs. Although different prompting protocols are examined, integration with retrieval, factual verification, or agentic systems is insufficiently evaluated. Separate baseline comparisons do not establish complementary gains, leaving SciCore’s cost–benefit tradeoff and orthogonality unclear.
Questions for authors
See Critical Weaknesses.
Dear reviewers,
Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.
Many thanks for your time and effort.