acceptodds
Under review as a conference paper at ICLR 2027

A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

Abstract

AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Peer Review, Rhetorical Robustness, LLM-as-a-Judge, Scientific Evaluation, Benchmarking, Content-Normalized Review

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GWwHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies rhetorical robustness in AI peer review, combining within-paper stability under content-preserving rewrites with between-paper discrimination to identify “false robustness” caused by score collapse. RobustReview comprises 1,260 versions of 60 ICLR 2026 submissions evaluated across 30 reviewer configurations. SciCore averages judgments from the full manuscript and an extracted, structured science core. In the primary GPT-5.5 evaluation, it improves the stability–discrimination profile while maintaining competitive human alignment. Validation includes content-fidelity audits, manuscript-averaging controls, repeated-review baselines, and paper-family bootstrap analyses.

Critical strengths
  1. The paper identifies an important finding about AI peer review. AI review scores can change substantially with manuscript wording, even when rewrites are intended to preserve scientific content. This finding highlights sensitivity to presentation and has important implications for the reliability of AI reviewing.

  2. The empirical study includes substantial coverage and useful controls. The benchmark spans multiple rewrite conditions, two rewrite producers, and several reviewer families. Human fidelity checks, paper-family bootstrap analyses, same-PDF repeatability tests, and manuscript-averaging baselines help assess alternative explanations for the observed results.

Critical weaknesses
  1. Scientific-content equivalence and science-core fidelity are not sufficiently established.

    Rewrites of novelty stance, scope, and evidence framing may change claim strength, qualifications, or argumentative meaning, making some score changes legitimate. Human audits are valuable but focus on tables, numbers, experiments, methods, and results, offering less direct validation of semantic equivalence. Comparisons with detected discrepancies remain included; analyses restricted to fidelity-verified samples would strengthen the conclusions. Similarly, embedding similarity does not establish whether science cores preserve critical assumptions, derivations, visual evidence, negative results, and qualifications. Direct factual and coverage audits could distinguish content normalization from stability caused by omitted information.

  2. The study of rhetorical sensitivity is valuable, but specific rewrite effects remain underexplored.

    Understanding how wording changes affect scores helps identify susceptibility to rhetorical manipulation. However, aggregate robustness results do not sufficiently reveal which rewrite types matter most, their direction, or how effects vary across models and paper-quality levels. Condition-specific score changes and confidence intervals for novelty emphasis, evidence framing, contribution salience, and linguistic changes, supported by representative cases, would make the findings more explanatory.

  3. Deployment cost and compatibility with other review methods are insufficiently evaluated.

    Scientific-judgment accuracy and review quality remain central to practical AI reviewing. As a safety and reliability intervention, SciCore should justify its additional cost and demonstrate compatibility with methods that improve reviewing performance. SciCore requires three calls: manuscript review, science-core extraction, and core review. Aggregate expenditure does not isolate per-manuscript cost, token consumption, or latency, and the useful three-call manuscript-averaging control does not ensure equal actual costs. Although different prompting protocols are examined, integration with retrieval, factual verification, or agentic systems is insufficiently evaluated. Separate baseline comparisons do not establish complementary gains, leaving SciCore’s cost–benefit tradeoff and orthogonality unclear.

Questions for authors

See Critical Weaknesses.

Reviewer_okaKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies rhetorical robustness in AI peer review: scores should remain stable when the same science is expressed differently, while retaining differences across papers. RobustReview contains 60 original papers and 1,200 rewrites, evaluated with 30 reviewer configurations. SciCore averages a Strict full-manuscript rating with a review of an extracted science core. The main GPT-5.5 experiments show improved robustness over individual manuscript reviewers and some manuscript ensembles, while retaining competitive human alignment.

Critical strengths
  1. A useful evaluation perspective: The paper shows that low score variation can arise because different papers receive nearly identical ratings. Separately evaluating stability, cross-paper discrimination, and human alignment helps avoid this misleading conclusion.
  2. Meaningful ensemble controls: The authors compare against ordinary manuscript-score averages rather than only individual reviews. With the Strict branch fixed, SciCore improves all five robustness metrics over adding a Standard or Persistent branch, supported by paired intervals.
Critical weaknesses
  1. The science core needs a direct check for missing information: Neither of Table 3 or Table 10 directly checks whether the core leaves out important assumptions, results, or limitations. The authors account for extraction uncertainty, but do not report such a check. It therefore remains unclear whether the more stable scores come at the cost of missing information.
  2. The added benefit over strong manuscript ensembles is still unclear: Table 11 shows some additional benefit from SciCore. However, compared with averaging three manuscript reviews, the gains are modest and mixed: SciCore is better on some metrics, but slightly worse on ICC and Drift SD. The paper needs to clarify when the extra extraction step is worthwhile.
Questions for authors
  1. Did you check whether the science cores omit important assumptions, results, or limitations? If so, please report the findings separately from the rewrite-fidelity checks in Table 10.
  2. Could you provide confidence intervals for this comparison in Table 11, along with the actual cost per paper for both methods? This would help assess whether the relatively small gains justify the extra step.
  3. Which PDFs, and how many, were included in Appendix F.3? Did each SciCore repetition extract a new science core, or reuse an existing one?
Reviewer_qmh7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RobustReview, a benchmark with 60 ICLR 2026 papers and 1,200 rhetorically rewritten variants, and evaluates 30 reviewer configurations. The experiments show that score stability under rewriting does not necessarily mean that a reviewer is robust, since some systems are stable simply because they do not distinguish papers well. The authors also propose SciCore, which combines a full-paper review with a separate review of the extracted science core. In the main GPT-5.5 experiments, SciCore improves the robustness metrics with similar human alignment.

Critical strengths
  1. Measuring score stability alone can be misleading when a reviewer assigns similar scores across different papers. Evaluating both within-paper stability and between-paper discrimination is therefore reasonable.
  2. RobustReview evaluates rhetorical robustness at the full-manuscript level across multiple rewrite conditions and reviewer systems. The experiments cover 1,260 manuscript versions and 30 reviewer configurations.
Critical weaknesses
  1. The rewritten papers are intended to preserve the original scientific content, but the fidelity check still finds some mismatches. Since all variants are kept in the benchmark, it is unclear how much these mismatches affect the robustness results.
  2. The full SciCore system is mainly tested with GPT-5.5. The cross-backbone experiment only evaluates the Core-Adapted branch, so it is unclear whether the same gains hold for the full method with other backbones.
Questions for authors
  1. How many of the 1,200 rewritten variants are flagged as having at least one technical-content mismatch? If these variants are removed, do the main reviewer rankings and SciCore results remain similar?
  2. Have you evaluated the complete SciCore system with GPT-5-mini or GLM-5.2? Does the dual-branch fusion still improve the robustness metrics for these backbones?

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.