- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates the following question: When the methods and experimental results remain largely unchanged, can changing only how a paper is written lead AI reviewers to assign different scores? Compared with previous studies, this paper breaks down the established phenomenon of score manipulation into six rhetorical dimensions (Novelty stance, Evidence framing, Scope framing, Technical register, Contribution salience, Linguistic complexity). Through interventions in both positive and negative directions (for example, for Novelty stance, the positive rewrite is "more assertive and decisive statements of contributions supported by existing evidence", while the negative rewrite is "more cautious wording with more qualifiers"), it examines which dimensions elicit greater sensitivity and how this sensitivity varies across models and review protocols. Ultimately, the paper reports three findings: 1. What matters most is not complex vocabulary, but how evidence and novelty are presented. 2. More complex rewriting does not necessarily mean more effective rewriting. 3. Making AI stricter does not necessarily make it less susceptible to rhetoric.
Critical strengths
- The research question is important and addresses an ongoing issue of broad concern.
- The object of study closely reflects actual academic writing. The paper studies ordinary manuscript presentation rather than hidden instructions or special attack strings. This risk has practical significance.
- The experimental design is excellent. The bidirectional design can reveal directional sensitivity. Positive-direction rewriting alone would not distinguish the effects from random score fluctuations.
Critical weaknesses
-
Related Work should be moved from Section 6 to Section 2. With the page limit increased to 10 pages, the Related Work in the main text should also be substantially expanded, particularly the studies in Appendix A.3 that are most closely related to this paper (Paraphrasing Adversarial Attack, Stop Automating Peer Review Without Rigorous Evaluation, No Hidden Prompts Needed, etc.). This is because the paper builds heavily on the prior finding that "rewriting papers can influence AI review scores" and investigates the effects of six rhetorical dimensions on that basis. Therefore, the related work should be discussed in detail.
-
This paper focuses on categorizing rhetorical dimensions in AI review, but lacks citations to previous work categorizing the risks of AI review (for example, [1] AI Review Is a Systemic Risk to Peer Review: Toward a Blockchain-Supported Claim-Level Ledger for Accountability, ICML 2026). The introduction and related work should provide a more comprehensive discussion of, and citations concerning, the risks that AI reviewers may pose [1]. Citing and describing [1] in the introduction and related work would help establish the importance of this work: AI reviewers may pose risks, and this work investigates this issue through rhetorical dimensions.
-
The contribution of this paper is somewhat incremental. Previous studies have already established the central finding that "rewriting papers can influence AI review scores". The authors' three new conclusions do not offer me new insights. These conclusions are ones I could already have anticipated, and I do not find them surprising or exciting. The authors do not conduct further attribution analysis. Therefore, I would like to see how the paper's conclusions could become deeper and more interpretable, rather than remaining surface-level observations. Even an experimental/analytical plan included in the discussion could make this paper more meaningful.
-
The paper does not distinguish between "unreasonable rhetorical preferences" and "legitimate improvements in presentation".
-
The study covers only one conference and does not include conferences or journals outside artificial intelligence.
-
There is no human-reviewer control group. How would human reviewers respond to the same rewrites? If human reviewers behave in the same way as AI reviewers, then this is not reward hacking, but ordinary academic persuasion.
-
The conclusions are mainly supported by the two worst-calibrated reviewers. Gemini 3.5 FL is the worst, yet several of the largest values reported in the paper come from Gemini 3.5 FL (+0.93, -0.73, +0.775).
-
The experiments do not test the scenario of "simply adding a prompt instructing AI reviewers to ignore these six dimensions when reviewing".
-
The experiments evaluate only models, not review agents.
I will decide whether to raise or lower my score based on the authors' rebuttal.
Questions for authors
- This paper reports several findings. However, as models/agents improve, could these findings quickly cease to hold?
- Could simply adding a prompt instructing AI reviewers to ignore these six dimensions (treating different rhetorical styles equally) completely resolve this issue?
- This paper evaluates models. Could review agents exhibit different patterns?
Dear Reviewer Puxn,
Could you please read our response and briefly share whether your main concerns have been addressed? With the author response period ending on November 18 (AoE), we would greatly appreciate your follow-up before then. Thank you for your time and feedback.