acceptodds
Under review as a conference paper at ICLR 2027

How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review

Abstract

As large language models increasingly participate in scientific evaluation, we investigate how rhetorical rewriting can reward-hack AI reviewers when reported scientific content is largely preserved. We construct 4,200 full-paper manuscripts from 120 anonymized ICLR 2026 submissions. Two LLM rewriters modify six rhetorical dimensions in opposing directions under scientific-content preservation constraints, and five LLM reviewers assess the manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, followed by scope framing, with this hierarchy persisting across human-rating ranges. More elaborate rewriting does not reliably yield larger gains: joint-rewrite gains depend strongly on the rewriter, reviewer guidance does not consistently outperform an unguided second pass, and recursive gains vary across configurations. Strict review lowers mean overall assessment by 1.36 points, but positive-negative rhetorical contrasts persist. These findings reveal selective, configuration-dependent vulnerabilities in AI scientific review and motivate evaluation systems robust to rhetorical variation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI-based Peer Review, LLM-as-a-Judge, Rhetorical Sensitivity, Reward Hacking, Scientific Evaluation, Robustness

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.40 over 5 reviews, above 68% of the others.
Reviewer_PuxnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the following question: When the methods and experimental results remain largely unchanged, can changing only how a paper is written lead AI reviewers to assign different scores? Compared with previous studies, this paper breaks down the established phenomenon of score manipulation into six rhetorical dimensions (Novelty stance, Evidence framing, Scope framing, Technical register, Contribution salience, Linguistic complexity). Through interventions in both positive and negative directions (for example, for Novelty stance, the positive rewrite is "more assertive and decisive statements of contributions supported by existing evidence", while the negative rewrite is "more cautious wording with more qualifiers"), it examines which dimensions elicit greater sensitivity and how this sensitivity varies across models and review protocols. Ultimately, the paper reports three findings: 1. What matters most is not complex vocabulary, but how evidence and novelty are presented. 2. More complex rewriting does not necessarily mean more effective rewriting. 3. Making AI stricter does not necessarily make it less susceptible to rhetoric.

Critical strengths
  1. The research question is important and addresses an ongoing issue of broad concern.
  2. The object of study closely reflects actual academic writing. The paper studies ordinary manuscript presentation rather than hidden instructions or special attack strings. This risk has practical significance.
  3. The experimental design is excellent. The bidirectional design can reveal directional sensitivity. Positive-direction rewriting alone would not distinguish the effects from random score fluctuations.
Critical weaknesses
  1. Related Work should be moved from Section 6 to Section 2. With the page limit increased to 10 pages, the Related Work in the main text should also be substantially expanded, particularly the studies in Appendix A.3 that are most closely related to this paper (Paraphrasing Adversarial Attack, Stop Automating Peer Review Without Rigorous Evaluation, No Hidden Prompts Needed, etc.). This is because the paper builds heavily on the prior finding that "rewriting papers can influence AI review scores" and investigates the effects of six rhetorical dimensions on that basis. Therefore, the related work should be discussed in detail.

  2. This paper focuses on categorizing rhetorical dimensions in AI review, but lacks citations to previous work categorizing the risks of AI review (for example, [1] AI Review Is a Systemic Risk to Peer Review: Toward a Blockchain-Supported Claim-Level Ledger for Accountability, ICML 2026). The introduction and related work should provide a more comprehensive discussion of, and citations concerning, the risks that AI reviewers may pose [1]. Citing and describing [1] in the introduction and related work would help establish the importance of this work: AI reviewers may pose risks, and this work investigates this issue through rhetorical dimensions.

  3. The contribution of this paper is somewhat incremental. Previous studies have already established the central finding that "rewriting papers can influence AI review scores". The authors' three new conclusions do not offer me new insights. These conclusions are ones I could already have anticipated, and I do not find them surprising or exciting. The authors do not conduct further attribution analysis. Therefore, I would like to see how the paper's conclusions could become deeper and more interpretable, rather than remaining surface-level observations. Even an experimental/analytical plan included in the discussion could make this paper more meaningful.

  4. The paper does not distinguish between "unreasonable rhetorical preferences" and "legitimate improvements in presentation".

  5. The study covers only one conference and does not include conferences or journals outside artificial intelligence.

  6. There is no human-reviewer control group. How would human reviewers respond to the same rewrites? If human reviewers behave in the same way as AI reviewers, then this is not reward hacking, but ordinary academic persuasion.

  7. The conclusions are mainly supported by the two worst-calibrated reviewers. Gemini 3.5 FL is the worst, yet several of the largest values reported in the paper come from Gemini 3.5 FL (+0.93, -0.73, +0.775).

  8. The experiments do not test the scenario of "simply adding a prompt instructing AI reviewers to ignore these six dimensions when reviewing".

  9. The experiments evaluate only models, not review agents.

I will decide whether to raise or lower my score based on the authors' rebuttal.

Questions for authors
  1. This paper reports several findings. However, as models/agents improve, could these findings quickly cease to hold?
  2. Could simply adding a prompt instructing AI reviewers to ignore these six dimensions (treating different rhetorical styles equally) completely resolve this issue?
  3. This paper evaluates models. Could review agents exhibit different patterns?
1 comment

Dear Reviewer Puxn,

Could you please read our response and briefly share whether your main concerns have been addressed? With the author response period ending on November 18 (AoE), we would greatly appreciate your follow-up before then. Thank you for your time and feedback.

Reviewer_5oRROpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper rewrites 120 manuscripts along six rhetorical dimensions using two rewriting models and evaluates the resulting papers with five AI reviewers. Novelty framing and evidence emphasis produce larger score changes than surface-level linguistic sophistication. Additional experiments investigate joint interventions, repeated editing, review-guided revision, and stricter reviewing instructions.

Critical strengths

The paired, bidirectional interventions support informative comparisons. Paper-level analysis, fidelity audits, length controls, and repeated-review checks strengthen the empirical evidence. The results provide a useful characterization of which rhetorical changes affect AI reviewers.

Critical weaknesses

The related-work discussion omits several important and closely relevant studies. Examples include:

Hwang et al. (2025), Can You Trick the Grader? Adversarial Persuasion of LLM Judges, which studies persuasion-induced score inflation, combined rhetorical strategies, and counter-prompting defenses.

Feuer et al. (2025), Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking, which separately manipulates response style and factual correctness to examine evaluator biases.

Wu and Aji (2025), Style Over Substance: Evaluation Biases for Large Language Models, which demonstrates how presentation quality can outweigh factual accuracy in evaluation.

These studies address central aspects of the paper’s motivation and experimental logic. Their omission leaves the discussion of established rhetorical and stylistic evaluation vulnerabilities incomplete and weakens the positioning of the contribution.

Some revisions may legitimately improve exposition or communicate existing evidence more effectively. Moreover, rewrites flagged for scientific-fidelity violations remain in the analysis, complicating attribution of score changes solely to rhetoric.

The scale of 120 papers is too tiny and toy. Although the study generates 4,200 manuscript versions, these originate from only 120 ICLR 2026 papers, with 20 papers per human-rating stratum. The paired design supports within-paper comparisons, but repeated rewriting and evaluation do not expand the diversity of the underlying research. Consequently, the findings provide limited evidence about generalization across venues, disciplines, and manuscript types, and the rating-stratified results should be interpreted cautiously.

Questions for authors

How does the study extend or qualify the findings of the omitted work, particularly beyond transferring established vulnerabilities to full-manuscript review? Do the main effects persist after excluding rewrites flagged for scientific-fidelity violations?

can you extend the analysis to at least 5k paper scale?

1 comment

Dear Reviewer 5oRR,

We wanted to follow up on our response before the author replies close on November 18 (AoE). We would appreciate it if you could take a look and briefly let us know whether it addresses your main concerns. Thank you again for your time and effort.

Reviewer_uLGqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how rhetorical rewriting of manuscripts affects AI-reviewer judgments. The authors sample 120 ICLR 2026 submissions and vary the rewriting models, strategies, and directions while aiming to preserve the underlying methods and experimental content. The findings show that some rhetorical dimensions affect scores more strongly than others, that some rewriting models are more effective at increasing scores, and that the effectiveness of different rewriting strategies depends on the configuration.

Critical strengths
  1. The paper attempts to disentangle the quality of the underlying scientific work from rhetorical presentation by comparing rewritten versions of the same manuscript. This provides a more controlled basis for evaluating rhetorical effects.

  2. The experiments are comprehensive and systematic, covering multiple rhetorical dimensions, rewriting models, rewriting strategies, and AI-reviewer configurations.

Critical weaknesses
  1. The distinction between improved manuscript quality and reward hacking remains insufficiently established. Although the paper emphasizes separating scientific quality from rhetorical rewriting, preserving the same methods and experiments does not necessarily preserve manuscript quality. Indeed, these aspects may be difficult to separate fully: better discussion and analysis of the same experimental results can provide additional insight and reasonably warrant higher scores. Such improvements should not automatically be considered reward hacking. The evidence therefore supports sensitivity to presentation more directly than the reward-hacking interpretation.

  2. Differences in intervention strength complicate the interpretation of the dimension ordering. A main finding concerns the relative effects of different rhetorical dimensions and configurations. However, rewrites may vary in how strongly they modify each dimension. It's better to also measure or account for these factors.

  3. The implications for mitigating presentation sensitivity remain underdeveloped. The paper examines how presentation affects AI-reviewer judgments but provides limited insight into how undesirable sensitivity might be reduced. More fundamentally, it remains unclear which forms of sensitivity should be reduced: a well-written paper with better analysis and clearer framing may deserve a higher score. Although this might be an open-ended issue, a fuller discussion of the boundary between legitimate responsiveness to writing quality and undesirable rhetorical influence would strengthen the paper.

Questions for authors
  1. How does the study distinguish improvements in scientific writing and argumentation from reward hacking? In particular, what evidence establishes that the observed score increases are unjustified, rather than reflecting clearer communication or better analysis of the same results?

  2. Was the magnitude of the intended change assessed for each rhetorical dimension? If so, does the reported ordering persist after accounting for differences in intervention strength?

  3. Which forms of presentation sensitivity do the authors consider undesirable, and what do the findings suggest about reducing those effects while retaining appropriate sensitivity to clarity and argument quality?

Reviewer_P8jmOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

As AI reviewers are being used increasingly in peer review, conditionally or unconditionally, questions arise about how AI reviewers score a paper when it is written in ways that are more or less different from what was originally intended. This paper takes 120 submissions from ICLR 2026 and asks two frontier models to rewrite each one along the six dimensions of novelty, evidence, scope, technical register, contribution, and linguistic complexity. Keeping the underlying science the same, the paper conducts around 4,080 rewrites. Five reviewer models then score those rewrites under two versions of the prompt: strict or standard. They do so to understand how rewrites might affect how the papers are perceived by a language model. Five human annotators annotate 300 pairs of the reviews. The core finding shown here is that scores move the most with better novelty, evidence, or scope, whereas hedging or being too cautious can result in penalization from AI reviewers. However, the paper does not show whether the score changes themselves are reward hacking or a correct reading of a more or less cautious paper, although the title suggests otherwise.

Critical strengths

Strength 1: Clean design and presentation. The design rewrites the same paper in both directions and keeps every outcomes. This shows which kinds of framing move AI scores and which do not.

Strength 2: The study is large and openly reported with costs and such. It uses five reviewers from four providers, two prompts, strata by human rating, and a human fidelity audit. It also missing reviews, prompts and a repeated-sampling check. This makes the main contrasts easy to trust as a description of how these models behave.

Critical weaknesses

Weakness 1: The paper runs many configurations but skips the checks that decide what the effect means in hindsight. Here, no human rates any rewrite, and the fidelity audit does not check claim strength or scope. A score drop might be a correct reading of a more cautious paper or reverse be a proper calibration when a score rises. A small human rating of sampled positive and negative pairs would separate reward hacking from legitimate sensitivity. This is the main reason my rating is not higher, since the title and framing rest on this difference.

Weakness 2: There is no plain rewrite to compare against. In Table 3, the gains and drops are measured against the original paper, which a human wrote, while every rewrite is written by an LLM. So part of each score change may come from the LLM rewriting itself, not from the positive or negative framing. Asking the LLM to rewrite each paper once without any framing goal would separate the two. I'd like to see the main claims rest on that comparison.

Questions for authors

Clarification:

  1. Do the contrasts hold when the pairs flagged by the auditor or the annotators are excluded? What is the agreement between the auditor and the human majority vote?
  2. Which reviewer models received the native PDF, and which received OpenRouter's parsed text?
  3. Please refer to weakness #1, I'm open to changing my rating if W1 is answered/considered promptly.

Suggestion:

  1. The reviews already contain written strengths and weaknesses. Coding them would show whether positive novelty and scope rewrites draw "overclaiming" comments, which would explain the mechanism.
  2. Please commit to releasing the rewrites, reviews and code as that isn't as clear as I'd have hoped for.
  3. The following related work is not cited and would be worth discussing:
    • Jiang et al., "BadScientist: Can a Research Agent Write Convincing but Unsound Papers that Fool LLM Reviewers?", ACL 2026 (arXiv:2510.18003).
    • Hatzel et al., "Review Arcade: On the Human Alignment and Gameability of LLM Reviews", arXiv:2605.28897.
    • Li et al., "LLM-REVal: Can We Trust LLM Reviewers Yet?", arXiv:2510.12367.
Reviewer_v2YiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Previous papers have uncovered that rhetorical rewriting of scientific manuscripts can “reward-hack” AI reviewers while largely preserving the scientific content. This paper fine-grained studies the phenomenon through a systematic rewrite-and-review pipeline. Based on 120 anonymized ICLR 2026 submissions, the authors generate 4,080 rewrites. The study covers positive and negative rewrites along six rhetorical dimensions: novelty stance, scope framing, evidence framing, contribution salience, technical register, and linguistic complexity. They also examine three enhanced workflows: combining all six dimensions in one rewrite, repeating the joint rewrite for three turns, and using AI reviewer feedback to guide further rewriting. Five AI reviewer models evaluate the manuscripts under standard and strict protocols. The authors use paired comparisons to measure score changes. The authors find that evidence framing and novelty stance produce the largest positive–negative differences in overall assessment, followed by scope framing. Stricter review lowers absolute scores but does not eliminate rhetorical sensitivity. More complex rewriting workflows do not consistently produce larger gains. Their effects depend on the rewriter, reviewer, and review protocol.

Critical strengths

S1. This paper considers an important research question. As fast development of AI Reviewers in scientific peer review, understanding their sensitivity to rhetorical changes is important. This paper compares score changes across rhetorical interventions and review conditions, providing useful evidence for assessing the robustness of AI reviewers. It extends previous work through a more fine-grained comparison of which rhetorical dimensions affect review scores. S2. The experiments provide broad coverage and include human validations. The paper combines paired comparisons of positive and negative rewrites across six rhetorical dimensions, multiple rewriter and reviewer models, and different review protocols. This allows a systematic examination of how rhetorical effects vary across conditions. The authors also use an independent model audit and human checks to assess whether scientific content is preserved, which strengthens confidence in the results.

Critical weaknesses

W1. Further experiments are needed to clarify what causes the score changes. W1.1. Are the rewrite effects stable across repeated generations? For each paper, dimension, and direction, each rewriter model generates only one version. Different generations may vary in wording and how strongly they change the target dimension. I suggest generating multiple independent rewrites under the same conditions for a small subset of papers and checking whether the score changes and main conclusions remain stable. W1.2. A neutral rewrite control would be useful. The paper compares the original manuscripts with positive and negative rewrites. An additional control could change the wording while keeping the target rhetorical stance largely unchanged. For example, novelty claims could be rephrased without making them stronger or weaker. This would help distinguish the effect of changing the stance from that of general polishing or clearer writing. W1.3. Changes in other dimensions should be checked. The authors acknowledge that the six dimensions may overlap, but do not measure how much they change together. For example, does strengthening novelty claims also increase contribution salience? Human checks could assess how often this happens and how large these changes are. If other dimensions also change substantially, it becomes difficult to attribute score changes to the target dimension or interpret the ranking of dimensional effects.

W2. The paper’s contribution and novelty need further clarification. Previous studies have already examined how specific forms of expression affect model judgments. For example, EMBER [1] studies certainty and uncertainty markers in general LLM evaluation. In scientific evaluation, TitleTrap [2] compares title styles, while When AI reviews science [3] compares cautious, neutral, and bold wording and finds that cautious wording is penalized. The authors should explain more clearly what new insights their six-dimensional, bidirectional analysis provides beyond these earlier findings. Testing whether these insights can help reduce the influence of rhetoric on review scores would further strengthen the paper’s practical value.

[1] Lee, Dongryeol et al. “Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation.” North American Chapter of the Association for Computational Linguistics (2024). [2] Du, Shurui. “TitleTrap: Probing Presentation Bias in LLM-Based Scientific Reviewing.” Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems (2025): n. pag. [3] Wang J., Liu Y., Xu H., et al. (2026). When AI reviews science: Can we trust the referee? The Innovation Informatics 2:100030.

Questions for authors

Q1. Rewrite stability: Have you tested multiple independently generated rewrites for the same paper, dimension, direction, and rewriter? If so, how much do the score changes vary across generations, and does the ranking of dimensional effects remain stable?

Q2. Neutral rewrite control: Beyond the original baseline and positive–negative comparisons, what evidence separates changes caused by the target rhetorical stance from changes caused by general polishing or improved clarity? Have you evaluated neutral rewrites that change wording while largely preserving the target stance?

Q3. Cross-dimensional changes: Have you measured whether an intervention also changes non-target dimensions? For example, how often does stronger novelty framing also increase contribution salience? How might these changes affect the interpretation of the dimensional ranking?

1 comment

Dear Reviewer v2Yi,

As we approach the author response deadline on November 18 (AoE), we would be grateful if you could review our response and briefly let us know whether your main concerns have been addressed. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.