acceptodds
Under review as a conference paper at ICLR 2027

Style over Substance? Linguistic Bias in LLM-Based Peer Review

Abstract

The number of paper submissions to major AI conferences has increased dramatically in recent years, substantially increasing the peer-review workload of researchers. At the same time, large language models (LLMs) are increasingly used to support peer review, both by individual reviewers and by conference organizations. While LLM capabilities have improved considerably, their evaluations may be influenced by linguistic characteristics that are unrelated to scientific content. In this work, we introduce a controlled experimental framework for studying such effects by systematically perturbing the linguistic style of research papers. Specifically, we rewrite papers to exhibit (i) linguistic features associated with native or non-native English writing, and (ii) high- or low-confidence linguistic features. To construct these interventions, we derive fine-grained rewriting instructions grounded in the linguistic literature and apply them using LLMs. Crucially, our interventions are designed to preserve the underlying scientific content. We then evaluate the resulting papers using different LLM reviewers. Our experiments expose a critical vulnerability in LLM-based peer review: reviewers systematically reward confident rhetoric independent of scientific content, raising serious concerns about the robustness of their assessments.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, linguistic bias, peer review

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XXQMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether LLM reviewers are affected by the writing style of a paper, and not only by its content. The authors propose a controlled experimental framework that changes the style of a paper but keeps the content. They take ICLR 2026 papers and use Claude Opus 5 to rewrite each paper in two directions, on two dimensions: native or non-native English writing, and high- or low-confidence language. Then six LLM reviewers review each version separately, and the authors compare the two versions of the same paper on scores, accept/reject decisions and review content. They find that expressed confidence has a strong and consistent effect on reviewer preferences, while effects of language proficiency are substantially smaller and vary across models. They conclude that LLM reviewers systematically reward confident rhetoric independent of scientific content.

Critical strengths
  1. Important problem. LLMs are now used a lot in paper reviewing, so it is important to know if they judge a paper by its content or by how it is written. The paper studies this in a controlled way. Each paper is rewritten into two versions that differ only in style, so the two versions of the same paper can be compared directly.

  2. Many analyses and detailed implementation. The paper does not only look at the overall score. It also looks at accept/reject flips, sub-scores, the arguments in the review text, the two styles combined, and a check with a prompt that asks reviewers to focus on style. This is done for six reviewers from different model families. The rewriting, reviewing and annotation prompts are given in full, together with the implementation details, which makes the work easier to reproduce.

Critical weaknesses
  1. No original paper as baseline. The paper only compares the two rewrites with each other and does not report scores for the original papers. This is a problem because the two rewrite directions do not change the text by the same amount. The high-confidence rewrite changes the hedge and booster counts only a little, but the low-confidence rewrite adds a lot of hedges. If the original papers get scores similar to the high-confidence versions, it may mean that hedges hurt the rating, but we cannot say that being confident helps. So I think the claim that reviewers "reward confident rhetoric" is too strong. The results only show that high-confidence versions score higher than low-confidence ones.

  2. The reviewed papers are LLM rewrites, and there is no human comparison. All reported scores are for LLM-rewritten versions, not for the papers as the authors wrote them. So the results are about how LLM reviewers judge LLM-rewritten text, and the claims about real submissions should be narrowed. Also, no human reviewer sees the two versions. Human reviewers may also prefer confident writing, and the paper itself cites work showing that writing style affects human review. So I don't think we can conclude that this is a vulnerability of LLM reviewers only.

  3. The native/non-native rewrites are too weak to support the "less bias" conclusion. The native and non-native versions are much closer to each other than the two confidence versions, since the proficiency metrics change only a little. The prompt also does not allow any grammar or spelling mistakes, and the paper does not check whether people would really read the text as non-native. So these rewrites may be quite different from real papers written by non-native authors. Because of this, the smaller effect on this dimension does not mean LLM reviewers have less bias on native/non-native writing.

Questions for authors
  1. Did you score the original papers and compare them with the rewrites? If yes, where do the original scores fall compared to the high- and low-confidence versions?
  2. Did you look at the results separately for accepted and rejected papers? This label could be another useful signal for "substance".
Reviewer_k2CMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the effect of linguistic style in LLM-based reviewing. Specifically, the authors introduce native/non-native and high-confidence/low-confidence rewriting, and use six different LLMs to review each rewritten paper. Through the evaluation, the authors report the existence of decision flipping and that LLMs favor high-confidence writing, but the effect of proficiency is small.

Critical strengths
  • Clear writing and easy to follow.
  • Interesting content and the evaluation design sounds.
Critical weaknesses
  • Compared with the original, low-confidence rewrites contain about nine times as many hedges, whereas high-confidence rewrites contain less than twice as many boosters. Accordingly, it is unclear whether LLM reviewers are penalizing hedges or rewarding boosters. The central claim requires reporting scores for the original papers and comparing each rewrite against them.
  • No human reviewer baseline. While I understand that the main purpose of this paper is to examine the decision shift of LLM reviewers, one may wonder whether the same phenomenon persists for human reviewers. In fact, human reviewers may show a similar tendency. I would like to recommend a small-scale human evaluation on a subset.
  • Are embedding similarity and TF-IDF useful metrics to figure out the content integrity against rewriting?
Questions for authors

See Weaknesses

Reviewer_1Wg1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a controlled stress-testing framework for linguistic bias in LLM-based peer review. The authors rewrite 504 anonymized ICLR 2026 papers along two dimensions: native- versus non-native-associated English and high- versus low-confidence language, while aiming to preserve scientific content. They validate the rewrites using lexical/linguistic metrics and embedding- and TF-IDF-based content-similarity measures, then obtain reviews from six LLM reviewers. The main finding is that language-proficiency rewrites have small and model-dependent effects on overall ratings, whereas high-confidence rewrites receive systematically higher ratings across all reviewers and can cross the paper-defined acceptance threshold more often. The paper further studies interactions between the two dimensions, prompt sensitivity, reviewer variance, criterion-level scores, and the aspects mentioned in generated reviews.

Critical strengths
  1. The paper addresses an important and timely reliability problem in LLM-assisted peer review.
  2. The paper provides an empirical analysis: it verifies that the intended linguistic features are successfully manipulated, quantifies reviewer variance, tests prompt sensitivity, examines interactions between the two linguistic dimensions, and analyzes both ratings and review content.
Critical weaknesses
  1. The paper only shows that, within its chosen LLM-reviewer setup, ratings are affected by linguistic style, particularly confident wording. However, it does not evaluate the alignment of these LLM reviewers with human reviews, real reviewer scores, or actual final decisions, nor does it calibrate their ratings against real acceptance probabilities. If these LLM reviewers are themselves unusable, for example, if they are highly inconsistent or misaligned with human reviewers, then findings built on them may have limited practical value or implications.
  2. Regarding whether the rewrites successfully preserve content while changing only expression, embedding similarity and TF-IDF similarity do not appear to be robust validation methods.
  3. It would be useful to compare the scores of high-confidence rewrites with those of the original papers, since low-confidence writing appears to be relatively uncommon
  4. The combined two-factor experiment lacks evaluation on strong models.
Questions for authors

n/a

Comments on the paper

1 comment
Area_Chair_uRN8OpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.