- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether LLM reviewers are affected by the writing style of a paper, and not only by its content. The authors propose a controlled experimental framework that changes the style of a paper but keeps the content. They take ICLR 2026 papers and use Claude Opus 5 to rewrite each paper in two directions, on two dimensions: native or non-native English writing, and high- or low-confidence language. Then six LLM reviewers review each version separately, and the authors compare the two versions of the same paper on scores, accept/reject decisions and review content. They find that expressed confidence has a strong and consistent effect on reviewer preferences, while effects of language proficiency are substantially smaller and vary across models. They conclude that LLM reviewers systematically reward confident rhetoric independent of scientific content.
Critical strengths
-
Important problem. LLMs are now used a lot in paper reviewing, so it is important to know if they judge a paper by its content or by how it is written. The paper studies this in a controlled way. Each paper is rewritten into two versions that differ only in style, so the two versions of the same paper can be compared directly.
-
Many analyses and detailed implementation. The paper does not only look at the overall score. It also looks at accept/reject flips, sub-scores, the arguments in the review text, the two styles combined, and a check with a prompt that asks reviewers to focus on style. This is done for six reviewers from different model families. The rewriting, reviewing and annotation prompts are given in full, together with the implementation details, which makes the work easier to reproduce.
Critical weaknesses
-
No original paper as baseline. The paper only compares the two rewrites with each other and does not report scores for the original papers. This is a problem because the two rewrite directions do not change the text by the same amount. The high-confidence rewrite changes the hedge and booster counts only a little, but the low-confidence rewrite adds a lot of hedges. If the original papers get scores similar to the high-confidence versions, it may mean that hedges hurt the rating, but we cannot say that being confident helps. So I think the claim that reviewers "reward confident rhetoric" is too strong. The results only show that high-confidence versions score higher than low-confidence ones.
-
The reviewed papers are LLM rewrites, and there is no human comparison. All reported scores are for LLM-rewritten versions, not for the papers as the authors wrote them. So the results are about how LLM reviewers judge LLM-rewritten text, and the claims about real submissions should be narrowed. Also, no human reviewer sees the two versions. Human reviewers may also prefer confident writing, and the paper itself cites work showing that writing style affects human review. So I don't think we can conclude that this is a vulnerability of LLM reviewers only.
-
The native/non-native rewrites are too weak to support the "less bias" conclusion. The native and non-native versions are much closer to each other than the two confidence versions, since the proficiency metrics change only a little. The prompt also does not allow any grammar or spelling mistakes, and the paper does not check whether people would really read the text as non-native. So these rewrites may be quite different from real papers written by non-native authors. Because of this, the smaller effect on this dimension does not mean LLM reviewers have less bias on native/non-native writing.
Questions for authors
- Did you score the original papers and compare them with the rewrites? If yes, where do the original scores fall compared to the high- and low-confidence versions?
- Did you look at the results separately for accepted and rejected papers? This label could be another useful signal for "substance".
Please review the authors’ responses and join the discussion
Dear Reviewers,
The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.
Thank you for your time and for contributing to the discussion.
Best regards,
Your AC