- Rating
- 2(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper empirically investigates whether LLM evaluators’ preference for their own responses reflects superior response quality or harmful evaluation errors. Using verifiable benchmarks in mathematical reasoning, factual knowledge, and code generation, the authors distinguish legitimate self-preference from cases where a judge favors its own incorrect response over a correct alternative. Across multiple model families and scales, they find that stronger models generally exhibit more legitimate self-preference, but can also show a higher conditional tendency to favor themselves when their own responses are incorrect. The paper further examines these patterns on LMArena data and evaluates standard CoT prompting and distilled reasoning models as strategies for reducing harmful self-preference.
Critical strengths
Relevant research question. The paper investigates whether LLM self-preference reflects genuinely better responses or incorrect judgments in favor of the model’s own outputs. This distinction is practically relevant to LLM-based evaluation and self-improvement.
Systematic empirical evaluation. The study covers mathematical reasoning, factual knowledge, and code generation across multiple model families and scales. Using verifiable answers provides a stronger basis for assessing correctness than subjective preferences alone. The fixed evaluatee pool and response-order swapping also improve comparability and reduce position bias.
Useful descriptive findings. Separating legitimate self-preference from harmful self-preference highlights behavior that aggregate win rates can obscure. The same-model comparison between direct judgments and standard CoT provides useful evidence that additional reasoning can reduce erroneous self-favoring judgments.
Critical weaknesses
Limited incremental contribution relative to prior work. The distinction between response quality and self-preference bias has already been explicitly studied, notably in Beyond the Surface: Measuring Self-Preference in LLM Judgments [1]. Although this paper introduces verifiable benchmarks and conditional metrics, it does not sufficiently explain what substantially new understanding these additions provide beyond existing quality-aware analyses. So it is not appropriate to say "However, whether self-preference behavior is really harmful, or simply reflects the genuinely higher-quality outputs of stronger models, has been overlooked" in the abstract. Could the author reframe the words in the rebuttal phase?
The central interpretation is not adequately established. HSPP measures how often a model selects its own incorrect response over a correct alternative, but this does not by itself isolate self-preference bias. A model may reproduce the same misconception during generation and evaluation. Furthermore, stronger models may fail on systematically harder questions, making their conditional error subsets different from those of weaker models. Difficulty-controlled comparisons and independent judges evaluating the same response pairs would help distinguish these explanations.
The long-CoT results contain important confounders. Comparing instruction-tuned models with DeepSeek-R1-distilled models changes the model weights and training procedure, in addition to reasoning length and decoding settings. Consequently, the observed improvements cannot be attributed specifically to longer reasoning. The standard-CoT comparison offers cleaner evidence, but the broader inference-time scaling claim requires stronger controls.
Some conclusions are broader than the evidence supports. The correlation between model capability and harmful self-preference is weak on MBPP+ (r = 0.150), limiting the generality of the claimed trend. The LMArena extension also lacks a fixed query set and evaluatee pool across judges, and human preference labels should not be treated as equivalent to objective correctness.
[1] Chen, Zhi-Yuan, Hao Wang, Xinyu Zhang, Enrui Hu, and Yankai Lin. 2025. Beyond the Surface: Measuring Self-Preference in LLM Judgments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1653–1672. Association for Computational Linguistics.
Questions for authors
Could the authors move more results to main page but not in the appendix?
Additionally, For the reference: Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing, pp. 1–4, 2009.
“Pearson Correlation Coefficient” contains duplicate names. The correct author order is Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen, with each author listed once.