acceptodds
Under review as a conference paper at ICLR 2027

When Do LLM Judges Favor Themselves? Dissociating Self-Recognition from Self-Preference

Abstract

LLM-as-a-judge has become a standard evaluation paradigm, yet language models can exhibit self-preference, favoring their own outputs over those of other models. Existing explanations attribute this effect to self-recognition, familiarity, or genuine quality differences, but these factors have rarely been isolated within a unified controlled setting. We re-examine self-preference under stricter controls by fixing candidate pairs to identical incorrect answers and clustering inference at the question level. Across seven judges and four tasks, we find that self-preference persists, but is substantially smaller and more task-dependent than conventional pairwise analyses suggest. The effect increases with model scale and is strongest on tasks with greater expressive latitude, reaching +0.24 on MMLU, whereas it vanishes or reverses on verifiable numerical tasks. Our linear-probe experiments recover self-authorship with near-perfect accuracy across models from 1B to 72B parameters, even after surface cues are neutralized. Crucially, however, self-recognition and familiarity remain strong even when self-preference disappears, showing that neither is sufficient to explain the bias. After controlling for judge capability, the residual self-specific effect is only about two percentage points, yet excluding self-votes can still change the top-ranked model. These results show that self-preference is real but modest, strongly task-dependent, not reducible to surface features, and potentially consequential for model evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, self-preference bias, self-recognition, model evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fnTjOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores how self-recognition signals can persist without self-preference in behavioural analysis. Previous literature in this domain often attributed self-recognition to self-preference. Alternative explanations for self-preference included familiarity and quality.

In their method, the paper controls for 'shared-belief' by fixing a wrong answer from both the judge and the evaluatee. They find that, in this setting, self-preference exists only in some tasks. To test for recognition, they use a linear probe.

To assess surface features, the paper rewrites the judge’s own responses to remove four kinds of signal: style, characteristic vocabulary, reasoning structure, and alignment-style phrasing like hedges and disclaimers. They test all 16 combinations and attribute B₀ with exact Shapley values. On MMLU, vocabulary contributes most and style almost nothing. The part that survives all four rewrites (ρ) is the largest component for every judge with a positive residual.

Critical strengths

The work decomposes self-preference signals using Shapley values. This enables us to study different surface cues and their impact on self-preference. This method is neat, as it handles the overlapping impact of signals in a principled way. The unexplained part is measured, allowing the work to test the surface feature hypothesis.

Critical weaknesses

The matched 'proxy' is used based on the incorrect answer to the same question. The paper argues that this eliminates agreement on conclusions or answer quality. However, an open-ended response that yields the same incorrect answer may reflect different philosophies or reasoning, and that reasoning may also reflect quality.

The technical rigour of the probe design is underspecified. It is not convincing that the probe measures some confound in the data. Recognition may be a function of data attributes that the internal layer captures.

The paper focuses heavily on equations and tables, which makes the main takeaway harder to grasp. One recommendation is to focus on the intuition behind the perturbations and on plots of the residual decomposition for clarity.

Questions for authors

Please see the critical weaknesses section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fnTj

We thank the reviewer for the detailed and constructive review. Your positive view of the Shapley decomposition is very encouraging, and all three points have helped us improve the paper.

W1. The same incorrect answer may still reflect different reasoning.

Thank you for pointing this out. We agree. Fixing the same incorrect answer removes agreement at the level of the conclusion, but the two responses can still reason differently. To exclude the influence of reasoning as far as possible, we ran a new crossed experiment for Qwen2.5-32B on MMLU. The judge's argument and the proxy's argument are each written once by the judge J and once by the proxy K, so that reasoning and writing vary separately.

Written by JWritten by KWriter effect
J's argument0.3910.193+0.198
K's argument0.3520.158+0.194
Argument effect+0.039+0.035

Main effects (95% CI half-width): writer +0.196 ± .045, argument +0.037 ± .031.

As you expected, reasoning does contribute. Changing the argument shifts the preference by +0.037. With the reasoning held fixed, however, changing the writer shifts it by +0.196. This suggests that the residual may depend more on how the response itself is written than on its reasoning.

W2. Probe design and confounds.

Thank you for raising this. We agree that the original description of the probe was too brief, and we have added the full setup to the appendix for reproducibility. In short, for each judge we take the residual-stream representation at the last token of the response and train an L2-regularized logistic-regression probe to separate the judge's own response o_J from the matched proxy response o_K to the same question, with 5-fold cross-validation grouped by question.

This design already includes several controls. Since o_J and o_K reach the same incorrect answer, the probe can hardly rely on the content of the answer. Since the folds are grouped by question, no question appears in both training and test, so the probe can hardly rely on question features. With shuffled labels the AUROC falls between 0.46 and 0.56, close to chance. After style and lexical signals are removed, the AUROC drops by at most 0.020 for Qwen2.5-32B and Llama-3.3-70B, and to 0.862 for Qwen2.5-72B, still clearly above chance.

Inspired by your comment, we also ran a stricter check. We train a logistic regression on surface features alone (length, line count, markdown use, last token, answer-line format) to separate o_J from o_K, and then regress these features out of the activations and retrain the probe.

JudgeOriginal probeSurface onlySurface removed
Qwen2.5-7B0.9750.710.957
Qwen2.5-14B0.9600.690.940
Qwen2.5-32B0.9550.740.931
Llama-3.3-70B0.9960.770.984

These attributes do carry some information, but they reach only about 0.7, and removing them barely changes the probe.

We also agree with your caution about the word recognition. The probe shows that authorship information is linearly decodable from the hidden states. This is recognition at the level of representations, and it does not show that the judge uses this information when it votes. We now state this definition in the method section and have tightened the wording in the abstract, the introduction, and the experiments accordingly. Thank you for pointing it out.

Finally, we believe this issue does not change the main conclusion. Whatever signal the probe captures, it is present both where preference appears and where it does not. For Qwen2.5-7B on MMLU, for example, the AUROC is 0.975 while B₀ is −0.041. A signal that is present in both cases can hardly explain why preference appears in only one of them. The dissociation therefore does not depend on which information the probe reads.

W3. Presentation.

Your suggestion is well taken. We have added an intuitive description of each rewrite to the Remark in the method section. The style rewrite retells the same content in another writer's voice, the lexical rewrite swaps a model's favorite words for neutral synonyms, the reasoning rewrite keeps the facts and conclusion but changes the order of the argument, and the alignment rewrite strips assistant-style packaging such as hedges and disclaimers. We have also added a figure of the residual decomposition that shows its main conclusions at a glance, and reformatted the main tables to make them easier to read.

Thank you again for the valuable feedback.

Reviewer_XL6FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on Roytburg et al.’s work by asking whether self-preference remains when both responses reach the same incorrect conclusion. This is a useful addition because it addresses the possibility that a judge favours its own answer simply because it agrees with the conclusion.

While the paper eliminates some more confounding variables that might get labelled as self-preference, it is a bit hard to read as the authors do not focus on one main result but a set of results that might be a bit scattered and hard to derive the main scope of the paper from.

Critical strengths

The paper presents some interesting empirical results that help them make progress towards their research question, in particular, to note a few interesting results:

  1. Authorship information can be present without self-preference. For Qwen2.5-7B on MMLU, a probe distinguishes its own responses from another model’s pretty well even though it cannot detect self-preference. This shows a demarcation between identifiable authorship information and "behavioural" preference. It does not establish that the judge itself recognises authorship or uses that information.

  2. The. authors show that some self-preference can remain when both responses reach the same incorrect conclusion. This extends the earlier work by addressing agreement with the final conclusion. Agreement with the reasoning, or differences in response quality, could still explain some of the effect.

  3. The authors try to isolate a lot of signals that could get mislabelled as self preference such as rewrite style, replace words, reconstruct reasoning templates, and remove alignment-related phrases. Their decomposition attributes at most half of the robust MMLU effect to these changes. This suggests those particular features are insufficient to explain the effect; it does not establish that the remainder comes from recognising authorship and explaining what causes the remainder remains tricky.

  4. Self-votes can affect the resulting model ranking. In their leaderboard experiment, removing self-votes changes the top-ranked model. This gives the issue practical relevance, although the result depends on the evaluation setup and reference model.

Critical weaknesses
  1. The main finding leaves the explanation of self-preference partly unresolved. I find it interesting that authorship can be recovered from internal representations even when a model does not favour its own response. However, this does not show whether the judge uses authorship information when making a decision. Matching the same incorrect conclusion is a useful addition to previous work, but the responses can still differ in reasoning, assumptions, quality and more confounding factors. Similarly, preference remaining after the tested rewrites does not establish what causes it. These experiments narrow the possible explanations, but I am still unclear about what the paper establishes about why self-preference occurs.

  2. Given that the results vary across tasks, it is a bit unclear and not easy to note a trend in how self-preference occurs. Self-preference varies across tasks, but the paper does not sufficiently explain what drives those differences. This makes it difficult to draw a conclusion beyond the individual benchmark results or anticipate when self-preference will matter in another setting. The ranking experiment illustrates a possible practical consequence, but its dependence on the reference model also needs to be connected to this broader question. I found the results scattered: a clearer focus on the recognition–preference distinction, with the other experiments used to investigate its meaning and limits, would make the contribution easier to assess.

Questions for authors
  1. When both responses reach the same incorrect conclusion, what evidence helps distinguish preference for the model’s own response from preference for its reasoning or explanation? Could differences in response quality account for the remaining preference?

  2. The probes show that authorship information can be recovered from internal representations. Is there evidence that the judge uses this information when choosing a response? How narrowly should we interpret the distinction between self-recognition and self-preference?

  3. What can we conclude from the differences across tasks beyond the fact that self-preference varies? Do the experiments identify any property that helps explain when preference appears or disappears, or should these results be understood as observations specific to the tested settings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer XL6F (1/2)

We thank the reviewer for the careful and thoughtful review. Your three questions helped us bring the results together around one main line. We first answer the questions and then respond to the two weaknesses.

Q1. Own response or own reasoning, and whether quality differences explain the remaining preference.

Thank you for this question. Fixing the same incorrect conclusion removes agreement at the level of the conclusion, but the two responses can still differ in reasoning and in quality. Inspired by your comment, we designed two new experiments.

The first is a new crossed experiment that separates reasoning from writing. For Qwen2.5-32B on MMLU, the judge's argument and the proxy's argument are each written once by the judge J and once by the proxy K, so that the two vary separately.

Written by JWritten by KWriter effect
J's argument0.3910.193+0.198
K's argument0.3520.158+0.194
Argument effect+0.039+0.035

Main effects (95% CI half-width): writer +0.196 ± .045, argument +0.037 ± .031.

Reasoning does contribute. Changing the argument shifts the preference by +0.037. With the reasoning held fixed, however, changing the writer shifts it by +0.196.

The second is a stricter control for quality. Two external models rate the reasoning quality of each response without knowing its author, and we re-estimate B₀ on pairs whose two responses receive the same score.

JudgeOriginal B₀B₀ after quality matching
Qwen2.5-14B+0.140+0.108
Qwen2.5-32B+0.242+0.191
Llama-3.3-70B+0.230+0.172
Qwen2.5-72B+0.161+0.124

Bold estimates have question-clustered 95% confidence intervals that exclude zero.

B₀ decreases after quality matching, but its confidence interval still excludes zero for all four judges. Differences in reasoning and in quality each explain part of the residual, but not all of it, and the residual may depend more on how the response itself is written.

Q2. Whether the judge uses the authorship information, and how narrowly to read the distinction.

We agree with your reading. We currently have no evidence that the judge uses this information when it votes. In this paper, recognition refers to recognition at the level of representations, meaning that authorship information is linearly decodable from the hidden states. This differs from the behavioral recognition that prior work measures by asking the model. We now state this definition in the method section and have tightened the wording in the abstract, the introduction, and the experiments accordingly.

The distinction should therefore be read narrowly. Our claim runs in one direction only. Authorship information being decodable is not sufficient for preference. This claim does not depend on what exactly the probe reads. Whatever signal it captures, that signal is present both where preference appears and where it does not. For Qwen2.5-7B on MMLU, for example, the probe AUROC is 0.975 while B₀ is −0.041. A signal that is present in both cases can hardly explain why preference appears in only one of them. Whether and how the judge uses this information can only be answered by causal interventions on its internal representations, which we see as a valuable direction for future work.

AuthorsOpenReview ↗

Response to Reviewer XL6F (2/2)

Q3. Whether some property explains when preference appears.

Thank you for this question. Our original draft touched on these patterns but did not state them clearly enough, and your question helped us organize them better. We now summarize two. First, self-preference is related to model scale. On MMLU, judges at 8B and below have residuals of at most 0.05 in magnitude, while judges at 14B and above range from +0.14 to +0.24, with the same trend in both model families. Second, on objective tasks, the residual tends to be larger where reasoning is harder to verify.

For the second pattern we ran a new and finer-grained experiment. The original evidence was the contrast between MMLU and Math500, but these two benchmarks also differ in question type and answer format. To test the pattern within a single benchmark, we split the MMLU subjects into two groups. One contains subjects whose reasoning can be checked by computation (mathematics, physics, chemistry, computer science, and formal logic, with the list fixed in advance). The other contains the official humanities and social science categories.

JudgeComputableHumanities / social scienceDifference
Qwen2.5-14B+0.041+0.173+0.132
Qwen2.5-32B+0.071+0.288+0.217
Llama-3.3-70B+0.049+0.279+0.230
Qwen2.5-72B+0.046+0.198+0.152
Pooled+0.052+0.235+0.183

Bold estimates have question-clustered 95% confidence intervals that exclude zero.

The four judges agree. With question type and format held the same, the residual is close to zero on computable subjects and clearly larger on humanities and social science.

On open-ended tasks, the direction of the residual depends on the judge and the task. The Shapley decomposition shows that alignment-associated phrasing contributes more on open-ended tasks than on MMLU, in all seven judge-task cells.

W1. The cause of self-preference remains unclear.

Thank you for this comment. The focus of this paper is to give a sharper characterization of self-preference after ruling out several existing explanations. The preference does not come from agreement on the conclusion. Differences in reasoning and in quality each explain only a small part. The four surface signals explain at most half. Authorship information and familiarity remain present when the preference disappears, so they are not sufficient either. After these exclusions, the crossed experiment suggests that the remaining preference mainly follows how the response itself is written, and that it is more pronounced when the judge is large enough and the reasoning is hard to verify. This paper does not give a complete mechanism for self-preference, but we believe it narrows the possible explanations to a fairly specific range. Why a judge prefers this way of writing is a promising direction for future work.

W2. The results are scattered, and a clearer focus on recognition versus preference would help.

Your suggestion is very helpful, and we have revised the narrative accordingly. The main line of the paper is that authorship information is decodable under all conditions, while preference appears only under specific ones. The other experiments address the two questions this raises, namely under which conditions preference appears and which part of the response carries it. We have revised the abstract, the introduction, and the conclusion accordingly, and placed the new experiments at their positions along this line.

Thank you again for the valuable feedback.

Reviewer_scxWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper re-measures self-preference of LLM judges under a strict control: only pairs where judge and proxy reach the same incorrect answer, with a correct reference, so conclusion agreement cannot explain self-preference. The differential is estimated with question-level clustered inference and Benjamini–Hochberg correction. The effects of self-preference measured this way vary: the largest on MMLU (+0.242 for Qwen2.5-32B) and near zero or reversed on Math500. The paper also uses Shapley values to measure the effects of controlled rewrites, and linear probes to check whether the LLM is "aware" that it authored the text.

Critical strengths

The paper is statistically very sound. §3.2 conditions on judge and proxy reaching the same incorrect conclusion with a correct reference (Eq. 4); this adjusts for conclusion-agreement. Inference is then reported honestly: §4.1 and Appendix B cluster at the question level and BH-q for all 28 cells.

Critical weaknesses

The paper's conclusion suggests the self-preference is benchmark specific, so benchmark designers should endeavor to measure self-preference on their benchmark. The paper stops short of providing guidance to benchmark designers on how to report the susceptibility of their benchmark to self-preference, and ways of adjusting for self-preference in evaluation protocols.

Questions for authors

I suspect the answer to the self-preference question is so nuanced and complicated because self-preference is an aggregate effects. There are certain writing styles (eg, those tested in the rewrite experiments), and models have a (1) context-dependent preference for each of these writing styles, and (2) a (context-dependent) tendency to write in each of these writing styles, so the self-preference is really the alignment of its preference with its tendencies. What does your data say about this hypothesis.

1 comment
AuthorsOpenReview ↗

Response to Reviewer scxW

We thank the reviewer for the careful reading and the positive assessment. Your recognition of the statistical design is very encouraging, and both of your points have helped us improve the paper.

Q1. Self-preference as alignment between style preference and style tendency.

Thank you for this insightful hypothesis. We find it convincing, and our results support it in three ways.

First, under this hypothesis, removing writing tendencies from a response should weaken the preference, which is what our Shapley decomposition measures. The contribution of alignment-associated phrasing is larger on the open-ended tasks than on MMLU in all seven judge-task cells. The freer the response, the larger the role of such writing tendencies, which fits the context dependence in your hypothesis. We note that the overall preference on open-ended tasks is less uniform across judges, so this evidence concerns the role of writing signals rather than the size of the preference.

Second, the alignment appears in wording rather than in generic style. On MMLU the style rewrite contributes almost nothing, while lexical choice contributes most, followed by alignment-associated phrasing. The judge favors its own habitual words and expressions, which gives your hypothesis a more specific form.

Third, inspired by this hypothesis, we ran a new crossed experiment for Qwen2.5-32B on MMLU. The judge's argument and the proxy's argument are each written once by the judge J and once by the proxy K, so that writing and argument vary separately.

Written by JWritten by KWriter effect
J's argument0.3910.193+0.198
K's argument0.3520.158+0.194
Argument effect+0.039+0.035

Main effects (95% CI half-width): writer +0.196 ± .045, argument +0.037 ± .031.

When the proxy's argument is written by the judge, the judge's preference for it rises by +0.194, close to the gain for its own argument (+0.198). This suggests that the preference may depend more on how a response is written than on what it argues, in line with your hypothesis. It also offers one reading of the remainder on MMLU. Between half and four fifths of B₀ survives our four rewrites, and this part may come from writing habits that the rewrites do not cover.

Your hypothesis brings several of our results together and points to a promising direction for understanding self-preference. We have added the related discussion to the revision.

W1. Guidance for benchmark designers.

Thank you for this constructive suggestion. We agree that, since the effect is benchmark-specific, it needs to be audited on each benchmark. We have added an appendix that packages our method as an automated audit. It needs only model outputs and judge verdicts. The audit first freezes conclusions, keeping questions where the judge and a proxy reach the same incorrect answer. It then estimates B₀ with question-clustered confidence intervals, reports ICC and the design effect, and applies Benjamini–Hochberg correction across judges. Next, it quantifies the effect on rankings by removing self-votes, using Kendall's τ and the difference-in-differences self-advantage. When B₀ is significant, we recommend rerunning the leaderboard without self-votes, or with a cross-family judge outside the candidate set, as a sensitivity analysis. We will release the audit script with the paper.

Thank you again for the valuable feedback.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Summary of Revisions

We thank the three reviewers for their careful and constructive comments. We have uploaded a revised paper, with all changes marked in blue. The main changes are summarized below, and point-by-point replies are given in the individual responses.

New experiments

  1. Crossed experiment separating writing from reasoning (Section 4.2, Table 3, with details in Appendix E). The judge's argument and the proxy's argument are each written once by the judge and once by the proxy. The result suggests that the residual may depend more on how a response is written than on its reasoning. (scxW, XL6F, fnTj)
  2. Quality matching (Section 4.2, Table 6, with details in Appendix E). After responses are matched on externally rated reasoning quality, the residual remains for all four large judges. (XL6F)
  3. Split by subject type (Section 4.1, Appendix E, Table 13). Within MMLU, the residual is close to zero on subjects whose reasoning can be checked by computation and clearly larger on humanities and social science. (XL6F)
  4. Surface-feature control for the probe (Appendix E, Table 14). The probe barely changes after surface features are removed. We also add the full probe setup. (fnTj)

Clarifications

  1. Definition of recognition (Section 3.4). Recognition in this paper is recognition at the level of representations, meaning that authorship information is linearly decodable. It does not mean that the judge uses this information. Prior work mostly measures recognition by asking the judge whether a response is its own. Our measure reads authorship information directly from the hidden states and does not rely on prompts or on the judge's self-report, which may make it a more objective measure of whether this information is present. The wording in the abstract, the introduction, and Section 4.3 is tightened accordingly. (XL6F, fnTj)
  2. Main conclusions stated more clearly (abstract, introduction, Section 4.1, conclusion). The focus of the paper is to give a sharper characterization of self-preference after ruling out several existing explanations. We state two patterns. Self-preference emerges with model scale, and on objective tasks the residual tends to be larger where reasoning is harder to verify. (XL6F)
  3. Narrative and tables (abstract, introduction, conclusion, main tables). We revised the narrative to make the main line clearer and reformatted the main tables to make them easier to read. (XL6F, fnTj)

Presentation and practical use

  1. Intuition for the rewrites and a decomposition figure (Remark in Section 3.3, Appendix E, Figure 3). We add an intuitive description of each rewrite and a visualization of the residual decomposition. (fnTj)
  2. Audit for benchmark designers (Appendix H, Algorithm 1). We package the method as an audit that can be run directly. (scxW)

We thank the reviewers again for their valuable feedback.

Dear Reviewers,

We sincerely thank you for the time and effort you have devoted to reviewing our manuscript. Your comments have been very valuable in improving it. We have addressed your concerns in our responses and updated the manuscript accordingly. Please feel free to let us know if any concerns remain, we would be happy to clarify before the discussion period closes. Thanks again for your time and help in improving the quality of this manuscript.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.