acceptodds
Withdrawn from ICLR 2027

Self-Preference Is a Sum: A Frame-Conditional Decomposition for LLM Judges

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models can encounter a prior answer alongside conflicting information. We test whether describing that answer as the model's own changes its choice. On held-out PopQA questions, we evaluate Qwen3, Qwen2.5, Phi-3.5 and OLMo-2. We pair a prepared answer with a conflicting document; either the answer or the document is correct. The answer text and number of appearances stay the same, verified in code; its description names the model itself, an unspecified language-model assistant or a different assistant. When the prior answer is wrong, self-attribution increases its selection by 7.5, 18.4 and 20.4 percentage points in three families relative to different-assistant attribution. This post-hoc analysis derives choices from answer-letter log probabilities. Different-assistant wrong-choice rates are 1.4%–6.8%; the fourth family's −0.7-point change is not statistically significant. Comparing own with unspecified authorship, and unspecified with different authorship, gives two log-probability differences that sum exactly to the total on the same questions. In a post-hoc analysis, rephrasing changes this total in all eight combinations of model family and prior-answer correctness. Only three remain after excluding conditions that choose one answer side at least 90% of the time, all with correct prior answers. We withdraw earlier interpretations that these differences vary independently or differ in their robustness to wording. Nine of fifteen retrospectively selected conditions miss their original criteria; none is established under our reporting policy adopted after seeing the data. The core study's preregistration remains unverified. Separately, relabelling model-generated answers from the model's own to an unspecified assistant's improves final-decision accuracy by 6.4–9.4 percentage points in three families. After exploratory grading exclusions, two gains retain confidence intervals above zero. Relabelling can also harm decisions on correct prior answers. Authorship wording therefore affects decisions, with outcomes depending on phrasing, analysis and prior-answer correctness.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-as-a-judge, self-preference bias, evaluation validity, construct validity, knowledge conflict, attribution, prompt sensitivity, open-weight language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_swmjOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether LLMs prefer a prior answer when it is described as their own rather than as coming from an unspecified or different assistant. Using 147 PopQA questions and four open-weight model families (Qwen3, Qwen2.5, Phi-3.5, and OLMo-2), the authors construct knowledge-conflict settings in which a supplied prior answer contradicts a reference document, while keeping the answer content fixed.

The paper introduces two attribution contrasts, self-elevation (P) and other-attribution discount (D), whose sum equals the total own-versus-different contrast. The experiments examine both correct and incorrect prior answers, alternative attribution wordings, and the effects of relabelling model-generated answers. The results show that attribution wording can influence model preferences and decisions, but the effects vary considerably across models and prompts. Several original success criteria are not met under the final reporting policy, and the authors withdraw earlier interpretations concerning independent mechanisms and differential wording robustness.

Critical strengths
  • The experimental setup keeps the prior answer content and its number of appearances fixed while varying its stated authorship (L119–140). Importantly, the authors examine both wrong prior answers paired with correct documents and correct prior answers paired with incorrect documents. This helps distinguish harmful persistence in incorrect answers from potentially beneficial retention of correct answers. I consider this a useful experimental design for studying attribution effects in knowledge-conflict settings.

  • The paper provides grouped bootstrap confidence intervals, permutation tests, multiple-comparison analyses, prompt specifications, and supplementary controls (L681–722; Appendix F). I particularly appreciate that the authors explicitly disclose post-hoc decisions, unsuccessful validation criteria, and withdrawn interpretations. These analyses provide useful methodological insights into the difficulties of evaluating attribution effects in LLMs.

Critical weaknesses
  1. The central decomposition lacks an independently established empirical contribution, and the main conclusions are weaker than the paper's framing suggests.

The proposed decomposition, T1 = P + D, is an algebraic identity because the unspecified-assistant reference cancels out (L59–61, L674–676). As the authors acknowledge, this equality does not provide independent evidence for distinct psychological mechanisms (L258–260, L903–909). Therefore, the contribution of the decomposition depends on whether the two contrasts reveal meaningful and distinguishable empirical properties. However, the original interpretation that these components differ in their robustness to wording has been withdrawn. The diagnostic simulation shows that the proposed criterion can be satisfied in 73.5% of cases even when the two contrasts have equal wording variability but different distances from zero (L303–307, L873–885). The activation intervention also fails its specificity requirement (L1417–1422).

More importantly, the main own-versus-unspecified criterion requires the expected effect for both correct and incorrect prior answers in at least three model families. Although all four families satisfy this criterion before exclusions, only Phi-3.5 and OLMo-2 remain under the paper's primary reporting policy (L193–208). Table E.1 further shows that none of the fifteen retrospectively selected conditions is established under that policy, although these conditions are neither exhaustive nor independent (L1526–1532). I recognize that the paper still documents measurable attribution-related effects, including significant behavioral differences between own and different-assistant conditions. However, these findings do not establish the decomposition as a distinct explanatory framework. In my view, the paper's remaining contribution is closer to a methodological analysis of prompt-dependent attribution effects than to the central conceptual claim suggested by its title.

  1. The empirical evidence is sensitive to wording and analysis choices, with limited generalizability and unresolved attribution confounds.

The main evaluation relies on 147 questions from one factual QA dataset and four relatively small open-weight model families. Although the paper includes additional experiments, the evidence for a generalizable self-preference effect remains limited. First, the results depend substantially on the exact attribution wording. In the six-wording experiment, only 12 of 24 wrong-prior own-minus-unspecified comparisons show significantly positive effects, while seven show significant reversals (L267–269, L859–863). The total own-versus-different contrast also changes significantly across paraphrases in all eight model-by-correctness combinations, with only three comparisons surviving the primary exclusions (L1124–1127). Moreover, replacing "different assistant" with alternative expressions substantially changes effect magnitudes and sometimes directions (L1030–1040). These findings suggest that the measured differences may partly reflect lexical or pragmatic features of the prompts rather than a stable attribution bias.

Second, the behavioral evidence is less straightforward than the headline results suggest. The abstract's 7.5, 18.4, and 20.4 percentage-point increases come from post-hoc own-versus-different choice-rate comparisons. These three comparisons survive Bonferroni correction, but for the more direct own-versus-unspecified comparison, only OLMo-2 remains significant after Bonferroni-24 (L756–773). In addition, the 90% concentration threshold was specified in advance as a flag, but its use to exclude comparisons from the primary success count was adopted after inspecting the results (L693–699). These distinctions matter when interpreting whether the effect reflects increased preference for one's own answer or reduced preference for an explicitly different assistant's answer.

Third, the practical implications of relabelling model-generated answers remain uncertain. The reported accuracy gains arise from samples heavily dominated by incorrect prior answers. For Qwen3, relabelling improves decisions on wrong prior answers but reduces accuracy by 13.0 percentage points on the much smaller correct-prior subset (L360–377). After exploratory grading exclusions, only Phi-3.5 and OLMo-2 retain clearly positive overall confidence intervals (L347–348). The subgroup effects also lack confidence intervals. Furthermore, the SciQ and free-form extensions miss their success criteria (L1343–1359), and the wrong-prior pattern is not replicated across the tested commercial-model configurations (L1470–1479). Taken together, these limitations make it difficult to determine how much of the reported effect represents a general self-preference phenomenon rather than a task-, wording-, or model-specific response pattern.

Questions for authors
  • Given that T1 = P + D is an algebraic identity and the interpretations concerning independent mechanisms and differential wording robustness have been withdrawn, what empirically testable claim remains specific to the proposed decomposition, beyond reporting three pairwise attribution contrasts?

  • How sensitive are the main conclusions to the 90% concentration threshold, for example at 80% or 95%? Could the authors clearly distinguish the results satisfying the original written criteria from those supported only under subsequent reporting decisions or post-hoc behavioral analyses?

  • To what extent can the own-versus-different behavioral effect be attributed specifically to self-preference rather than the wording "different assistant"? Given the substantial magnitude changes under alternative source descriptions (L1030–1040), what evidence supports interpreting these comparisons as measuring a general attribution effect?

  • How reliable is the reported accuracy benefit from relabelling when the prior answer is correct? In particular, could the authors provide paired confidence intervals or significance tests for the correctness-specific results in Table 4, and clarify under what prior-answer error rates the proposed relabelling strategy would be expected to improve overall accuracy?

Reviewer_838vOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether an LLM changes its final choice when the content of a prior answer stays the same but its stated source changes. There are three RQs: (1) Is an LLM more likely to choose a prior answer when it is described as its own ("I was asked this and answered: X.")? (2) Does this effect hold when the authorship sentence is rephrased? (3) What happens when the model's own generated answer is relabelled as an unspecified assistant's answer ("An assistant was asked this and answered: X.")? To answer these questions, the authors define self-elevation P (own − unspecified), other-attribution discount D (unspecified − different) and the total T1 = P + D, all computed on the wrong-minus-correct log-probability score of the answer letters. Each prompt has three turns: (1) the user asks a factual question; (2) a prepared prior answer is inserted as the assistant's reply, preceded by the authorship sentence; (3) the user gives a reference document with a conflicting answer and asks the model to choose between the two. Either the prior answer or the document is correct. Four small open models (Qwen3-8B, Qwen2.5-7B, Phi-3.5-mini, OLMo-2-7B) are evaluated on PopQA. SciQ, larger Qwen models and commercial models are in the appendix. The main results show that authorship wording affects the final decision (e.g., relabelling generated answers improves final accuracy by 6.4–9.4 points in three models). However, the effect depends on wording, analysis choices and correctness of the prior answer.

Critical strengths
  1. Controlled setup. Only the authorship sentence changes; the answer text, option order and number of answer occurrences stay fixed. This separates the effect of the stated source from the content and quality of the answer.

  2. The unspecified-assistant condition gives a third reference besides own and different. This shows whether the effect comes from naming the model itself or from naming a different assistant, not only their total.

  3. The paper reports both log-probability shifts and wrong-choice rates. This shows when a log-probability shift does not change the decision. It also reports the criteria that failed, not only those that passed.

Critical weaknesses
  1. The paper is hard to read. The abstract contains many new terms and results that cannot be understood without the appendix. The same manipulation is given different names ("authorship description", "attribution sentence", "authorship sentence", "source description", "frame"). P, D and T1 are introduced and then replaced by "own-minus-unspecified" and similar names. Much of the main text is defensive and states what the results do not show. Eg, the point that the total is not independent evidence of separate mechanisms is made three times around Figure 2's caption. These caveats are useful, but stating one main claim, using one term, and moving the study history and caveats to the appendix would make the paper easier to follow.

  2. Limited novelty and practical value. Prior work already varies self and other labels on the same content (Kumaran et al., 2025; Khullar et al., 2026). The new parts are mainly the unspecified-assistant reference and the knowledge-conflict setting. The decomposition T1 = P + D adds little because it holds by construction. The practical value is also unclear: the relabelling gain is measured on an error-heavy sample (about 76–84% wrong prior answers) and can hurt correct prior answers (−13.0 points for Qwen3).

  3. The title and framing do not match the experiments. The title refers to "LLM judges" and "self-preference", but the main task is a forced A/B choice under knowledge conflict, not a judging task. In the main experiment the prior answer is only described as the model's own; it is not generated by the model. The Discussion itself lists LLM judges as a setting for future tests.

  4. Many analysis choices were made after seeing the data, so it is hard to tell which results are reliable. The 90% exclusion rule became the primary reporting rule after seeing the data, the choice-rate analysis behind the abstract's behavioral numbers is post-hoc, and the matched own-answer sentence was added to the exploratory study after its results were inspected. The core study's preregistration also cannot be verified. Under the authors' own primary rule, the main own-attribution criterion is met in only two of four families (three required), and none of the fifteen selected conditions in the abstract is established.

Questions for authors

See weakness.

Reviewer_iN3kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The research question the paper studies: when an LLM has to choose between an earlier answer and conflicting evidence, does merely labelling that earlier answer as its own make the LLM stick with it more, with the content held fixed? This paper approaches self-preference measurement by labeling an answer explicitly to control for the quality confounder---when a judge favors its own outputs because they are actually better.

The paper's findings aren't definitive.

  • Labeling the prior answers has (partial) expected effect. “Mine” increases support for the prior answer, and “different assistant” decreases it. The effect is significantly more visible in the log probabilities than in answer choices.
  • The effect doesn't seem stable. The effect size has a high variance across different phrasing of the attribution sentence, sometimes even flipping the sign.
  • “Different assistant” labeling more consistently reduces support across phrasings than the increase from "mine" label. It's unclear if these are two separate mechanisms, as the tests can’t tell spread from distance-to-zero.
Critical strengths
  • The experiment design is effective at controlling for the familiarity and quality confound. Overall the experiments are quite comprehensive, and gives me confidence that the findings paint an accurate picture about this type of self-preference in the models studied.
Critical weaknesses
  • The high variance in the effect of attribution across attribution wordings is unfortunate. It means we can't be too confident about whether: models tested are confused, or the experiments cannot fully elicit self knowledge. So overall I don't know what to take away from this.
  • The decomposition into two mechanisms has no independent evidence.
  • There are many post-hoc rules. In particular, the 90% exclusion rule was chosen after seeing the data, and it changes most headline results. The 15 conditions mentioned in the abstract were selected post-hoc too.
  • The sample is small and narrow. Models only go up to 14B, and the size trend is inconsistent.
Questions for authors
  • I'm curious how you think about the overall framing and the title. own-other = (own-unspecified) + (unspecified-other) is correct by arithmetic, but we don't seem to have evidence for separate mechanisms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.