- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether LLMs prefer a prior answer when it is described as their own rather than as coming from an unspecified or different assistant. Using 147 PopQA questions and four open-weight model families (Qwen3, Qwen2.5, Phi-3.5, and OLMo-2), the authors construct knowledge-conflict settings in which a supplied prior answer contradicts a reference document, while keeping the answer content fixed.
The paper introduces two attribution contrasts, self-elevation (P) and other-attribution discount (D), whose sum equals the total own-versus-different contrast. The experiments examine both correct and incorrect prior answers, alternative attribution wordings, and the effects of relabelling model-generated answers. The results show that attribution wording can influence model preferences and decisions, but the effects vary considerably across models and prompts. Several original success criteria are not met under the final reporting policy, and the authors withdraw earlier interpretations concerning independent mechanisms and differential wording robustness.
Critical strengths
-
The experimental setup keeps the prior answer content and its number of appearances fixed while varying its stated authorship (L119–140). Importantly, the authors examine both wrong prior answers paired with correct documents and correct prior answers paired with incorrect documents. This helps distinguish harmful persistence in incorrect answers from potentially beneficial retention of correct answers. I consider this a useful experimental design for studying attribution effects in knowledge-conflict settings.
-
The paper provides grouped bootstrap confidence intervals, permutation tests, multiple-comparison analyses, prompt specifications, and supplementary controls (L681–722; Appendix F). I particularly appreciate that the authors explicitly disclose post-hoc decisions, unsuccessful validation criteria, and withdrawn interpretations. These analyses provide useful methodological insights into the difficulties of evaluating attribution effects in LLMs.
Critical weaknesses
- The central decomposition lacks an independently established empirical contribution, and the main conclusions are weaker than the paper's framing suggests.
The proposed decomposition, T1 = P + D, is an algebraic identity because the unspecified-assistant reference cancels out (L59–61, L674–676). As the authors acknowledge, this equality does not provide independent evidence for distinct psychological mechanisms (L258–260, L903–909). Therefore, the contribution of the decomposition depends on whether the two contrasts reveal meaningful and distinguishable empirical properties. However, the original interpretation that these components differ in their robustness to wording has been withdrawn. The diagnostic simulation shows that the proposed criterion can be satisfied in 73.5% of cases even when the two contrasts have equal wording variability but different distances from zero (L303–307, L873–885). The activation intervention also fails its specificity requirement (L1417–1422).
More importantly, the main own-versus-unspecified criterion requires the expected effect for both correct and incorrect prior answers in at least three model families. Although all four families satisfy this criterion before exclusions, only Phi-3.5 and OLMo-2 remain under the paper's primary reporting policy (L193–208). Table E.1 further shows that none of the fifteen retrospectively selected conditions is established under that policy, although these conditions are neither exhaustive nor independent (L1526–1532). I recognize that the paper still documents measurable attribution-related effects, including significant behavioral differences between own and different-assistant conditions. However, these findings do not establish the decomposition as a distinct explanatory framework. In my view, the paper's remaining contribution is closer to a methodological analysis of prompt-dependent attribution effects than to the central conceptual claim suggested by its title.
- The empirical evidence is sensitive to wording and analysis choices, with limited generalizability and unresolved attribution confounds.
The main evaluation relies on 147 questions from one factual QA dataset and four relatively small open-weight model families. Although the paper includes additional experiments, the evidence for a generalizable self-preference effect remains limited. First, the results depend substantially on the exact attribution wording. In the six-wording experiment, only 12 of 24 wrong-prior own-minus-unspecified comparisons show significantly positive effects, while seven show significant reversals (L267–269, L859–863). The total own-versus-different contrast also changes significantly across paraphrases in all eight model-by-correctness combinations, with only three comparisons surviving the primary exclusions (L1124–1127). Moreover, replacing "different assistant" with alternative expressions substantially changes effect magnitudes and sometimes directions (L1030–1040). These findings suggest that the measured differences may partly reflect lexical or pragmatic features of the prompts rather than a stable attribution bias.
Second, the behavioral evidence is less straightforward than the headline results suggest. The abstract's 7.5, 18.4, and 20.4 percentage-point increases come from post-hoc own-versus-different choice-rate comparisons. These three comparisons survive Bonferroni correction, but for the more direct own-versus-unspecified comparison, only OLMo-2 remains significant after Bonferroni-24 (L756–773). In addition, the 90% concentration threshold was specified in advance as a flag, but its use to exclude comparisons from the primary success count was adopted after inspecting the results (L693–699). These distinctions matter when interpreting whether the effect reflects increased preference for one's own answer or reduced preference for an explicitly different assistant's answer.
Third, the practical implications of relabelling model-generated answers remain uncertain. The reported accuracy gains arise from samples heavily dominated by incorrect prior answers. For Qwen3, relabelling improves decisions on wrong prior answers but reduces accuracy by 13.0 percentage points on the much smaller correct-prior subset (L360–377). After exploratory grading exclusions, only Phi-3.5 and OLMo-2 retain clearly positive overall confidence intervals (L347–348). The subgroup effects also lack confidence intervals. Furthermore, the SciQ and free-form extensions miss their success criteria (L1343–1359), and the wrong-prior pattern is not replicated across the tested commercial-model configurations (L1470–1479). Taken together, these limitations make it difficult to determine how much of the reported effect represents a general self-preference phenomenon rather than a task-, wording-, or model-specific response pattern.
Questions for authors
-
Given that T1 = P + D is an algebraic identity and the interpretations concerning independent mechanisms and differential wording robustness have been withdrawn, what empirically testable claim remains specific to the proposed decomposition, beyond reporting three pairwise attribution contrasts?
-
How sensitive are the main conclusions to the 90% concentration threshold, for example at 80% or 95%? Could the authors clearly distinguish the results satisfying the original written criteria from those supported only under subsequent reporting decisions or post-hoc behavioral analyses?
-
To what extent can the own-versus-different behavioral effect be attributed specifically to self-preference rather than the wording "different assistant"? Given the substantial magnitude changes under alternative source descriptions (L1030–1040), what evidence supports interpreting these comparisons as measuring a general attribution effect?
-
How reliable is the reported accuracy benefit from relabelling when the prior answer is correct? In particular, could the authors provide paired confidence intervals or significance tests for the correctness-specific results in Table 4, and clarify under what prior-answer error rates the proposed relabelling strategy would be expected to improve overall accuracy?