- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether a language model gives more weight to a history when that history is presented as its own rather than another agent’s. The authors keep the history, task rules and answer options fixed while changing the identity instructions. They also introduce a FOCAL condition, where the historical owner is explicitly called the focal agent and the assistant acts as an outside evaluator.
The evaluation uses 320 instances from four task templates, two model checkpoints and five mixed-training runs per checkpoint. For Qwen, mixed training produces no detectable change in the overall SELF–OTHER sensitivity contrast, but changes the comparisons involving FOCAL. The results differ across task types and checkpoints. The main conclusion is that a SELF–OTHER difference alone is not enough to establish a self-specific effect.
Critical strengths
-
The matched comparisons are a useful part of the design. The history, policy, options and correct answers remain unchanged across identity conditions, and the authors test both header orders. Comparing these conditions within the same model avoids some of the problems with comparing separately trained SELF-only and OTHER-only models. Using two histories that require different correct actions also gives a clear way to test whether the model responds to relevant information.
-
The authors are open about what the results do and do not establish. They report five independent training runs for each checkpoint, separate planned analyses from later checks, and provide results by task type and training seed. They also report accuracy and changes in selected answers alongside the main sensitivity measure. The limitations of FOCAL, the small number of training runs and the differences between checkpoints are acknowledged clearly.
Critical weaknesses
-
The main comparison still does not separate self-binding from focal-agent effects.
FOCAL changes several things together: it adds the focal label, changes the assistant’s role, uses different wording and was not included in mixed training. The authors acknowledge this, but it remains difficult to tell which change is responsible for the result.
I am also not clear about what extra task relevance FOCAL introduces. In OTHER, the historical owner’s state already determines the correct action under the printed policy. The history is therefore relevant in that condition too. FOCAL seems to add an explicit label and an outside-evaluator instruction, rather than making previously non relevant information relevant.
While the decomposition is mathematically correct, but it does not resolve this issue. The training result can also be described as FOCAL sensitivity increasing more than SELF or OTHER sensitivity. This is an observation about the three prompt conditions, but it does not tell us why the difference occurs. I recognise that the authors limit their claims in the abstract and discussion. Still, the title’s promise of "Separating" these effects, and the description of a "causal ownership control", suggest more than the current comparison establishes.
Header order also changes the size of the effect substantially. For mixed Qwen, the average SELF–OTHER contrast of 0.730 and order interaction of 0.704 imply contrasts of approximately 1.082 and 0.378 under the two orders. Both remain positive, so order does not explain away the average result. However, this difference deserves more discussion. The corresponding order-specific FOCAL contrasts would help show whether the decomposition is similarly sensitive to mention order.
The discussion of related work could also be more focused on closely related attribution experiments. Kumaran et al. (2026), Competing Biases underlie Overconfidence and Underconfidence in LLMs, study how attributing a prior answer to another model changes its influence. Khullar et al. (2026), Self-Attribution Bias: When AI Monitors Go Easy on Themselves, distinguish explicit authorship wording from attribution through conversational structure. These studies use different tasks, but a clearer comparison would help explain what the present focal condition adds.
-
The strongest result depends heavily on one narrow task template.
The 320 items are instances of four templates, with limited variation in the underlying decisions. In evidence quality, for example, all 80 instances compare 2/10 with 4/10 biased measurements. This task dominates the mixed Qwen SELF–OTHER result: its estimate is 2.57, compared with 0.23, 0.05 and 0.08 for the other three tasks. Removing it reduces the aggregate estimate to 0.1169, with an interval that includes zero.
The authors report this clearly. My concern is whether the result would hold beyond this particular rule and numerical comparison. Changing policy wording and thresholds would help establish whether this is a broader finding or a feature of one template.
The practical meaning of the sensitivity measure also needs clearer explanation. It measures changes in the relative log odds of two options, rather than whether the model correctly follows the policy under both histories. In mixed Qwen, SELF and OTHER have policy agreement of 66.1% and 64.8%, and their selected answers differ in 2.27% of matched cases. These figures do not invalidate the sensitivity result, but they should be discussed alongside it so that readers can judge its effect on actual decisions.
Presentation Issues
I could follow the experiment, but the main results take more effort to interpret than necessary. The abstract reports several sensitivity values without clearly naming their log-odds scale. Accuracy and changes in selected answers should be shown alongside these values.
The result tables, particularly Table 3, need larger text. Table 7 puts three different measures into each slash-separated cell, which makes it difficult to compare results. Separate labelled columns would be easier to read. The repeated version numbers and development history could also be shortened in the main text, with more space given to explaining the final experiment and its results.
Questions for authors
-
What does FOCAL add beyond explicit salience and the outside-evaluator role, given that the historical owner is already task relevant in OTHER? Is there evidence from a comparison that keeps the evaluation role and wording closely matched while varying ownership and focal designation? Results for the NEUTRAL condition used during training would also help distinguish a SELF benefit from possible distraction caused by assigning "you" to the irrelevant agent in OTHER.
-
Does the evidence-quality result hold across different policy wording and numerical thresholds? From the existing outputs, can you also report the proportion of items answered correctly under both H0 and H1, separately by condition and task type? This would make the connection between sensitivity and correct use of history clearer.
-
How sensitive are the training-change conclusions to uncertainty estimation with only five seeds? Comparing the base estimates in Table 3 with seed 256 in Table 9 gives a SELF–FOCAL change of +0.035 (0.320 - 0.285) and a FOCAL-OTHER change of -0.085 (0.346 - 0.431). These run opposite to the aggregate changes of -0.333 and +0.347. This does not contradict the stability of leave-one-seed-out averages, but shows that the pattern is not shared by every run. Would the conclusions remain supported under an alternative interval calculation suited to a small number of independent runs?
-
Are the current-facts and irrelevant-metadata control results available for SELF-FOCAL and FOCAL-OTHER? The reported SELF-OTHER controls do not directly test whether the main FOCAL-related changes also appear when the history is irrelevant.