- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies why LLMs follow a historical user preference over stronger task evidence. The authors select the Memory-Evidence Conflict task of MemSyco-Bench in an A/B format. The paper select 100 tasks where Qwen 2.5 choose the preferred option but answers correctly once the preference turns are deleted. Then stop the model from reading the preference turns or the correct evidence at middle layers, fixing the input to models. And then measure the changes of answer margins and explore where the changes causes in later layer states. The experimental results show that correct evidence affects the final asnwer of LLMs, but preference has much stonger effect on the final answer. The paper finds that the repeated preference and larger attention mass on the preference turns cause that.
Critical strengths
- The causal analysis is careful. Sites are selected on discovery scenarios and tested on held out scenarios, also consider the option order. Key effects are also compared with matched historical windows and equal-norm random directions, which makes the causal claims more convincing.
Critical weaknesses
- Limited novelty and contribution. The methods are standard activation patching and attention ablation. The main message, that a misleading context wins in later layers while the correct signal stays active, is close to prior findings on false demonstrations and sycophancy (Halawi et al., 2024; Wang et al., 2026). The paper cites these works only to say that the history setting is underexplored, and does not show how its mechanism differs from theirs.
Questions for authors
See weaknesses.