- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies the correlation between early contents and LLM performance in Gemma and Qwen. Two models showed different results. For Gemma, if you refer the early parts less, the accuracy decreased. On the other hand, Qwen does not show a similar tendency. For Qwen, tendency was more complex.
Critical strengths
This paper tackles important part that LLM is currently struggling to solve. It goes beyond observing attention patterns and directly tests whether changing attention affects accuracy. Comparisons across token positions, layer groups, and two model architectures help clarify where these effects occur and the limits of their generalization.
Critical weaknesses
The practical implications of the findings remain limited. Although moderate early context amplification improves Gemma’s accuracy, the paper does not establish a mitigation strategy that generalizes across models. The Qwen results reveal opposing layer level effects, but it remains unclear how these observations can guide an effective intervention to improve conversational reliability.
Questions for authors
Section 5.5 (p.9, lines 432–433) states that every detected layer-band effect in Gemma is carried by the deep half of the network. However, Appendix G (p.21, lines 1118–1121) reports that, under strong early-window amplification (β=+2), the shallow half shows a significant improvement while the deep half does not. Could you clarify your claim?