acceptodds
Under review as a conference paper at ICLR 2027

Early Context Starvation in Conversational LLMs: Causal Evidence from Attention Interventions

Abstract

Large language models degrade over long conversations, but standard evaluation offers no mechanistic account of why. On human multi-turn dialogue rather than synthetic probe tasks, we show that phase-dependent degradation persists across three generations of two proprietary model families despite real capability gains. On two open-weight models, Gemma 4 31B and Qwen3.6-27B, we then measure and intervene on attention directly. In Gemma, attention to a conversation's earliest content collapses as the dialogue grows, because its sliding-window layers lose sight of early tokens and only its global layers still reach them. A dose-response intervention that biases attention toward or away from early context shows this attention is causal for late-turn accuracy. The effect is position-specific: the same intervention on middle-context keys reverses direction, and on late-context keys produces a sharp threshold collapse. Layer-band interventions localize the early-context effect to Gemma's deep layers. On Qwen3.6 the intervention moves attention just as much but leaves behavior unchanged, because its shallow and deep layers push in opposite directions and cancel. Whether attention to early context matters is therefore a property of the architecture, which motivates phase-resolved evaluation and architecture-aware accounts of multi-turn failure.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, attention, causal intervention, multi-turn dialogue, long-context degradation, sliding-window attention, hybrid architectures, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eoYiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the correlation between early contents and LLM performance in Gemma and Qwen. Two models showed different results. For Gemma, if you refer the early parts less, the accuracy decreased. On the other hand, Qwen does not show a similar tendency. For Qwen, tendency was more complex.

Critical strengths

This paper tackles important part that LLM is currently struggling to solve. It goes beyond observing attention patterns and directly tests whether changing attention affects accuracy. Comparisons across token positions, layer groups, and two model architectures help clarify where these effects occur and the limits of their generalization.

Critical weaknesses

The practical implications of the findings remain limited. Although moderate early context amplification improves Gemma’s accuracy, the paper does not establish a mitigation strategy that generalizes across models. The Qwen results reveal opposing layer level effects, but it remains unclear how these observations can guide an effective intervention to improve conversational reliability.

Questions for authors

Section 5.5 (p.9, lines 432–433) states that every detected layer-band effect in Gemma is carried by the deep half of the network. However, Appendix G (p.21, lines 1118–1121) reports that, under strong early-window amplification (β=+2), the shallow half shows a significant improvement while the deep half does not. Could you clarify your claim?

Reviewer_3i2DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the observation that LLM performance deteriorates over multi-turn conversations; they hypothesize that this is because models slowly lose access to information presented earlier in the dialogue. The authors first evaluate turn-dependent degradation across several model families. They then investigate this mechanism in a Gemma and Qwen model by modifying the attention scores of different positional windows. They found that increasing attention to early-context tokens improves downstream accuracy for Gemma (on a customer-service dataset). In contrast, Qwen showed little response to the intervention.

Critical strengths

--the study performs causal interventions to ask about performance degradation in multi-turn conversations, which is a novel idea that builds nicely on previous observational studies of multi-turn degradation.

--the authors benchmarked a wide range of model families. in general, the experiments seem to be thorough and comprehensive.

Critical weaknesses

--the writing seems quite unfocused and is confusing to follow. many paragraphs are missing main topic sentences (take section 5.2 as an example). overall, the writing is quite difficult for readers to follow.

--it is good that the authors tested for confounders of early context attention decay (like task difficulty), but there seemed to be no evidence that task difficulty is not a confounder (the results of section 4.3 are inconclusive). this makes me wonder whether there were other obvious confounders that could have been tested and controlled for (did the style, e.g., leading questions versus follow ups, or category of discussions change from early to late-stage in a conversation)?

--(minor weakness) please increase the axis sizes and labels to match the font size; they are currently a bit hard to read.

Questions for authors

--given the conclusion is “whether early attention matters is thus architectural”; it would be beneficial to test on at least 1-2 more families. have you run these tests or would you be able to do so?

Reviewer_JxTzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors investigate phase dependent performance degradation when language models evaluate extended human dialogues focusing on attention to early context as a potential mechanism. They combine behavioral comparisons across three generatios of models families with attention measurements and interventions. The main intervention adds a scalar bias to attention logits for keys in an early, middle or late positional window. Evaluations are provided on HRI-CUES, ABCD and MultiWOZ and ESConv dataset.

Critical strengths
  1. The use of HRI-CUES and ABCD provides a broad empirical setting beyond fact retention and personal compliance tasks and the datasets offer complementary measurements on both subjective ratings and objective actions. This is valuable to the community.
  2. The attention intervention are interesting and hold the preediction instance and prompt fixed while varying an internal computation. For gemma's late phase early window boost, there is a net improvement of 4.9 pp which is evidence that the intervention affects action prediction.
  3. The mask diagnostic in Sec 5.1 is a strong component since the authors show that global layer attention remains approximately constant and the observed pooled plateau is explained by the proportion of global layers.
Critical weaknesses
  1. The sliding window mechanism is established on HRI CUES whereas all behavioral interventions happen on ABCD and there is a mismatch between the number of prompt tokens across the two and what fits in the local window. The intervention would therefore establish sensitivity to positional reweighting where structural exclusion generally does not occur. It does not directly establish that the exclusion observed on HRI CUES causes its behavioral degradation.
  2. The paper does not provide the exact ABCD propmt or establish which parts of this window contain the dialogue, task and action definitions.
  3. Mcnemars test pairs natural and intervened predictions for each instance, but its standard exact form does not account for dependences between multiple instances from the same conversation.
Questions for authors

See weaknesses section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.