acceptodds
Under review as a conference paper at ICLR 2027

Head Dimension Matters in Language Models for Long-Context Retrieval

Abstract

Multi-head attention retrieves and combines context representations through query–key interactions and value aggregation, with head dimension constraining the per-head representation space. In long-context processing, it remains unclear whether allocating a fixed parameter budget to fewer, wider heads improves information retrieval. In this paper, we investigate how head dimension affects the performance of long-context retrieval tasks. Our empirical studies reveal that pretraining loss and short-context non-retrieval task performance are not affected much by head dimension, while long-context retrieval performance improves with wider heads at fixed model sizes. We find that models with wider heads have a higher proportion of retrieval heads, associating more parameters with retrieval. Our ablation studies also show that the retrieval performance improves with wider heads even when the projection weights in attention modules are frozen during long-context continual pretraining, suggesting that wider heads not only benefit updates in attention modules but lead to more effective adaptation of components such as embeddings and MLPs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language models, Transformer, self attention, long context

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_haztOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the impact of the head dimension on long-context retrieval performance, training models at both 0.6B and 3B across head dimension/counts. The authors find that the going from a dimension size of 128 to 512 (with fewer total heads to match parameter count) improves long-context performance, and perform some analysis for why this occurs.

Critical strengths
  • Studying the impact of different design choices on long-context retrieval performance is an important area of research for future work on model design.
  • The d512 model does show large gains on RULER compared to the other two baselines, showing there is a clear effect of this architecture design.
Critical weaknesses
  • The biggest issue with the paper is the core result in Table 2. First, the experimental setup chooses to study a single, specific architecture loosely inspired by Gemma 4 but makes general claims about LMs at long-context. Second, the implementation of partial RoPE as described in the paper seems to be a major confounder in the results. The authors say "We use a simpler parameterization, applying RoPE to the first 64 dimensions with a small base" but are specifically ablating the head dimension, which means the fraction of dimensions with RoPE is changing across models. One simple explanation for these results is the authors are just implicitly moving closer to the SWAN-GPT architecture and replicating their known long-context gains.
  • The results in Figure 3 are not convincing. The average number of retrieval heads is essentially constant across these three architecture designs, yet the authors make the claim that "models with wider heads allocate a higher ratio of full-attention heads for retrieval". This difference is caused directly by the changing denominator, making this interpretation fairly circular.
Questions for authors
  • "We call this variant RULER-100, which we confirmed does not affect the stability of the evaluation." This is a very odd bare assertion that does not seem to be backed up anywhere in the paper. Also, why use a nonstandard number of samples?
  • Could the authors provide more information on their implementation of partial RoPE? The paper would likely benefit from ablating this design choice as well to remove it as a confounder.
Reviewer_TjRcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether allocating a fixed attention budget to fewer, wider heads improves long-context retrieval. The authors train 0.6B and 3B hybrid sliding-window/full-attention language models, varying the head dimension and head count in full-attention layers while keeping total projection widths and per-token KV-cache size fixed for the main symmetric configurations. Wider heads achieve higher average RULER-100 scores after long-context continual pretraining, with the advantage also observed after SFT at 3B. The paper further examines retrieval-head distributions, training with frozen attention projections, attention maps, and residual-stream effective rank to explore explanations for the performance differences.

Critical strengths
  1. The fixed-budget comparison is useful for practical model design. The main symmetric configurations keep total projection widths and per-token KV-cache size fixed, and the performance differences are substantial: after CPT, increasing head dimension from 128 to 512 raises the mean RULER-100 score from 31.25 to 41.21 at 0.6B and from 35.93 to 50.93 at 3B (Table 2). These results suggest that standard training loss and short-context scores may not be sufficient to select a head allocation for long-context retrieval.

  2. The wider-head advantage persists when Q/K/V/O projection matrices are frozen during CPT. This is an informative observation about the relationship between head allocation and subsequent adaptation, even though it does not establish the full mechanism. The three independent 0.6B runs, individual-run and task-level results in the appendix, and explicit reporting of exceptions to the overall ranking make the evidence easier to assess. The related-work discussion also helps clarify the scope of the contribution.

Critical weaknesses
  1. The evidence does not sufficiently isolate the proposed explanation in terms of adaptation of embeddings and MLPs. Frozen CPT leaves the Q/K RMSNorm gain vectors trainable, while input representations, MLPs, and other parameters change together (Appendix B.3 and Table 9). Adaptation within attention therefore remains possible. The attention-entropy, ranking-change, and effective-rank analyses do not separate these contributions. Although the explanation is qualified as plausible in places, the introduction makes a stronger distinction between adaptation of the remaining components and plasticity within attention. Moreover, the interpretation of adaptation gains depends on the scaled-PT baseline, and similar mean Full-minus-Frozen gains across three runs do not establish independence from head dimension. These issues limit the explanatory contribution beyond the performance observation.

  2. The interaction between head dimension and positional encoding limits the scope of the architectural conclusion. Each full-attention head uses 64 rotary dimensions, so the rotary fraction changes from 50 to 25 to 12.5 percent as head dimension increases from 128 to 256 to 512. Some of the improvement may arise from this change or its interaction with head width. This does not invalidate the reported comparisons under the stated recipe, but the effects are not separated. Both model scales use the same hybrid architecture and positional-encoding setup, and long-context evaluation is concentrated on RULER, including eight needle-retrieval variants. The evidence therefore leaves uncertainty about how broadly the architectural recommendation applies. The variation across 0.6B runs and single run per 3B configuration further limit the strength of this inference.

Questions for authors
  1. Is there evidence that the wider-head advantage and the reported adaptation pattern persist when the Q/K RMSNorm gains are also frozen? Such a control, if available, would help distinguish adaptation within attention from adaptation of embeddings and MLPs. Which component-level conclusions do the authors consider supported by the current experiments?

  2. Have the authors examined whether the head-dimension ranking persists with a fixed rotary fraction or another reasonable positional-encoding configuration? Please clarify how the fixed-64 rotary choice affects the intended scope of the finding.

  3. Why is scaled PT used as the reference for interpreting adaptation gains? From Table 11, Frozen CPT improves over scaled PT by approximately -0.01, 3.97, and 7.35 points, but over unscaled PT by 10.15, 15.80, and 15.37 points for d128, d256, and d512. Please discuss how the interpretation changes across these baselines and report or explain the run-to-run variability of the within-run gains, including Full CPT minus Frozen CPT.

  4. Please clarify the RULER-100 evaluation protocol, including whether the same generated examples are used across configurations, the decoding settings, and the evidence supporting stability with 100 examples per task. Does the ranking remain stable across evaluation-example generation seeds or sample sizes?

Reviewer_xJ58OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper obtains empirical results on the effect of changing head dimension during LM training. The main findings are that larger head dim leads to stronger performance on longer-context tasks, and benefits long-context continual training.

Critical strengths

The paper has a clean experimental setup, training models of multiple sizes from scratch, and convincingly showing that d=512d = 512 is optimal for performance. This is informative for those who want to train LMs from scratch.

Critical weaknesses
  • the paper investigates some downstream implications, such as effective rank of the residual stream, but there is much more to explore. Why does wider head benefit retrieval (does it increase retrieval accuracy for example?)? Why does effective rank of the residual stream become larger? etc.
  • results show that long-context performance improves with head width, but this is not properly ablated against stronger performance overall!
Questions for authors
  • You show that perhaps narrower heads have larger attention entropy and this is (general) associated with worse long-context performance. There are several works that consider controlling entropy for improved long-context performance. How do the results look if you account for this?
  • can you include details about dataset / pretraining mix in the main text?
  • The abstract says: 'We find that models with wider heads have a higher proportion of retrieval heads', so why not show that metric in Table 4? -I feel this metric would be better shown on a figure/plot where we vary the retrieval-score threshold. This way we can check that we consistently see more retrieval heads as we increase the head dim. (Also, table 4 suggests that d=256 gives more retrieval heads than d=512, which is not necessarily what the paper says?)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.