- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the impact of the head dimension on long-context retrieval performance, training models at both 0.6B and 3B across head dimension/counts. The authors find that the going from a dimension size of 128 to 512 (with fewer total heads to match parameter count) improves long-context performance, and perform some analysis for why this occurs.
Critical strengths
- Studying the impact of different design choices on long-context retrieval performance is an important area of research for future work on model design.
- The d512 model does show large gains on RULER compared to the other two baselines, showing there is a clear effect of this architecture design.
Critical weaknesses
- The biggest issue with the paper is the core result in Table 2. First, the experimental setup chooses to study a single, specific architecture loosely inspired by Gemma 4 but makes general claims about LMs at long-context. Second, the implementation of partial RoPE as described in the paper seems to be a major confounder in the results. The authors say "We use a simpler parameterization, applying RoPE to the first 64 dimensions with a small base" but are specifically ablating the head dimension, which means the fraction of dimensions with RoPE is changing across models. One simple explanation for these results is the authors are just implicitly moving closer to the SWAN-GPT architecture and replicating their known long-context gains.
- The results in Figure 3 are not convincing. The average number of retrieval heads is essentially constant across these three architecture designs, yet the authors make the claim that "models with wider heads allocate a higher ratio of full-attention heads for retrieval". This difference is caused directly by the changing denominator, making this interpretation fairly circular.
Questions for authors
- "We call this variant RULER-100, which we confirmed does not affect the stability of the evaluation." This is a very odd bare assertion that does not seem to be backed up anywhere in the paper. Also, why use a nonstandard number of samples?
- Could the authors provide more information on their implementation of partial RoPE? The paper would likely benefit from ablating this design choice as well to remove it as a confounder.