- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper analyses the functional organisation of RoPE Transformers and uses it to guide the design of hybrid FA–LA architectures. The analysis relies mainly on two metrics, RFIS and RPD. RFIS measures how removing a single frequency's contribution changes a head's attention distribution, while RPD measures dependence on positional modulation by removing only the rotation of that frequency. Based on these analyses, the authors suggest that RoPE Transformers have two types of heads, namely retrieval heads and positional heads. Building on this, they propose HwH, which combines NoPE FA heads for retrieval with GDN heads for local modelling within each layer, and compare it with a Transformer, a pure GDN model and a layer-wise hybrid at the 380M and 1.4B scales, with ablations at 380M.
Critical strengths
- Using the RFIS and RPD analysis to guide the design of hybrid architectures is a clever link.
- The analysis covers several model families, and the evaluation goes beyond perplexity with a broad set of benchmarks.
Critical weaknesses
I think the authors should address three weaknesses, which would affect my score. The main one concerns how the allocation ratios are chosen (1). The other two concern whether the analysis behind them holds in the authors’ own setting. They ask whether the head distribution measured in Qwen3 transfers to 2K training (2), and whether NoPE attention is really position-free (3).
(1) The paper argues that existing FA–LA allocation strategies are largely heuristic, but it is unclear how the proposed approach reduces this reliance on heuristics. In particular, the choice of the 1:3 and 1:7 ratios is not clearly justified, and the paper does not provide a general guideline for choosing allocation ratios in new hybrid architectures.
(2) Linear attention is proposed for local token relationships, as suggested by the RoPE analysis, while NoPE attention is proposed for position-free retrieval. However, the layer-wise head distribution that motivates the hybrid ratios is measured on Qwen3-4B, which was trained with a 32K context, whereas HwH and its baselines are trained with a 2K context. If documents are shorter than the 2K window, the model can in practice learn to process each document independently, so its frequency usage, and with it the number and placement of retrieval heads, may differ at this context length. In that case, the recommended hybrid ratios may not transfer. Applying the head-level RFIS/RPD and QRscore analysis to the authors' own 2K RoPE Transformer baselines would show whether their distribution matches the one observed in Qwen3.
(3) Because each token can attend only to previous tokens, the causal mask allow the model to learn some positional information indirectly, as shown by Haviv et al. (2022) (https://arxiv.org/abs/2203.16634). NoPE heads could therefore also take on local tasks, which would be inefficient. Inductive biases often do not work as intended, so further analysis of this would strengthen the paper.
Questions for authors
Q1: How were the specific ratios derived from the Qwen3 head distribution? Is there a quantitative mapping, for example per-layer retrieval-head fraction mapped by relative depth, and how should practitioners choose ratios for a different depth or scale? (See Weakness 1. I would also like to discuss this in light of Weaknesses 2 and 3, since both affect whether the Qwen3-derived ratios hold in practice.)
Q2: To what extent do the head-level RFIS/RPD and QRscore distributions in your own 2K RoPE baselines align with the patterns observed in Qwen3, measured across relative depth? (See Weakness 2.)
Q3: How are documents packed into training sequences? Are they separated by EOS/BOS tokens, and is cross-document attention masked? This matters for interpreting GPBand, since without document masking the effective context may be closer to the typical document length than to 2K. (See Weakness 2)
5 comments
Response to zbfw (2/4): Diagnostics in our own 2K models [W2, Q2]
We have added the requested analysis in our own 380M Transformers trained at 2K, including both the QKNorm model and the original Transformer without QKNorm (Appendix B.4, Functional differentiation in our pretrained Transformers).
Across depth: Figure 11. QRscore follows the query-focused retrieval-head analysis of QRHead: Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (arXiv:2506.09944); Appendix A specifies our probe samples and scoring. We show head-wise QRscores in their original order and sorted within every layer for each model. High retrieval scores concentrate at intermediate depths in both, consistent with the layer-specific pattern seen in Qwen3-4B. This provides a direct qualitative comparison across depth without turning continuous retrieval scores into arbitrary binary Head counts.
Frequency dependence and behavior: Figure 12. We align RFIS/RPD heatmaps with the QRscores of the same Heads in Layer 11 of the model with QKNorm and Layer 12 of the model without QKNorm. Several high-QRscore Heads use low frequencies with substantial RFIS and relatively weak RPD. Other Heads emphasize the salient band and can also exhibit isolated low-frequency RFIS peaks. Thus, the small models exhibit the same implicit functional differentiation and mixed Head behavior that motivate our interpretation of the larger pretrained models.
Training-scale dependence: Figure 3. The band shifts with the RoPE base at a fixed training length and shifts again when the training length changes from 2K to 4K. Each diagnostic uses its model's training length. The recurrence of the functional pattern does not require the band to occupy the same group index, or the Heads to have identical counts and scores, across scales and context lengths.
These observations address the transfer concern empirically: the phenomena are present in the models trained under our experimental setup. We use them to motivate layer-specific allocation and test that choice through the architectural ablations. We do not assume that Qwen3's exact Head distribution transfers to 2K or determines an optimal hybrid ratio. The training-sequence construction and the effect of document boundaries are addressed in the next response.
We have also included the complete diagnostic visualizations redrawn in the paper's style and original QR/LD/RFIS/RPD score files in the updated supplementary materials, covering our 380M Transformers and the larger pretrained models. These retain all measured layer/Head/frequency grids beyond the selected manuscript panels; the indexes identify the available metrics for each configuration. Please feel free to check MODEL_CONSISTENCY_GUIDE.txt, data/diagnostics/, and figures/diagnostics/ for the full cross-scale comparison.
Response to zbfw (3/4): Data construction and document boundaries [W2, Q3]
Dataset construction. Our corpus is the sample-100BT subset of FineWeb-Edu, introduced in The FineWeb Datasets (arXiv:2406.17557). FineWeb-Edu filters web documents from FineWeb using an educational-quality classifier trained on Llama-3-70B-Instruct annotations, retaining documents at a score threshold of 3. The released subset is randomly sampled and contains approximately 100B GPT-2-counted tokens. This corpus-level count is separate from the training tokens obtained with our tokenizer. The subset contains documents, not prepacked 2K examples; the training framework constructs sequences after tokenization. See the FineWeb-Edu dataset card.
Exact packing and attention in our experiments. We train with Flame: Flash Language Modeling Made Easy, a torchtitan-based training framework integrating Flash Linear Attention (FLA) and Transformers. We use its default fixed-length setting, varlen=false, with a sequence length of 2,048. The OnlineTokenizedIterableDataset tokenizes each document separately, appends its tokens to a stream, and takes consecutive 2,048-token blocks, carrying the remainder forward. Cutting does not respect document boundaries: a block can contain multiple documents, and a document can continue into the next block.
The tokenizer prepends BOS <s> (ID 1) to each document and does not append EOS </s> (ID 2). The concatenation procedure inserts no additional separator. A boundary is therefore marked by the next document's BOS. Because blocks are cut at a fixed length, a training sample need not begin with BOS.
With varlen=false, the collator does not construct document-level cu_seqlens. Each block is one causal sequence with positions 0-2,047 and no padding. Cross-document attention is not masked, and BOS is not an attention boundary. This is Flame's standard fixed-length path; see its data pipeline, training loop, and default tokenizer configuration.
Comparison with production LLM training. This shares the fixed-sequence-budget autoregressive training setup used by models such as Qwen3. The Qwen3 Technical Report (arXiv:2505.09388, Section 3.2) specifies 4,096-token sequences in the first two stages and 32,768-token sequences in the long-context stage. For the latter, 75% of texts are 16K-32K long and 25% are 4K-16K. Its report does not disclose the exact BOS/EOS packing or cross-document masking implementation, so those details cannot be equated with Flame's from the report alone. Our controlled experiment simplifies the training curriculum to a known fixed length, allowing the learned positional scale to be examined directly.
Can the model learn document-aware behavior? Yes, and this can coexist with training-scale positional dependence. In our setup, each query still scores all preceding keys in the 2K block. A document boundary can signal that some context is irrelevant, but it does not remove distant keys from the attention computation. Distinguishing nearby, useful context from distant context can therefore involve positional dependencies over the available training span, even when useful semantic dependencies are mostly within documents. Typical document length and the span over which attention must distinguish positions are not interchangeable.
The interpretation is supported by measurements, not packing alone. Figure 3 shows the band changing with training length and RoPE base; Figures 11-12 show the same qualitative QR/RFIS/RPD relationships in our own 380M, 2K models. Thus, document boundaries do not invalidate the functional differentiation observed in this setting, and the architectural tests for validating the two principles do not rely on transferring exact Head counts from a long-context Qwen3 model.
Response to zbfw (1/4): Ratios and transferable design guidance [W1, Q1]
Thank you for asking how to select the allocation for a different model. Appendix F.3 now describes how pretraining could choose the layer-wise proportions under a fixed Q/K/V representation-width budget.
We kindly ask you to first read our global response. It clarifies our core claim and the evidence supporting the two principles.
How were the reported ratios chosen? We used the Qwen3 observations to choose where to allocate more retrieval width. We did not convert measured Head counts into numerical ratios. Inter assigns one of every four layers to NoPE full attention. This gives a global width fraction of 1/4, which we used in the middle half of HwH-std. We halved that fraction to 1/8 in the remaining non-boundary layers and used only local mechanisms in the first and last layers. These choices give the 1:3 and 1:7 global:local Q/K/V width ratios in Appendix E, Table 8. At 380M the full-attention and Gated DeltaNet Heads have different widths, so a Head count alone does not specify the allocation.
What transfers to a new depth or scale? The QR and LD profiles that guided HwH-std show prominent retrieval in the middle layers, with no pronounced retrieval tendency and strong local-streaming behavior in the first and last layers (Figure 1 and Appendix A.2). This observation motivated using only local positional modeling at both ends of HwH-std. Appendix F.3 discusses a possible explanation: local computation can build contextualized retrieval keys early and integrate previously retrieved content before prediction. However, this boundary prior leaves the mix in the intervening layers open. In a 1.3B comparison, the linear-to-full layer ratios with the highest measured RULER recall were 3:1 for GatedDeltaNet and 6:1 for HGRN2 (Wang et al., 2025, Table 5). This motivates learning the proportions during pretraining for the mechanisms in use. In Appendix F.3, I propose fixing each layer's total Q/K/V representation width. Each eligible width-h Head slot has two independently parameterized branches, global NoPE retrieval and local positional modeling. Two learned scalar parameters per slot give softmax weights initialized to (0.5,0.5). The same weights mix the branches' width-h outputs for every input. An entropy penalty alongside the language-modeling objective encourages one weight to approach one. Once choices have stabilized across the network, retaining the higher-weight branch at each slot and pruning the other gives a deterministic architecture. Both the layer-wise ratios and the overall retrieval share emerge from training.
What is established empirically? In our fixed configurations, dense RULER NIAH-Single-2 means were 60.06 for HwH-rmfl and 68.42 for HwH-std, with the latter using less global width (Table 2). These results establish the performance of the reported schedules. Learning the schedule during pretraining is a proposal in Appendix F.3. Its performance on new models remains to be evaluated. The following replies address our own 2K diagnostics and the interpretation of NoPE.