acceptodds
Under review as a conference paper at ICLR 2027

Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design

Abstract

Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains largely heuristic. We seek an evidence-grounded basis in the head-level functional organization learned by modern RoPE-based Transformers. Behavioral retrieval and local-streaming probes reveal useful tendencies but do not yield a complete taxonomy. We therefore propose two faithful intervention-based metrics: RoPE Frequency Importance Score (RFIS), which measures how each frequency contribution affects a head's complete attention distribution, and RoPE Positional Dependence (RPD), which isolates the effect of rotary positional modulation. Applied to Qwen3-series models and Llama3.1, RFIS suggests and RPD verifies a complete two-type taxonomy comprising retrieval and positional heads in RoPE Transformers, separated by a salient mid-low-frequency band. Controlled Transformers show that this functionally separating band follows the training-length positional scale; we term this mechanism-level boundary the Global Positional Band (GPBand). Its global positional dependence suggests a potential cause of zero-shot length-extrapolation failure and, together with the layer-specific head distribution, yields two design principles: (i) positional modeling should operate only locally, while global access should be implemented through position-independent retrieval; and (ii) retrieval and positional functions should be assigned at head granularity with layer-specific allocation. We instantiate these principles in the Head-wise Hybrid Architecture (HwH), using NoPE FA for global retrieval and LA for local positional modeling. With an overall FA-to-LA ratio less than 1:3, HwH retains strong language modeling and commonsense reasoning while improving retrieval and substantially strengthening zero-shot long-context extrapolation over Transformer, LA, and a 1:3 layer-wise hybrid baseline. Ablations validate both principles and the component roles, highlighting principled hybrid architecture design as a promising route toward future foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid architectures, Transformers, rotary position embeddings, attention head specialization, linear attention, length extrapolation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zbfwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyses the functional organisation of RoPE Transformers and uses it to guide the design of hybrid FA–LA architectures. The analysis relies mainly on two metrics, RFIS and RPD. RFIS measures how removing a single frequency's contribution changes a head's attention distribution, while RPD measures dependence on positional modulation by removing only the rotation of that frequency. Based on these analyses, the authors suggest that RoPE Transformers have two types of heads, namely retrieval heads and positional heads. Building on this, they propose HwH, which combines NoPE FA heads for retrieval with GDN heads for local modelling within each layer, and compare it with a Transformer, a pure GDN model and a layer-wise hybrid at the 380M and 1.4B scales, with ablations at 380M.

Critical strengths
  • Using the RFIS and RPD analysis to guide the design of hybrid architectures is a clever link.
  • The analysis covers several model families, and the evaluation goes beyond perplexity with a broad set of benchmarks.
Critical weaknesses

I think the authors should address three weaknesses, which would affect my score. The main one concerns how the allocation ratios are chosen (1). The other two concern whether the analysis behind them holds in the authors’ own setting. They ask whether the head distribution measured in Qwen3 transfers to 2K training (2), and whether NoPE attention is really position-free (3).

(1) The paper argues that existing FA–LA allocation strategies are largely heuristic, but it is unclear how the proposed approach reduces this reliance on heuristics. In particular, the choice of the 1:3 and 1:7 ratios is not clearly justified, and the paper does not provide a general guideline for choosing allocation ratios in new hybrid architectures.

(2) Linear attention is proposed for local token relationships, as suggested by the RoPE analysis, while NoPE attention is proposed for position-free retrieval. However, the layer-wise head distribution that motivates the hybrid ratios is measured on Qwen3-4B, which was trained with a 32K context, whereas HwH and its baselines are trained with a 2K context. If documents are shorter than the 2K window, the model can in practice learn to process each document independently, so its frequency usage, and with it the number and placement of retrieval heads, may differ at this context length. In that case, the recommended hybrid ratios may not transfer. Applying the head-level RFIS/RPD and QRscore analysis to the authors' own 2K RoPE Transformer baselines would show whether their distribution matches the one observed in Qwen3.

(3) Because each token can attend only to previous tokens, the causal mask allow the model to learn some positional information indirectly, as shown by Haviv et al. (2022) (https://arxiv.org/abs/2203.16634). NoPE heads could therefore also take on local tasks, which would be inefficient. Inductive biases often do not work as intended, so further analysis of this would strengthen the paper.

Questions for authors

Q1: How were the specific ratios derived from the Qwen3 head distribution? Is there a quantitative mapping, for example per-layer retrieval-head fraction mapped by relative depth, and how should practitioners choose ratios for a different depth or scale? (See Weakness 1. I would also like to discuss this in light of Weaknesses 2 and 3, since both affect whether the Qwen3-derived ratios hold in practice.)

Q2: To what extent do the head-level RFIS/RPD and QRscore distributions in your own 2K RoPE baselines align with the patterns observed in Qwen3, measured across relative depth? (See Weakness 2.)

Q3: How are documents packed into training sequences? Are they separated by EOS/BOS tokens, and is cross-document attention masked? This matters for interpreting GPBand, since without document masking the effective context may be closer to the typical document length than to 2K. (See Weakness 2)

5 comments
AuthorsOpenReview ↗

Response to zbfw (1/4): Ratios and transferable design guidance [W1, Q1]

Thank you for asking how to select the allocation for a different model. Appendix F.3 now describes how pretraining could choose the layer-wise proportions under a fixed Q/K/V representation-width budget.

We kindly ask you to first read our global response. It clarifies our core claim and the evidence supporting the two principles.

How were the reported ratios chosen? We used the Qwen3 observations to choose where to allocate more retrieval width. We did not convert measured Head counts into numerical ratios. Inter assigns one of every four layers to NoPE full attention. This gives a global width fraction of 1/41/4, which we used in the middle half of HwH-std. We halved that fraction to 1/81/8 in the remaining non-boundary layers and used only local mechanisms in the first and last layers. These choices give the 1:31{:}3 and 1:71{:}7 global:local Q/K/V width ratios in Appendix E, Table 8. At 380M the full-attention and Gated DeltaNet Heads have different widths, so a Head count alone does not specify the allocation.

What transfers to a new depth or scale? The QR and LD profiles that guided HwH-std show prominent retrieval in the middle layers, with no pronounced retrieval tendency and strong local-streaming behavior in the first and last layers (Figure 1 and Appendix A.2). This observation motivated using only local positional modeling at both ends of HwH-std. Appendix F.3 discusses a possible explanation: local computation can build contextualized retrieval keys early and integrate previously retrieved content before prediction. However, this boundary prior leaves the mix in the intervening layers open. In a 1.3B comparison, the linear-to-full layer ratios with the highest measured RULER recall were 3:13{:}1 for GatedDeltaNet and 6:16{:}1 for HGRN2 (Wang et al., 2025, Table 5). This motivates learning the proportions during pretraining for the mechanisms in use. In Appendix F.3, I propose fixing each layer's total Q/K/V representation width. Each eligible width-hh Head slot has two independently parameterized branches, global NoPE retrieval and local positional modeling. Two learned scalar parameters per slot give softmax weights initialized to (0.5,0.5)(0.5,0.5). The same weights mix the branches' width-hh outputs for every input. An entropy penalty alongside the language-modeling objective encourages one weight to approach one. Once choices have stabilized across the network, retaining the higher-weight branch at each slot and pruning the other gives a deterministic architecture. Both the layer-wise ratios and the overall retrieval share emerge from training.

What is established empirically? In our fixed configurations, dense RULER NIAH-Single-2 means were 60.0660.06\\% for HwH-rmfl and 68.4268.42\\% for HwH-std, with the latter using less global width (Table 2). These results establish the performance of the reported schedules. Learning the schedule during pretraining is a proposal in Appendix F.3. Its performance on new models remains to be evaluated. The following replies address our own 2K diagnostics and the interpretation of NoPE.

AuthorsOpenReview ↗

Response to zbfw (2/4): Diagnostics in our own 2K models [W2, Q2]

We have added the requested analysis in our own 380M Transformers trained at 2K, including both the QKNorm model and the original Transformer without QKNorm (Appendix B.4, Functional differentiation in our pretrained Transformers).

Across depth: Figure 11. QRscore follows the query-focused retrieval-head analysis of QRHead: Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking (arXiv:2506.09944); Appendix A specifies our probe samples and scoring. We show head-wise QRscores in their original order and sorted within every layer for each model. High retrieval scores concentrate at intermediate depths in both, consistent with the layer-specific pattern seen in Qwen3-4B. This provides a direct qualitative comparison across depth without turning continuous retrieval scores into arbitrary binary Head counts.

Frequency dependence and behavior: Figure 12. We align RFIS/RPD heatmaps with the QRscores of the same Heads in Layer 11 of the model with QKNorm and Layer 12 of the model without QKNorm. Several high-QRscore Heads use low frequencies with substantial RFIS and relatively weak RPD. Other Heads emphasize the salient band and can also exhibit isolated low-frequency RFIS peaks. Thus, the small models exhibit the same implicit functional differentiation and mixed Head behavior that motivate our interpretation of the larger pretrained models.

Training-scale dependence: Figure 3. The band shifts with the RoPE base at a fixed training length and shifts again when the training length changes from 2K to 4K. Each diagnostic uses its model's training length. The recurrence of the functional pattern does not require the band to occupy the same group index, or the Heads to have identical counts and scores, across scales and context lengths.

These observations address the transfer concern empirically: the phenomena are present in the models trained under our experimental setup. We use them to motivate layer-specific allocation and test that choice through the architectural ablations. We do not assume that Qwen3's exact Head distribution transfers to 2K or determines an optimal hybrid ratio. The training-sequence construction and the effect of document boundaries are addressed in the next response.

We have also included the complete diagnostic visualizations redrawn in the paper's style and original QR/LD/RFIS/RPD score files in the updated supplementary materials, covering our 380M Transformers and the larger pretrained models. These retain all measured layer/Head/frequency grids beyond the selected manuscript panels; the indexes identify the available metrics for each configuration. Please feel free to check MODEL_CONSISTENCY_GUIDE.txt, data/diagnostics/, and figures/diagnostics/ for the full cross-scale comparison.

AuthorsOpenReview ↗

Response to zbfw (3/4): Data construction and document boundaries [W2, Q3]

Dataset construction. Our corpus is the sample-100BT subset of FineWeb-Edu, introduced in The FineWeb Datasets (arXiv:2406.17557). FineWeb-Edu filters web documents from FineWeb using an educational-quality classifier trained on Llama-3-70B-Instruct annotations, retaining documents at a score threshold of 3. The released subset is randomly sampled and contains approximately 100B GPT-2-counted tokens. This corpus-level count is separate from the training tokens obtained with our tokenizer. The subset contains documents, not prepacked 2K examples; the training framework constructs sequences after tokenization. See the FineWeb-Edu dataset card.

Exact packing and attention in our experiments. We train with Flame: Flash Language Modeling Made Easy, a torchtitan-based training framework integrating Flash Linear Attention (FLA) and Transformers. We use its default fixed-length setting, varlen=false, with a sequence length of 2,048. The OnlineTokenizedIterableDataset tokenizes each document separately, appends its tokens to a stream, and takes consecutive 2,048-token blocks, carrying the remainder forward. Cutting does not respect document boundaries: a block can contain multiple documents, and a document can continue into the next block.

The tokenizer prepends BOS <s> (ID 1) to each document and does not append EOS </s> (ID 2). The concatenation procedure inserts no additional separator. A boundary is therefore marked by the next document's BOS. Because blocks are cut at a fixed length, a training sample need not begin with BOS.

With varlen=false, the collator does not construct document-level cu_seqlens. Each block is one causal sequence with positions 0-2,047 and no padding. Cross-document attention is not masked, and BOS is not an attention boundary. This is Flame's standard fixed-length path; see its data pipeline, training loop, and default tokenizer configuration.

Comparison with production LLM training. This shares the fixed-sequence-budget autoregressive training setup used by models such as Qwen3. The Qwen3 Technical Report (arXiv:2505.09388, Section 3.2) specifies 4,096-token sequences in the first two stages and 32,768-token sequences in the long-context stage. For the latter, 75% of texts are 16K-32K long and 25% are 4K-16K. Its report does not disclose the exact BOS/EOS packing or cross-document masking implementation, so those details cannot be equated with Flame's from the report alone. Our controlled experiment simplifies the training curriculum to a known fixed length, allowing the learned positional scale to be examined directly.

Can the model learn document-aware behavior? Yes, and this can coexist with training-scale positional dependence. In our setup, each query still scores all preceding keys in the 2K block. A document boundary can signal that some context is irrelevant, but it does not remove distant keys from the attention computation. Distinguishing nearby, useful context from distant context can therefore involve positional dependencies over the available training span, even when useful semantic dependencies are mostly within documents. Typical document length and the span over which attention must distinguish positions are not interchangeable.

The interpretation is supported by measurements, not packing alone. Figure 3 shows the band changing with training length and RoPE base; Figures 11-12 show the same qualitative QR/RFIS/RPD relationships in our own 380M, 2K models. Thus, document boundaries do not invalidate the functional differentiation observed in this setting, and the architectural tests for validating the two principles do not rely on transferring exact Head counts from a long-context Qwen3 model.

Reviewer_XswJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes two metrics (RFIS and RPD) to measure how attention heads use RoPE frequencies, and finds two head types: retrieval heads (low, position-independent frequencies) and positional heads (a training-length-dependent band). Based on this, it builds HwH, which mixes NoPE full attention and Gated DeltaNet heads within each layer. HwH keeps language modeling quality and improves length extrapolation on NIAH over Transformer, GDN and layer-wise hybrid baselines.

Critical strengths

1.The question is relevant and timely. The paper grounds the allocation of full and linear attention in hybrid models, currently chosen heuristically, in a head-level analysis of RoPE frequency use in pretrained Transformers.

  1. RFIS and RPD are well defined and bounded. RPD separates frequency importance from positional dependence.

  2. The analysis is checked across several models and families.

Critical weaknesses
  1. The claim that RFIS and RPD establish a complete two-type taxonomy is not quantitatively supported. It rests on visual inspection of heatmaps for two layers per model, with no classification rule and no head counts. Moreover, near-zero RPD at low frequencies may arise from RoPE itself rather than from learned behavior, since very slow rotations are close to the identity. The authors should compare against an untrained (randomly initialized) model to show that this pattern is learned.

  2. The extrapolation claim rests on narrow evidence. Length generalization is evaluated only on the three single RULER NIAH tasks. The real-world retrieval tasks do not appear to test long contexts, and all results come from single training runs. Evaluating the existing models on the remaining RULER tasks would strengthen this claim.

  3. The experiments do not show which design choice explains the gains. HwH-uni and Inter use the same FA budget and differ only in granularity, but no direct comparison is provided, and in Figure 15 HwH-uni does not appear to extrapolate better than Inter. Please report both models, adding 380M real-world retrieval for Inter and RULER NIAH for both.

Questions for authors
  1. What is the RPD of low-frequency groups in a randomly initialized or early-training model with the same RoPE settings? How much of the near-zero RPD is explained by long rotation periods alone?

  2. Could it be tested directly, for example by modifying only the GPBand frequencies in a trained RoPE model and measuring NIAH beyond the training length?

  3. The released evaluation script truncates each real-world retrieval context to about 2K tokens around the answer. Was this the setting used in the paper?

5 comments
AuthorsOpenReview ↗

Response to XswJ (1/3): Taxonomy and slow-rotation geometry [W1, Q1]

Thank you for distinguishing frequency importance from positional dependence.

We kindly ask you to first read our global response. It clarifies our positioning, core claim and the reasoning and experimental evidence supporting it, providing the context for our responses to your specific comments below.

We agree that near-zero low-frequency RPD, considered alone, does not show learned specialization. The revision clarifies both this point and the object of the taxonomy.

Functional differentiation, rather than exclusive Head bins. RFIS and RPD characterize retrieval and positional modeling in RoPE-frequency dependence; many Heads exhibit both patterns. Section 3 no longer requires each Head to be assigned exclusively to one category. We use continuous diagnostics and their association with QRscore, rather than a binary threshold or a Head count. The claim of a complete mechanism-level taxonomy concerns these complementary functions. It does not imply a complete partition of Heads. The full-depth QR patterns and layer-mean frequency analyses complement the representative head-level panels (Figures 1-3; Appendices A-B).

How much is explained by slow rotations? The near-identity behavior of sufficiently slow RoPE rotations over the observed offsets is a geometric property, so low RPD alone does not establish learned functional differentiation. Our interpretation concerns the selective use of these channels by trained Heads: high-QRscore Heads often depend strongly on slow channels according to RFIS, while other Heads depend on position-sensitive bands. RFIS establishes that a channel matters; RPD tests whether its rotary modulation matters. Their combination and correspondence to retrieval behavior support the functional interpretation.

Random initialization and additional evidence. We have not added random-initialization or early-training RPD measurements. The revised explanation acknowledges the geometric contribution directly instead of treating low RPD as proof of a learned phenomenon. We have added analyses of our own 380M, 2K Transformers with and without QKNorm: Figure 11 shows head-wise and sorted QRscores across depth, and Figure 12 aligns RFIS/RPD with QRscore in Layer 11 with QKNorm and Layer 12 without QKNorm. Both exhibit the same qualitative functional differentiation. The GPBand interventions in Table 1 and Appendix B.6 further test how the learned use of positional frequencies relates to task performance.

AuthorsOpenReview ↗

Response to XswJ (2/3): Evaluation scope and direct baselines [W2-W3, Q3]

W2: What do the RULER NIAH experiments establish? Our RULER NIAH (arXiv:2404.06654) evaluation follows an established protocol at comparable model scales. Gated Delta Networks (1.3B; arXiv:2412.06464v3, Section 3.2/Table 2) and Mamba-3 (1.5B; arXiv:2603.15569, Table 4) report the same three NIAH-Single tasks; neither reports multi-key or full-suite RULER results. Their comparisons include Mamba-2 baselines. GDN explicitly uses single-needle retrieval as a case study to distinguish memory mechanisms.

For our 380M and 1.4B pretrained models, these tasks already reveal substantial architectural differences. Recovering a fact beyond the training length tests extrapolation of global access, a necessary capability for more complex long-context tasks. Multi-key and broader tasks add competing associations, reasoning, or output requirements. If a model has not mastered those requirements within its training window, failure at longer lengths cannot cleanly diagnose extrapolation. Our choice therefore combines an established evaluation suite with a controlled, interpretable test of the proposed principles; the evidence comes from matched architectural interventions, not task difficulty alone.

We report all three NIAH-Single tasks at 1K/2K/4K. Figure 4 adds eight 380M architectures over 15 lengths (1K-8K), 11 depths, and 10 examples per cell. Figure 5 extends the 1.4B comparison to 32K with identical SWAN-GPT (arXiv:2504.08719) query scaling, α=300\alpha=300. Dense-grid and task-level sampling are distinguished. Inter-swa isolates organization from component replacement (Appendix E.2). These are single-run qualitative comparisons; Appendix D separately explains the small-model 1K response failures.

Separately, real-world cloze tasks support retrieval gains: Inter/HwH-std exceed Transformer at both scales (28.97/28.98 versus 24.95; 43.08/44.64 versus 40.17). The NIAH heatmaps concern extrapolation and its depth robustness, including a possible lost-in-the-middle connection.

Q3: Was answer-centered cropping used? Yes. Both cited studies also truncate their six real-world cloze tasks to 2K (GDN, Table 4; Mamba-3, Table 4), including their Mamba-2 baselines. GDN's Appendix B.1 and Mamba-3's Appendix D explain why cloze completion suits models without instruction tuning: it matches next-token prediction, while original QA formats can be challenging for solely pretrained models. Our protocol follows this established pretrained-model evaluation practice.

We use EleutherAI's Language Model Evaluation Harness (lm-eval-harness; arXiv:2405.14782) with cloze task extensions and datasets from Based (arXiv:2402.18668) and Just Read Twice (JRT; arXiv:2407.05483). These tasks assess retrieval from a supplied passage. JRT's Appendix B.2 describes answer-preserving truncation; its task implementation uses a 48-token generation limit and answer-containment scoring. Our six-task evaluation follows this cloze protocol.

Our 2,048-token budget reserves 48 tokens for generation and up to 48 for the question suffix and tokenization boundaries. Cleaned contexts longer than 1,952 tokens are cropped around the answer; shorter contexts retain their original length, giving prompts of at most 2,000 tokens. Appendix D, Table 7 reports the original prompt-length statistics and cropping counts. These results assess retrieval within the training-length budget; beyond-training-length evidence comes from RULER NIAH.

W3: Direct Inter versus HwH-uni comparison. You are correct that the previous presentation did not make this comparison sufficiently clear. Table 2 now reports both models together, Figure 4 provides both heatmaps, and Appendix E.1, Table 10 gives all six cloze tasks for Inter and the HwH variants. The direct comparisons are below (percent; Grid NIAH is the dense heatmap mean).

ModelCSRetr.Grid NIAH
Inter42.0728.9749.52
HwH-uni42.9327.1049.39
HwH-rmfl42.8826.9960.06
HwH-std42.7828.9868.42

This result is informative: uniform within-layer mixing alone does not improve extrapolation. The additional gains support the layer-specific allocation required by Principle 1. HwH-rmfl removes global retrieval from the first and last layers, raising the grid mean to 60.06%; HwH-std further reduces retrieval allocation in shallow and deep layers outside the middle half, reaching 68.42%. This is why Principle 1 combines separate functional assignment, within-layer mixing, and layer-specific allocation. The revised Section 4 explicitly discusses this sequence instead of attributing the gains to granularity alone.

AuthorsOpenReview ↗

Response to XswJ (3/3): Direct GPBand intervention [Q2]

We added your requested post-training intervention and a complementary pretraining comparison. The first tests learned positional dependence; the second tests how the availability of GPBand during learning affects extrapolation. All scores are percentages; CS and Retr. are commonsense and six-task cloze means, and S1/S2/S3 denote NIAH-Single-1/2/3.

1. Post-training intervention: what the model has learned (Appendix B.6, Table 6). We replace GPBand rotations (groups 21-22) with the identity in Table 1's trained full-RoPE checkpoint, retaining all Q/K content dimensions, learned weights, and the causal mask. The intervention removes rotary positional modulation while preserving the content-bearing dimensions. Global response (2/3) reports all ten adjacent-group masks; the key contrast is:

MaskCSRetr.FDASWDENQSQuADTriviaQADROP
None42.6627.7217.6029.3414.6734.6549.5320.51
21-2242.6321.572.1817.648.7733.6548.4618.74

GPBand masking gives the largest mean retrieval loss while short-context commonsense remains nearly unchanged. FDA, SWDE, and NQ each lose more than five points. They are the only tasks containing original prompts beyond 2048 tokens; their longer examples are cropped to the 2K budget, with prompts capped at 2000 tokens (Table 7). The other three tasks remain shorter and each lose less than two points. Other adjacent-pair masks do not cause comparable losses on FDA/SWDE/NQ.

This directly demonstrates reliance on GPBand for positional computation already learned by the model. Together with RFIS/RPD and GPBand's shift with training length, it supports the conclusion that global positional modeling fits the training sequence length. Removing a learned representation damages that computation within the training span; recovery of extrapolation is not the expected outcome of this post-training test.

2. Pretraining intervention: consequences for extrapolation (Table 1). Both restricted variants disable slower rotations below GPBand. Their targeted difference is whether groups 21-22 retain RoPE during learning: 00-2222 retains them, whereas 00-2020 withholds them.

RoPECSRetr.S1@4KS2@4KS3@4K
full42.6627.720.211.02.8
0-2242.4930.780.011.80.0
0-2042.6330.2498.283.226.6

Without GPBand, 00-2020 gains substantial extrapolation with comparable commonsense reasoning and real-world retrieval performance relative to 00-2222. Thus, these gains do not accompany the training-span retrieval losses caused by removing an already learned representation. This provides evidence that making GPBand available during learning can impair extrapolation.

3. Why not use 00-2020 directly? The dense heatmaps of the same checkpoints (Appendix B.5, Figure 13) show 99.09% at context budget 2560, 23.64% at 3072, 80.00% at 3584, 82.73% at 4096, and 10.91% at 8192. Task-level and dense-grid sampling differ; the dense comparison directly establishes that the improvement is not stable across lengths.

The position-sensitive mechanism still has global access. Although every retained rotation's period is below the training length, higher-frequency combinations may still support residual global positional fitting. Removing GPBand does not constrain positional modeling to local interactions. Together with the NoPE/RoPE and GDN/SWA architectural controls, these results support Principle 2's separation of local positional modeling from position-independent global retrieval.

Reviewer_uWN4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper is interested in the subject of hybrid architecture that combines Full Attention and Linear Attention, and that are typically based on heuristic decision choices about how to allocate each mechanism. The author propose a mechanistic study and analysis to better understand these choices. They propose two alternatives to measure how removing particular RoPE frequency and replacing rotary modulation with a position independent and its effects on the overall attention distribution.

Critical strengths
  • I think the overall considered problem of finding better ways to decide on the architectural choices is a very interesting direction and specifically in the case of the hybrid architectures that the authors are interested in.
  • I find the intervention metrics and how the authors connect them to have some principled architecture design very innovative. Specifically both the RFIS and RPS seems to be distinguishing important element about the attention distribution.
  • The considered analysis is based on different pretrained model families, including Qwen models, Llama and Gemma providiong therefore an idea about the generalization of the insights.
  • It’s definetly very informative to include all the ablation studies on the importance of NoPE versus RoPE, GDN in comparison to the sliding-window attention.
Critical weaknesses
  • The main claim about the RFIS and RPD is not demonstrated in my opinion. Specifically the experiments have indeed identified two dominant frequency-dependence patters but that doesn’t necessary establish that every head belongs exclusively to one of these identified head taxonomy.
  • Related to the previous point, the overall analysis doesn’t establish a clear causality relationship between the GPBand and length-extrapolation failure. Again, the experiments have clearly establish such link and relationship, yet, the claim that the positional fitting causes poor length extrapolation remain from my understanding a hypothesis.
  • The efficiency claim is yet to be confirmed as well. The idea of combining the FA and LA is to have better model’s efficiency. However, while the authors have reported some dimension ratios (in terms of the Q/K/V), they didn’t provide a clear empirical comparison of training efficiency or memory consumption or inference latency.
  • Regarding the experimental improvement, in some specific use-cases, the improvement is very small. While obviously I understand the overall goal of the analysis is to kind of have a principled way to choose the architecture, if it doesn’t produce improvement over specific heuristics, then I am not very confident about its usage and worth and would be great if the authors could clarify this.
Questions for authors

Overall the paper is very interesting and would like to see these questions answered to have a better idea about the manuscript and its worth.

  • I am confused about how the authors distinguish retrieval heads from positional heads? And are there heads that kind of show a mixed behavior?
  • Referring to the weakness I cited, it would be great if the authors can show causaly that manipulation the GPBand frequency directly change the length extrapolation, rather than just a correlation.
  • In terms of efficiency, can the authors provide some elements on the overall inference efficiency improvement?
  • Regarding the performance, are the small language-modeling extendable? Is there a better way to derive principled ideas about the architecture?
11 comments
AuthorsOpenReview ↗

Response to uWN4 (1/3): Functional differentiation [W1, Q1]

Thank you for highlighting both the value of the intervention metrics and the ambiguity in our original taxonomy claim.

We kindly ask you to first read our global response. It clarifies our positioning, core claim and the reasoning and experimental evidence supporting it, providing the context for our responses to your specific comments below.

We agree that two dominant frequency patterns do not establish an exclusive partition of every Head. The revised paper explicitly distinguishes complete functional differentiation from strict separation at the Head level.

How do we distinguish the functions? RFIS removes one frequency contribution from a Head's logits and measures the change in its complete attention distribution. RPD instead replaces that contribution's relative rotation with the identity, preserving its learned Q/K coordinates. Consequently, high RFIS with weak RPD indicates information that matters without substantial dependence on the rotary modulation over the observed offsets. This pattern is associated with high retrieval QRscores, especially below GPBand. At GPBand and above, RFIS and RPD often track one another, showing that the important contributions depend on position (Sections 3.2-3.3; Figures 1-2).

Can a Head exhibit both? Yes. Many Heads use both frequency regimes. Heads emphasizing positional dependence may also have sparse low-frequency RFIS peaks. We therefore do not impose a hard classification threshold, claim that all Heads belong to disjoint sets, or equate every low-frequency contribution with retrieval. The completeness claim concerns the mechanism-level distinction between retrieval and positional modeling in the frequency dependence under study. The patterns and their association with QRscore characterize predominant functional tendencies within potentially mixed Heads.

The revised Section 3 also explains why this matters for design. RoPE Transformers implicitly learn differentiated uses of their frequency channels, but the separation within individual Heads is incomplete. This motivates explicitly assigning the functions, mixing them within layers, and allocating them differently across depth. Their effectiveness is then tested by architectural interventions, rather than inferred from a taxonomy alone.

We have additionally examined our own 380M Transformers trained at 2K, with and without QKNorm. Their all-layer QR visualizations and RFIS/RPD/QR comparisons at the selected layers show the same qualitative trends (Appendix B.4, Figures 11-12). Cross-family analyses remain in Appendices A-B. These additions support the functional interpretation while avoiding the exclusive Head classification that concerned you.

AuthorsOpenReview ↗

Response to uWN4 (2/3): GPBand interventions and extrapolation [W2, Q2]

We added interventions before and after learning. The two interventions address different questions: post-training intervention tests the positional dependence already learned by the model; pretraining intervention tests how its availability during learning affects extrapolation. Scores below are percentages; CS/Retr. are task means and S1/S2/S3 denote NIAH-Single-1/2/3.

1. Post-training intervention: learned positional dependence (Appendix B.6, Table 6). Following your request, the GPBand rotations (groups 21-22) are replaced with the identity in the trained full-RoPE Transformer. All Q/K content dimensions and learned weights are preserved; only rotary positional modulation in these groups is removed. Global response 2 reports all ten adjacent-pair masks; the comparison below includes all six retrieval subtasks.

MaskCSRetr.FDASWDENQSQuADTriviaQADROP
None42.6627.7217.6029.3414.6734.6549.5320.51
21-2242.6321.572.1817.648.7733.6548.4618.74

GPBand masking causes the largest mean retrieval loss among the tested masks, while short-context commonsense remains nearly unchanged. FDA, SWDE, and NQ are the only tasks containing original prompts longer than 2048 tokens; their longer examples are cropped to fit the 2K budget, with prompts capped at 2000 tokens (Table 7). These three tasks each lose more than five points, whereas the three shorter tasks each lose less than two. Other adjacent-pair masks do not cause comparable losses on FDA/SWDE/NQ.

Together with RFIS/RPD and GPBand's shift with training length, this shows reliance on positional computation fitted to the training span. It supports the conclusion that a model with global positional modeling fits the training sequence length. Removing that learned representation damages useful computation; recovery of extrapolation is not the expected outcome of this post-training test.

2. Pretraining intervention: effect on extrapolation (Table 1). Both 00-2222 and 00-2020 disable rotations below GPBand. Their targeted architectural difference is whether GPBand rotations remain available during learning.

RoPECSRetr.S1@4KS2@4KS3@4K
full42.6627.720.211.02.8
0-2242.4930.780.011.80.0
0-2042.6330.2498.283.226.6

Withholding GPBand changes very poor extrapolation into substantially better, though unstable, extrapolation. Crucially, 00-2020 achieves comparable commonsense reasoning and real-world retrieval relative to 00-2222, while its training-length RULER NIAH performance is maintained or improved. Thus, these gains do not accompany the broad in-window breakdown that might be inferred from the post-training intervention. This comparison provides evidence that making GPBand available during learning can impair extrapolation in the tested setting.

3. Why the improvement can remain unstable (Appendix B.5, Figure 13). Every remaining rotary frequency in 00-2020 completes a cycle within the training span, but combinations of these frequencies can still distinguish global offsets. The model may therefore retain some global positional dependence, providing a possible explanation for the nonmonotonic failures, including the pronounced drop at 3072 tokens. Table 1 and the dense heatmaps establish the key result: withholding GPBand substantially improves extrapolation but does not ensure stable generalization across lengths. The position-sensitive mechanism still has global access; deleting one band does not restrict positional modeling to local interactions.

4. Architectural tests directly support Principle 2. Keeping allocation fixed while replacing global NoPE FA with RoPE FA reduces the dense NIAH-Single-2 mean from 68.42% (HwH-std) to 22.42% (HwH-std-r), with CS 42.78% and 42.64%. Replacing GDN with a hard 128-token local window increases the extrapolation mean to 89.88% (HwH-swa). These controls support separating local positional modeling from position-independent global retrieval.

Summary. GPBand is a measured signature of training-scale positional fitting. Post-training intervention establishes reliance on a learned positional representation; pretraining intervention provides evidence that its availability can impair extrapolation. Together with the architectural tests, these results support the principle without requiring GPBand alone to explain every extrapolation failure.

AuthorsOpenReview ↗

Response to uWN4 (3/3): Benefits, scalability, and efficiency [W3-W4, Q3-Q4]

W3/Q3: Efficiency. We appreciate that the original hybrid-architecture framing suggested a systems-efficiency contribution. We have not measured training throughput, memory consumption, or inference latency, and do not claim an empirical improvement in those quantities. Our use of FA and LA assigns complementary functional roles: NoPE FA provides global retrieval, while GDN provides a local positional mechanism. HwH-swa tests the same principles with a different local mechanism. The revised manuscript foregrounds this functional objective and its modeling/extrapolation evidence. The reported Q/K/V allocations describe the controlled architectural choices; they are not substitutes for efficiency measurements.

W4/Q4: Where are the benefits over heuristics? Real-world cloze tasks establish retrieval gains; RULER NIAH establishes length-extrapolation gains, alongside competitive language modeling and reasoning. The revised Table 2 and Figure 4 make the necessary comparisons explicit:

Scores are percentages; Grid NIAH averages the 15-length/11-depth heatmap.

ModelCSRetr.Grid NIAH
Transformer42.3024.9513.82
GDN42.1820.0242.36
Inter42.0728.9749.52
HwH-uni42.9327.1049.39
HwH-rmfl42.8826.9960.06
HwH-std42.7828.9868.42
HwH-std-r42.6423.6722.42
HwH-swa42.1329.3289.88

Inter already improves retrieval and extrapolation over Transformer. Uniform within-layer mixing alone (HwH-uni) gives no extrapolation gain; the layer-specific HwH-rmfl and HwH-std improve it despite lower FA allocation. At 380M, HwH-std and Inter have essentially the same cloze retrieval, so we do not present that small difference as a gain.

These contrasts support Principle 1's combined requirements of explicit functional separation and layer-specific within-layer mixing. The NoPE/RoPE and GDN/SWA interventions separately test Principle 2, as detailed in our preceding response.

Does the result extend beyond small models? At 1.4B, real-world cloze retrieval rises from 40.17% (Transformer) to 43.08% (Inter) and 44.64% (HwH-std). Separately, HwH-std improves length extrapolation over Inter: 4K NIAH-Single-2 rises from 79.6% to 99.6% (Tables 3-4). Language modeling and commonsense remain competitive; the commonsense means are 53.05% for Inter and 52.70% for HwH-std. These evaluations support retrieval and extrapolation gains, respectively.

We additionally train 1.4B HwH-swa and compare three NoPE hybrids over 1K-32K using inference-time logarithmic query scaling following SWAN-GPT (arXiv:2504.08719), with the same α=300\alpha=300 and no additional training (Figure 5). HwH-swa reaches 97.27% at 16K and 92.73% at 32K, whereas Inter and HwH-std fail at 8K and beyond in this setting. HwH-swa has weaker short-context language modeling, making the role of the local mechanism visible rather than obscuring the tradeoff.

The NIAH extrapolation heatmaps at 4K show fewer middle-depth failures in HwH-std than Inter at both scales. This suggests a possible lost-in-the-middle connection within extrapolation, to be tested beyond synthetic NIAH. RFIS/RPD also offer practical tools for broader mechanism analysis.

Finally, the new 380M Inter-swa control reaches 10.9% at 6K versus HwH-swa's 90.0%, despite the same 128-token local window (Appendix E.2). Thus, the extrapolation result is not explained by SWA replacement alone. Together, the two scales and targeted controls show how the principles guide design beyond merely selecting a hybrid ratio. The exact ratios remain implementation choices, while the functional roles and organization are the intended transferable guidance.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Global response: Intended contribution and clarification

We thank all three reviewers for their constructive feedback.

We kindly ask all reviewers to read this three-part global response first. It explains our core claim and the reasoning and experiments that support it.

The original emphasis on hybrid-architecture design obscured our intended contribution. Under the revised title Two Principles for Autoregressive Language Modeling, the argument proceeds from functional differentiation in Transformers, to two principles, to controlled architectural tests.

1. The main claim and the role of HwH. We seek guidance for improving LLM architectures by examining the functional differentiation implicitly learned by RoPE Transformers and testing whether making it explicit is beneficial. The two principles are:

  • Functional organization: Retrieval and positional functions should be assigned separately and mixed within layers with layer-specific allocation.
  • Positional dependence: Positional modeling should operate only locally, while global access should be implemented through position-independent retrieval.

HwH is the experimental framework used to test these principles. HwH-std implements global retrieval with NoPE Full Attention (FA) and local positional modeling with Gated DeltaNet (GDN), with different allocations across depth. Its variants change the allocation or components to test aspects of the principles. The aim is to identify functional requirements and their organization, rather than a universal FA:LA ratio or an efficiency frontier.

2. Complete functional differentiation does not mean an exclusive classification of Heads. The original wording could suggest that every Head belongs unambiguously to one of two categories. Our claim concerns a complete mechanism-level taxonomy of retrieval and positional modeling in the RoPE-frequency dependence we analyze. Heads tend to specialize, but many exhibit both functional patterns. A Head with strong positional dependence can also have isolated low-frequency importance; low-frequency use alone is not a classification rule.

RFIS is the primary diagnostic: it measures how removing a frequency contribution changes the complete attention distribution. RPD preserves the learned Q/K coordinates and removes only that frequency's rotary modulation, identifying whether its importance depends on position. Combined with QRscore as a behavioral retrieval probe, they distinguish important, weakly rotation-dependent contributions below the salient band from position-dependent contributions at the band and above. Small RPD for very slow rotations is consistent with RoPE geometry; the learned phenomenon is how trained Heads use these available frequency regimes. These findings motivate explicit functional separation and its layer-specific organization, without requiring binary Head labels.

3. GPBand characterizes training-scale positional fitting. The salient Global Positional Band changes with training length and RoPE base. Its location agrees with the Frequency Bands variance-maximization criterion over the training span (Section 3.4 and Appendix B.4). GPBand is evidence of the positional scale used by the trained model. It is not a frequency switch whose removal from an already trained model must improve extrapolation: removing a learned dependency can instead damage retrieval within the training range. We use post-training intervention to test learned dependence, then pretraining restrictions to examine its development and effect on extrapolation. A frequency mask can improve selected lengths without implementing local positional modeling.

4. The broader implication. The Conclusion gives four progressively broader directions:

Level 1 - Practical tools and architecture. Apply RFIS/RPD to Transformer analysis. Derive practical rules for functional allocation or develop strategies that let models learn their own allocations during pretraining (Appendix F.3).

Level 2 - Understanding existing gains. Explain hybrids' advantages over Transformer. NIAH depth profiles may offer clues about lost-in-the-middle.

Level 3 - Promising research directions. Abandon global positional modeling for better extrapolation; develop brand-new Linear Attention and related recurrent mechanisms for expressive local positional modeling. For illustration, We suggest parallelizable finite-window nonlinear local modeling for future work.

Level 4 - Functional hybridization. Combine better global retrieval and local modeling mechanisms, potentially beyond stacked multi-head attention. The aim is architectural innovation driven by functional requirements, beyond an efficiency-quality tradeoff.

As the first author, I hope to give architectural research clearer direction and reduce unproductive investment of time and compute. HwH may be superseded; I want the principles and evidence to provide lasting guidance about what to build next and why. I explain this motivation in Appendix G.

AuthorsOpenReview ↗

Global response: GPBand evidence and extrapolation stability

The revised Section 3 first uses post-training intervention to show that the full-RoPE model relies on positional computation adapted to the training span. It then uses pretraining intervention to examine how the availability of GPBand during learning affects extrapolation, followed by dense length testing to assess whether the improvement is stable. All models are 380380M QKNorm Transformers trained at 2K. All scores below are percentages; CS and Retr. denote the commonsense and six-task cloze means, and S1/S2/S3 denote NIAH-Single-1/2/3.

1. Post-training intervention: learned positional dependence (Appendix B.6, Table 6). We replace the rotations of two adjacent groups with the identity, retaining all Q/K content dimensions, learned weights, and the causal mask. Only rotary positional modulation in those groups is removed. Groups 21-22 form GPBand.

MaskCSRetr.FDASWDENQSQuADTriviaQADROP
None42.6627.7217.6029.3414.6734.6549.5320.51
1-242.4825.4314.7925.2912.7332.1747.8719.74
5-642.4925.9212.4328.1713.4032.9448.2820.32
9-1042.2525.2812.7928.5314.4430.1646.4519.31
13-1442.1223.8112.7927.0013.4325.8745.6218.16
17-1842.4523.4410.2525.7412.8028.0246.5617.25
19-2042.8127.3513.9729.2513.6235.8250.1221.32
21-2242.6321.572.1817.648.7733.6548.4618.74
23-2442.7424.8712.7923.0412.4833.0448.8219.02
25-2642.7326.4216.4226.1913.9733.6148.8219.50
29-3042.7627.6418.0629.0714.7034.5549.3520.12

GPBand masking causes the largest retrieval loss, while short-context commonsense remains nearly unchanged. FDA, SWDE, and NQ are the only tasks containing original prompts beyond 2048 tokens; longer examples are cropped to the 2K budget, with prompts capped at 2000 tokens (Table 7). Each loses more than five points; the three shorter tasks each lose less than two. Other adjacent-pair masks do not cause comparable losses on FDA/SWDE/NQ.

Together with RFIS/RPD and GPBand's shift with training length, these losses show reliance on positional computation fitted to the training span. This supports the conclusion that global positional modeling fits the training sequence length. Damaging this learned dependence impairs in-window performance; recovery of extrapolation is not the expected outcome of this test.

2. Pretraining intervention: effect on extrapolation (Table 1). Both restricted variants disable slower rotations below GPBand; 00-2222 retains groups 21-22, whereas 00-2020 withholds them.

RoPECSRetr.S1@4KS2@4KS3@4K
full42.6627.720.211.02.8
0-2242.4930.780.011.80.0
0-2042.6330.2498.283.226.6

Withholding GPBand substantially improves extrapolation with comparable commonsense reasoning and real-world retrieval performance relative to 00-2222. These gains occur without a broad in-window performance collapse. This provides evidence that making GPBand available during learning can impair extrapolation.

3. Why GPBand removal is insufficient (Appendix B.5, Figure 13). The same checkpoints use Figure 4's dense protocol: 15 context budgets from 1K to 8K, 11 depths, and 10 examples per cell. Selected depth-averaged NIAH-Single-2 scores follow; this sampling differs from the task-level scores above.

Context budgetfull0-220-20
256076.3689.0999.09
307231.8217.2723.64
358411.8215.4580.00
409613.6414.5582.73
512010.919.0915.45
81920.001.8210.91
Grid mean18.8521.0353.64

The 00-2020 model collapses at 3072, recovers at 3584-4096, then falls again. Its position-sensitive mechanism retains global access. Every retained rotation has a period below the training length (the slowest is about 1987 tokens), but their combinations may still support global positional fitting. The experiments establish substantial gains without stable generalization across lengths. Together with the architectural controls in Global response (3/3), these results support Principle 2: local positional modeling with position-independent global retrieval.

AuthorsOpenReview ↗

Global response: Architectural evidence and manuscript changes

1. Direct architectural tests at 380M (Table 2; Figure 4). The revision organizes all eight models around functional allocation and component choice. CS: commonsense; Retr.: real-world cloze retrieval; Grid NIAH: extrapolation, averaged over 15 lengths and 11 depths. All scores are percentages.

ModelAllocationGlobalLocalCSRetr.Grid NIAH
TransformerimplicitRoPE FA-42.3024.9513.82
GDNpure GDN-GDN42.1820.0242.36
Interacross layersNoPE FAGDN42.0728.9749.52
HwH-uniuniform withinNoPE FAGDN42.9327.1049.39
HwH-rmflno boundary FANoPE FAGDN42.8826.9960.06
HwH-stddepth-specificNoPE FAGDN42.7828.9868.42
HwH-std-rdepth-specificRoPE FAGDN42.6423.6722.42
HwH-swadepth-specificNoPE FASWA12842.1329.3289.88

No boundary FA removes retrieval from the first/last layers; depth-specific additionally lowers its width fraction outside the middle half. These HwH allocations mix components within layers. GDN provides decay-based locality and can retain distant information; SWA128 enforces a hard local window. Table 2 maps these configurations to all five aspects of the two principles.

Inter already improves retrieval and extrapolation over Transformer. Uniform within-layer mixing alone does not improve extrapolation: Inter and HwH-uni score 49.52 and 49.39. Layer-specific allocation raises this to 60.06 and 68.42. Global RoPE lowers this mean to 22.42. Keeping positional dependence more strictly local raises it to 89.88. Short-context scores need not improve in the same order. These contrasts distinguish the benefits of allocation and positional dependence.

2. Added checks across settings. Figures 11-12 now diagnose our own 380M, 2K Transformers with and without QKNorm: all-depth QR plots and RFIS/RPD/QR correspondence in Layer 11 with QKNorm and Layer 12 without it show the same qualitative functional differentiation. This supports testing the principles at the actual experimental scale.

Real-world retrieval is 24.95/28.97/28.98 for Transformer/Inter/HwH-std at 380M and 40.17/43.08/44.64 at 1.4B: separation improves retrieval at both scales; HwH-std adds a gain at 1.4B. Tables 3-4 show competitive modeling/reasoning.

Separately, NIAH heatmaps at 4K show fewer middle-depth failures in HwH-std than Inter (Figures 4-5), a possible lost-in-the-middle connection within the extrapolation analysis.

Figure 5 adds the 1K-32K comparison with identical inference-time logarithmic query scaling following SWAN-GPT (arXiv:2504.08719), using α=300\alpha=300 and no additional training. Appendix D specifies the six context budgets and scaling setting. HwH-swa has a NIAH extrapolation score of 92.73 at 32K; Inter and HwH-std fail from 8K. The added 380M Inter-swa control in Appendix E.2, Figure 16 reaches only 10.9 at 6K, versus HwH-swa's 90.0 with the same 128-token window. Component replacement alone does not explain sustained extrapolation.

3. Evaluation protocols and manuscript changes. Our evaluations follow widely used frameworks and published protocols. WikiText/LAMBADA perplexity and seven zero-shot tasks use EleutherAI's Language Model Evaluation Harness (lm-eval-harness; arXiv:2405.14782). Real-world retrieval uses the cloze tasks of Based (arXiv:2402.18668) and Just Read Twice (JRT; arXiv:2407.05483); length extrapolation uses RULER NIAH (arXiv:2404.06654). Representative precedents include Mamba-2 (arXiv:2405.21060) for the seven-task zero-shot suite, and Gated Delta Networks (arXiv:2412.06464) and Mamba-3 (arXiv:2603.15569) for cloze retrieval and RULER NIAH. GDN also reports WikiText/LAMBADA perplexity. Both studies truncate real-world cloze inputs to 2K, including for Mamba-2 baselines. Their reported RULER evaluations cover the same three single-needle tasks, with no multi-key or full-suite results. This provides a directly comparable evaluation protocol at the 1.3B-1.5B scale.

Section 3 derives the principles and Section 4 tests them. Appendix D gives evaluation protocols. Appendix F connects retrieval width, positional range and learning the mix of the two functions in hybrid architectures.

4. Updated supplementary materials. ICLR2027_supplementary.zip includes the original source tree, all paper data and figures, full diagnostic data and visualizations, and manuscript comparison PDFs. README.txt and PAPER_DATA_INDEX.csv guide navigation; MODEL_CONSISTENCY_GUIDE.txt points to the original small/large-model measurements. File checksums are included.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.