acceptodds
Desk rejected at ICLR 2027

HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

The quadratic complexity of attention poses a critical bottleneck for long-context processing, spurring interest in hybrid attention designs. Most open-source hybrid models adopt a layer-wise strategy. Yet, prior work has noted the inherent difficulty of integrating Linear Attention (LA) with Full Attention (FA), suggesting that the design space of attention hybridization remains underexplored. To probe this space, we conduct interpretability analysis and observe that layers exhibit block-wise functional similarity, while individual heads within the same layer display distinct functional specialization despite sharing input features. This head-level heterogeneity suggests that the head dimension provides a natural and principled granularity for fusing heterogeneous attention signals. Building on this insight, we introduce HydraHead, a novel architecture that hybridizes FA and LA along the head axis. HydraHead features two key innovations: (1) an interpretability-driven selection strategy that identifies retrieval-critical heads and preserves FA only for them, and (2) a scale-normalized fusion module that reconciles the distributional gap between FA and LA head outputs. By leveraging a three-stage transfer pipeline with parameter reuse and distillation, we achieve high-performance hybrid models with minimal training overhead. Under a unified training setup, HydraHead outperforms other hybrid designs in long-context tasks while maintaining strong general reasoning. With interpretability-driven head selection, it matches a 3:1 layer-wise hybrid's long-context performance at a 7:1 LA-to-FA ratio. Crucially, trained on only 15B tokens, HydraHead achieves over 69% improvement over the baseline at 512K context length, approaching Qwen3.5, a leading model of comparable size with a native context length of 256K. This highlights the significant scaling potential of head-level hybridization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hybrid Attention, Interpretability-Driven Head Selection, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AVe5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

HydraHead targets long-context retrieval under reduced full-attention (FA) budgets through head-level computation allocation. Intervention-based importance scores determine which heads retain FA and which convert to Gated DeltaNet, supported by per-head normalization, scaling, and three-stage distillation. Experiments on Qwen3-1.7B examine allocation strategies, fusion components, FA-budget trade-offs, and general capabilities.

Critical strengths
  1. Translating intervention-based head analysis into exclusive FA–GDN allocation meaningfully connects interpretability with architecture conversion. Global selection substantially improves retrieval over fixed allocation, and a 7:1 configuration approaches the retrieval average of a denser 3:1 layerwise hybrid.
  2. Calibration sensitivity, ranking stability, intervention controls, shared baseline improvements, and component ablations support the method’s empirical motivation.
Critical weaknesses
  1. Incomplete and training-unmatched flagship experiment: The emphasized approximately 15B-token results lack a complete checkpoint-specific recipe, while the disclosed schedule accounts for only 5.8B tokens. Reproducibility and attribution require complete training and evaluation configurations, including 512K evaluation, plus an FA continuation baseline matched for data, training budget, and appropriate positional adaptation.
  2. Unestablished resource-equivalent efficiency: Relative KV-cache usage is 0.35 for HydraHead versus 0.25 for the layerwise baseline, so equal FA-head proportions do not establish equal resource budgets.
  3. Global-Interp changes both head identities and per-layer FA allocation, while layer coverage alone produces substantial retrieval gains. Converted-model controls must preserve its exact per-layer quotas while randomizing head identities across multiple seeds with uncertainty estimates, original-model interventions do not isolate this contribution.
Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response [1/3]

Thank you for recognizing the connection between intervention-based head analysis and architecture conversion, and the supporting ablations. Your comments highlight important requirements for reproducibility and attribution. We focus below on training details, actual resource budgets, and controls that distinguish head identity from per-layer allocation and coverage.

【Q1】Training recipe, matched baselines, and reproducibility of the 512K evaluation.

Thank you for pointing out the need to clarify the training and evaluation details. The 2.3B-token configuration in Table 6 is the default training setting for our ablation studies; the 5.8B-token configuration in Table 16 is used for the training-configuration optimization ablation; and the 15B-token configuration is used for the context-length extrapolation experiment.

For the 15B-token context-length extrapolation experiment, we provide the detailed three-stage training configuration below. The selected heads and the 3:1 configuration remain unchanged across all three stages.

StageBatch sizeContext lengthTraining stepsLR scheduleLearning rate
11282,0485,909Cosine1e-3 → 1e-5
22562,0488,104Cosine1e-4 → 1e-5
312832,7682,432Constant1e-5

For the 512K evaluation, we use the model trained with the above configuration as our final model and evaluate it using the default configuration of lm-harness. All models are evaluated with the same parameters.

Matched pure-FA continuation. We have completed two pure-FA controls initialized from Qwen3-1.7B: one follows the default training configuration used in Table 1 (approximately 2.3B tokens), and the other follows the extended 15B training schedule above. Both use the same FA-side gate and NoPE/scaling configuration as HydraHead, without introducing LA, and are matched to the corresponding HydraHead model in training data, schedule, and evaluation settings.

Training settingModelSingle (native)Single (extended)Multi-key (native)Multi-key (extended)Reasoning (hard)Reasoning (easy)
Default (2.3B tokens)HydraHead98.4787.4937.1027.3731.0362.12
Default (2.3B tokens)FA-only (gate + NoPE/scaling)2.270.163.400.0020.4060.62
Extended (15B setting)HydraHead99.9097.9393.4067.8350.6262.78
Extended (15B setting)FA-only (gate + NoPE/scaling)14.402.242.550.0040.7161.10

The FA-only controls recover little retrieval capability at either budget: even after extended training, extended single-key retrieval is 2.24 versus 97.93 for HydraHead, and extended multi-key retrieval is 0.00 versus 67.83. Hard reasoning is also lower (40.71 versus 50.62), while the easy-reasoning gap is smaller. Thus, the FA modifications and additional training alone do not reproduce HydraHead's performance; the LA-containing hybrid structure is important under this shared FA configuration.

We adopt NoPE in the FA branch of the hybrid specifically to improve context-length extrapolation. The contrast with the FA-only controls suggests that the effectiveness of this positional configuration depends on the architecture; LA's order-sensitive representations offer a plausible explanation for why it works better within the hybrid. This interpretation is consistent with HypeNet's HyPE design, which retains RoPE in RNN layers and uses NoPE with attention scaling in FA layers (Chen et al., 2026, Section 5.1). We will include the training and evaluation details, these matched controls, and the corresponding analysis in the revised manuscript.

AuthorsOpenReview ↗

Response [2/3]

【Q2】Resource budgets and the efficiency–quality trade-off.

Thank you for raising this point. Although both methods use a 3:1 mixing ratio, this ratio is defined over query heads; GQA grouping leads to a higher retained KV cache fraction in our head-wise configuration. We report measured inference latency and cache-memory costs against the layer-wise baseline below.

ModelPrefill 32K (ms)Prefill 64K (ms)Decode 32K (ms/token)Decode 64K (ms/token)Total cache 32K (MiB)Total cache 64K (MiB)
HypeNet, layer-wise 3:1772.71760.523.2741.72918.91814.9
HydraHead, head-wise 3:1840.51895.123.3239.611303.22583.2
HydraHead, head-wise 5:1802.71743.217.8327.82889.31753.3
HydraHead, head-wise 7:1785.71671.414.7421.26730.31434.3

At 3:1. HydraHead uses more cache, but inference-latency differences are modest: prefill latency is 8.8% and 7.6% higher than the measured layer-wise baseline at 32K and 64K, respectively, while decoding is essentially matched at 32K and 5.1% faster at 64K.

At lower FA fractions. The 5:1 configuration provides a closer cache match: at 64K, total cache is 3.4% smaller, prefill latency is comparable, and decoding throughput is 1.50× higher. At 7:1, total cache is 21.0% smaller at 64K, with 5.1% lower prefill latency; decoding throughput reaches 1.58× at 32K and 1.96× at 64K. Thus, reducing the FA fraction yields larger cache and decoding efficiency gains, particularly at longer contexts. Practical efficiency also depends on implementation and kernel optimization.

Separately, Table 1 shows stronger task performance at 3:1 and broadly comparable retrieval performance at 7:1 relative to the layer-wise baseline, supporting a favorable performance–efficiency trade-off.

AuthorsOpenReview ↗

Response [3/3]

【Q3】Head selection, per-layer allocation, layer coverage, and uncertainty.

Thank you for suggesting this more rigorous comparison. Table 2 already provides complementary evidence: Layer-Rand (b) and Layer-Interp (d) both maintain a 3:1 ratio in every layer, differing only in random versus interpretability-guided head selection within each layer. Layer-Interp improves native single-key retrieval from 87.07 to 91.13, native multi-key retrieval from 27.70 to 35.10, and hard reasoning from 27.91 to 31.93. These gains support the benefit of interpretability-guided selection when the per-layer ratio is controlled.

To directly isolate the contribution of head identity, we compare Global-Interp with three randomized models at an overall 3:1 ratio. Each randomized model preserves Global-Interp's exact per-layer FA head counts and layer coverage, but randomly selects heads within each layer using a different seed. The remaining training and evaluation settings are unchanged. Evaluation is now complete for all six metrics. The random-seed mean and sample standard deviation (std) are reported in separate rows.

ModelSingle (native)Single (extended)Multi-key (native)Multi-key (extended)Reasoning (hard)Reasoning (easy)
Global-Interp98.7081.7338.3525.9331.7062.68
Random within layers (3 seeds): mean58.0939.0420.3710.7927.5162.51
Random within layers (3 seeds): std6.9211.485.553.160.780.23

Global-Interp outperforms all three random runs on all four retrieval metrics and hard reasoning. Relative to the random-seed mean, single-key retrieval improves by 40.61 and 42.69 percentage points at native and extended lengths, respectively; multi-key retrieval improves by 17.98 and 15.14 points. Hard reasoning also improves (31.70 vs. 27.51), while easy reasoning remains comparable (62.68 vs. 62.51). Because per-layer allocation and layer coverage are fixed, these converted-model controls demonstrate that head identity contributes to performance beyond allocation or coverage alone, with consistent retrieval gains across all three seeds. We will include the complete three-seed comparison and the corresponding analysis in the revised manuscript.

Reviewer_vj77OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HydraHead, a method for hybridizing a full attention model headwise rather than layerwise. First, the method characterizes which heads are important for retrieval by an interpretability-inspired permutation of activations on some probe examples. Then, the heads that are most related to retrieval capabilities are preserved as full attention and the remainder of heads are linearized to GDN heads through a lightweight alignment and 15B token extension. Through ablations, the work demonstrates that selecting these particular heads to preserve as full attention heads and allowing the percentage of heads hybridized at each layer are both important factors in the performance. The method generalizes to longer contexts well while reducing KV cache cost substantially.

Critical strengths

S1. I think an interpretability-guided approach to deciding what to hybridize seems obviously correct (and clearly effective; I appreciated the ablations in section 3.3). This also allows for some interesting demonstration of which layers are more responsible for retrieval capabilities (versus layers that can be easily hybridized).

S2. The method requires relatively little compute (while alignment + 15B token extension is not cheap, it's a lot cheaper than typical LC extension) to achieve some nice length generalization.

S3. The authors ablate many features of the method and compare against a good sweep of other hybridization strategies.

In general I quite like the paper; if weaknesses 1+2 can be addressed in the rebuttal, I would be happy to support acceptance.

Critical weaknesses

W1. I'm concerned about the fairness of the comparison to some of the baselines. The paper combines a number of improvements to attention, which is nice; however, it's not clear how much of the improvement from the baseline model comes from the improvements to FA and the 15B adaptation stage versus the strategy for LA.

W2. I don't quite agree with the authors' framing of the reasoning results. It seems that HydraHead trails behind the original Qwen and several other hybrid baselines on harder reasoning tasks, sometimes by wide margins (e.g. Table 5 and Table 9). This seems like an issue that requires more investigation: can you characterize the types of tasks this occurs on? Is this an issue with just training data (e.g. does the original Qwen also lose some performance when trained on your 15B token continued pretraining)?

W3. (less serious) I think the paper could be organized a bit better. The main text is a bit repetitious, reiterating the motivation for the method at the start of several sections. The appendices are great, but contain some information that would be helpful for the reading experience in the main text (e.g. figure 10, some of the related work and limitations discussion, further explanation for the methods in 2.1). This isn't a major factor in my score, but I left several presentation comments below that I believe would improve the paper.

Questions for authors

Q1. HydraHeads is competitive in terms of cache costs to other hybridization methods. Can you comment on the inference time for each method as you scale up context length? Is hybridizing headwise instead of layerwise slower, and if so by how much? I could imagine this being the case since you now have heterogenous layer types.

Q2. If you take Qwen 3 1.7B and applied just the FA fixes (head-specific reweighting, scaling instead of RoPE) and trained it for the same 15B tokens that you trained your adapted model for, how well does it do? This ablation would disentangle the FA tweaks from the proposed LA scheme.

Q3. Can you provide results on other long-context benchmarks, especially those targeting more reasoning-style capabilities? (e.g. Oolong or Babilong). Given the reasoning performance drops, I'm concerned that this method might preserve LC retrieval performance at the cost of other important capabilities.

Q4. Are the models in Table 1 also trained from Qwen 3 1.7B? (I assume yes.) Are they all also adapted for 15B tokens?

Q5. Line 439: are you adapting with yarn at each context length individually, or adapting to 256K context length immediately? The yarn results for shorter context lengths look weirdly low.

Q6. It looks like your method more-or-less concentrates the LA in the second half of the model. (1) would layerwise random selection over the second half of layers be effective? (2) do you expect this to be a general pattern across model families? (3) do you expect that this implies that FA layers in natively trained hybrids should be concentrated earlier in the network?

Q7. What is the scale on the “per-head logit” figure in figure 2? Confused what the numerical values represent.

Comments about presentation (no need to respond):

A few very AI-like sentences:

  • “Deciding which heads should retain full attention, and which can be safely approximated by linear recurrence, is the decision on which our efficiency–quality trade-off hinges.”
  • “A scalar readout m measures evidence for the clean target behavior “
  • Section 2.1 in general, actually

Line 347 typo: “Furhtermore”

Line 364: referencing a (very cool!) figure in the appendix. It would be nice to have this in the main text.

1026: confusing explanation of path patching

“quadratic complexity of attention” – specify time not space complexity

Figure 1: colors are really bad

General writing comment — sometimes a little overly flowery / verbose, would be clearer if you wrote more simply; especially in the start of the intro, e.g. “pivotal transition” -> “transition”, "unprecedented demands”

Line 318: headwise mixing vs headwise selection? In general a small figure here would be nice to visualize the different methods

5 comments
AuthorsOpenReview ↗

Response [1/4]

We appreciate your positive assessment of interpretability-guided hybridization, its length generalization, and the ablation study. We also take your concerns about baseline fairness and reasoning performance seriously. Our responses below distinguish the comparison settings, clarify the observed trade-offs, and address your suggestions for improving the presentation.

【Q1】Inference efficiency and head-wise versus layer-wise hybridization.

We agree that cache savings alone do not establish faster inference, and that combining heterogeneous mechanisms within each layer raises a practical concern about runtime overhead. To address this directly, we benchmark prefill and decoding as context length increases using Hugging Face at batch size 1, reporting median latency over repeated measurements.

At the same nominal ratio. HydraHead 3:1 has 8.8% and 7.6% higher prefill latency than layer-wise HypeNet 3:1 at 32K and 64K, respectively. However, decoding is essentially matched at 32K and 5.1% faster at 64K. Thus, the measured overhead is modest for prefill, rather than a uniform slowdown across inference stages.

The nominal 3:1 ratio also does not imply equal cache usage: it specifies the split of query heads. Under GQA, multiple query heads share one KV head, which must be retained if any query head in its group still uses FA. In our 3:1 layout, 25% FA query heads therefore require 35.7% of the original full-FA KV cache, compared with 25% for the 3:1 layer-wise hybrid. The table reports latency and total cache (FA KV plus GDN states):

ModelPrefill 32K (ms)Prefill 64K (ms)Decode 32K (ms/token)Decode 64K (ms/token)Total cache 32K (MiB)Total cache 64K (MiB)
HypeNet, layer-wise 3:1772.71760.523.2741.72918.91814.9
HydraHead, head-wise 3:1840.51895.123.3239.611303.22583.2
HydraHead, head-wise 5:1802.71743.217.8327.82889.31753.3

At similar cache budgets. HydraHead 5:1 retains an FA KV fraction of 24.1%, giving a closer cache match to layer-wise 3:1; at 64K, it uses 3.4% less total cache. With matched decoding implementations for the two hybrids, prefill latency is comparable (within 4% at the reported lengths), while decoding throughput reaches 1.31× at 32K and 1.50× at 64K. This addresses the runtime concern: at a similar cache budget, head-wise hybridization provides comparable prefill latency and faster decoding, with the decoding advantage increasing at the longer evaluated context. Practical efficiency also depends on implementation and kernel optimization.

【Q2】Fairness of comparisons and attribution of performance gains.

Thank you for suggesting this controlled comparison. We have completed pure-FA continuation experiments from the same Qwen3-1.7B checkpoint, using HydraHead's FA-side gate and NoPE/scaling configuration but without LA. We match the data, training schedule, and evaluation settings at both the default 2.3B-token budget and the extended 15B training setting.

Training settingModelSingle (native)Single (extended)Multi-key (native)Multi-key (extended)Reasoning (hard)Reasoning (easy)
Default (2.3B tokens)HydraHead98.4787.4937.1027.3731.0362.12
Default (2.3B tokens)FA-only (gate + NoPE/scaling)2.270.163.400.0020.4060.62
Extended (15B setting)HydraHead99.9097.9393.4067.8350.6262.78
Extended (15B setting)FA-only (gate + NoPE/scaling)14.402.242.550.0040.7161.10

At the extended budget, the FA-only control scores 2.24 and 0.00 on extended single-key and multi-key retrieval, versus 97.93 and 67.83 for HydraHead; hard reasoning is also lower (40.71 vs. 50.62). The same contrast is observed at the default budget. These matched comparisons show that the FA modifications and training budget alone do not explain HydraHead's gains: the LA-containing hybrid structure is important under the same FA configuration.

In the hybrid architecture, the LA branch's inherent positional bias and its use of RoPE allow the FA branch to adopt NoPE for improved context-length extrapolation. This positional design is also discussed in HypeNet (Chen et al., 2026, Section 5.1). We will include these controls and the corresponding analysis in the revised manuscript.

AuthorsOpenReview ↗

Response [2/4]

【Q3】Reasoning degradation and long-context reasoning evaluation.

Thank you for raising this important concern. We agree that the reasoning results require a clearer distinction between comparison settings and a closer examination of training-data effects.

(1) Different purposes of Tables 5 and 9. Table 9 compares hybridization strategies under the same training settings. Both token-wise and head-wise approaches achieve substantially higher general-reasoning scores than layer-wise hybrids. Token-wise mixing has an advantage over head-wise mixing on these reasoning benchmarks, but its much weaker needle-in-a-haystack retrieval results indicate that head-wise mixing offers a more favorable balance of reasoning and long-context retrieval. Table 5 is a separate scaling experiment comparing HydraHead, trained on more data, directly with other open-source models. Here, we acknowledge the substantial GSM8K drop relative to Qwen3-1.7B (52.84 vs. 69.45), although the average across the four general benchmarks is close to Qwen3-1.7B-YaRN (50.62 vs. 50.37). This average does not imply that every capability is preserved.

(2) Effects of continued pretraining. To examine the role of training data, we continued pretraining Qwen3-1.7B without changing its architecture or FA configuration, using the corresponding 2.3B and 15B training corpora. The following comparison reports needle-in-a-haystack retrieval and the more challenging coding and mathematical reasoning benchmarks:

ModelSingle (native)Single (extended)Multi-key (native)Multi-key (extended)MBPPGSM8K
Qwen3-1.7B99.9332.7195.9514.0743.0069.45
Qwen3-1.7B (CPT, 2.3B tokens)99.9327.8786.109.6346.0058.07
Qwen3-1.7B (CPT, 15B tokens)99.9029.6793.659.9752.8064.83
HypeNet-2B85.3377.2219.4520.104.804.93
HydraHead99.9097.9393.4067.8347.2052.84

Both continuation budgets reduce extended single-key and multi-key retrieval relative to the original model. In contrast, coding performance improves from 43.00 to 46.00 and then to 52.80. Mathematical reasoning initially decreases from 69.45 to 58.07, then recovers to 64.83 with more training, although it remains below the original score.

These results show that continued-pretraining effects are task-dependent rather than a uniform loss of general reasoning. After 15B continuation, the unchanged model still scores higher than HydraHead on MBPP and GSM8K. Thus, training data alone is insufficient to explain the hybrid's remaining general-reasoning gap; architectural conversion and adaptation must also be considered.

(3) Additional BABILong evaluation. Following your suggestion, we evaluate BABILong, which requires reasoning over information in long contexts, and report the mean score across qa1–qa5:

Model4K64K128K
Qwen3-1.7B0.6920.3080.000
Qwen3-1.7B (CPT, 15B tokens)0.6400.1640.004
HypeNet-2B (layer-wise)0.1480.0200.008
HydraHead 3:1 (head-wise)0.4200.0840.092

Qwen3-1.7B achieves the highest scores at 4K and 64K, indicating that it retains an advantage at the shorter evaluated context lengths. HydraHead retains a mean score of 9.2% at 128K and outperforms the layer-wise baseline at every evaluated length, consistent with its advantage over layer-wise hybridization on the general-reasoning benchmarks. Additionally, we include a comparison with Qwen3-1.7B after continued pretraining on 15B tokens. The results show that the model also exhibits performance degradation after continued pretraining, possibly due to the training data. This further indicates that our improvements in long-context performance are not primarily driven by the training data.

(4) Overall interpretation. Under matched training settings, HydraHead has a clear advantage over layer-wise hybridization. Comparisons with the original open-source models also highlight the importance of sufficient high-quality training data for preserving capabilities after conversion. We will clarify these comparison settings, task-dependent changes, and the role of training data in the revised manuscript.

【Q4】Initialization and training budgets of the models in Table 1.

All hybrid models compared in Table 1 are converted from the same Qwen3-1.7B checkpoint using the same default training configuration. The default schedule in Appendix C.1 and Table 6 uses approximately 0.3B, 1.0B, and 1.0B tokens across the three stages, totaling approximately 2.3B. The 15B-token training configuration is used for the model performance scaling experiment.

AuthorsOpenReview ↗

Response [3/4]

【Q5】YaRN configuration and performance at shorter context lengths.

We use a single fixed YaRN configuration with a scaling factor of 4, extending the original 32K context window to 128K. This configuration is unchanged across evaluation lengths; it is not retuned for each length or set initially to target 256K.

This fourfold extension is a commonly used YaRN setting, consistent with Qwen's official 32K-to-128K configuration (Qwen Team, Qwen3-32B model card). We also evaluated an eightfold extension, but observed a substantial performance drop compared with the fourfold setting. We therefore retained factor 4 for all reported results. Applying the same static long-context scaling to shorter inputs may contribute to their lower scores, as Qwen's documentation also warns that static YaRN can affect shorter-text performance (Qwen documentation, “vLLM—Extended Context Support”).

At 256K, evaluation goes beyond the 128K target. Table 4 shows the corresponding retrieval score falling from 62.49 at 128K to 27.20 at 256K under this fixed configuration.

【Q6】Head allocation across depth, model families, and native hybrid layouts.

We first clarify that the observed middle-to-late concentration concerns the retained FA heads, not LA heads. In Figure 10(e), 99 of the 112 retained FA heads lie in the last 14 of 28 layers. With this distinction, we address your three questions in order.

(1) Would layer-wise random selection over the second half of layers be effective?

It is a reasonable depth-based heuristic and may capture part of the benefit, but depth alone does not identify which heads are functionally important. Table 2 provides complementary evidence for function-guided selection: with the same per-layer FA budget, Layer-Interp (d) improves over Layer-Rand (b) on native-length single-key retrieval (91.13 vs. 87.07), native-length multi-key retrieval (35.10 vs. 27.70), and hard general reasoning (31.93 vs. 27.91). Although the gains are not uniform across metrics, this shows that head identity matters even when layer allocation is unchanged.

We further evaluate head identity in four source models by replacing selected head outputs with those from answer-edited counterfactual inputs. The table reports post-intervention first-answer-token accuracy on 80 held-out examples per model with 4K contexts and distant supporting facts, rather than accuracy decreases or complete-answer generation accuracy. Unperturbed denotes the original accuracy without replacement. Top-16 is selected across all available head types; the random mean averages 16 selections matching the head counts in every layer and attention type. Lower post-intervention accuracy indicates a larger behavioral effect.

Source modelUnperturbedSelected Top-16 perturbedMatched random mean
Qwen3-1.7B-Base100.00%8.75%81.64%
Llama-3.2-3B100.00%91.25%99.77%
Qwen3.5-4B-Base98.75%15.00%94.77%
Gemma 4 E2B Base100.00%8.75%54.37%

Perturbing selected heads has a larger accuracy effect than the matched random mean in all four models, supporting functional selection beyond depth alone.

Separately, we perform an early-half intervention on Qwen3-1.7B-Base under the same 4K, distant-fact evaluation conditions. This set contains the five FA heads in the first 14 of 28 layers whose aggregated localization scores are positive. We replace only these five heads, not all heads in the earlier half. The following table distinguishes this supplementary group from the whole-model Top-16:

Intervention set (Qwen3-1.7B-Base)Heads replacedUnperturbedSelected heads replacedMatched random mean
Whole-model Top-1616100.00%8.75%81.64%
Positive-score FA heads in the early half5100.00%8.75%91.33%

Both selected sets yield 8.75% accuracy, but they share only three heads; the early-half group is not the early-layer subset of the whole-model Top-16. This supplementary result shows that a later-half-only rule can omit important earlier components. We therefore view depth as a useful guide, rather than a substitute for functional head selection. These controls assess head identity at fixed layer allocations, rather than directly comparing converted models with later-half whole-layer random selection.

Reviewer_JzFBOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HydraHead, a head-wise hybrid attention architecture that selectively retains full attention for retrieval-critical heads while replacing others with linear attention. It introduces interpretability-guided head selection and scale-normalized fusion to improve long-context modeling.

Critical strengths
  • Novel architecture: Head-level hybridization offers a well-motivated alternative to conventional layer-wise designs.

  • Strong results: Extensive experiments demonstrate substantial improvements in long-context retrieval, supported by meaningful ablations.

  • Practical design: The transfer pipeline efficiently adapts pretrained models to hybrid architectures.

Critical weaknesses
  • W1. Limited Empirical Efficiency Analysis.

The paper primarily reports relative KV cache sizes but lacks end-to-end measurements of prefill latency, decoding throughput, and peak memory usage. Since the architecture retains a proportion of full attention heads, its overall prefill complexity remains quadratic.

  • W2. Limited Generalization Across Architectures and Tasks.

The experiments mainly focus on Qwen3-1.7B and GDN, while both head selection and evaluation rely heavily on RULER-style retrieval tasks. Although the appendix analyzes head-ranking stability, generalization to other models and more challenging long-context tasks remains insufficiently explored.

  • W3. Unclear Individual Contributions of Fusion Components.

Table 13 shows that head-wise scaling improves extended single-key retrieval from 81.73 to 87.49 compared with the already normalized Basic Model. Since normalization itself contributes substantially to performance, the individual benefits of RMSNorm and learnable scaling should be more clearly distinguished.

Questions for authors

How does HydraHead compare with layer-wise hybrids in actual inference speed and memory consumption under similar cache budget?

Can the proposed head selection strategy generalize to other models and attention mechanisms such as (mamba, deltanet) and (vanilla, mamba) ?

Could the authors evaluate, or at least discuss, the generalizability of HydraHead to different linear attention variants, such as PolaFormer++ (TPAMI 26) and FLatten Transformer (ICCV 23)? Under the same hybridization strategy, do linear attention variants that perform better in fully linear models consistently retain their advantages in hybrid architectures?

6 comments
AuthorsOpenReview ↗

Response [1/3]

Thank you for recognizing the motivation behind head-wise hybridization, the long-context retrieval results, and the practicality of the conversion pipeline. Your suggestions help sharpen our evaluation of inference efficiency, generalization, and the separate contributions of normalization and scaling, which we address below.

【Q1】Inference efficiency and computational complexity.

We compare three architectural families: GQA (Qwen3-1.7B), layer-wise hybridization (HypeNet), and head-wise hybridization (HydraHead). We benchmark inference using Hugging Face (HF) at batch size 1 and report median inference latency over repeated measurements. Practical inference efficiency also depends on implementation and kernel optimization, not only on the architectural design.

ModelPrefill 8K (ms)Prefill 32K (ms)Prefill 64K (ms)Decode 8K (ms/token)Decode 32K (ms/token)Decode 64K (ms/token)
HydraHead 3:1188.3840.51895.112.6923.3239.61
HydraHead 5:1187.1802.71743.211.2317.8327.82
HydraHead 7:1188.0785.71671.410.6314.7421.26
HydraHead 9:1187.8774.91626.910.1912.8616.73
HypeNet-2b-gdn176.8772.71760.510.0923.2741.72
Qwen3-1.7B188.31103.13101.427.5731.5638.53
ModelTotal cache 8K (MiB)Total cache 32K (MiB)Total cache 64K (MiB)Peak allocated memory 8K (MiB)Peak allocated memory 32K (MiB)Peak allocated memory 64K (MiB)
HydraHead 3:1343.21303.22583.24615725110761
HydraHead 5:1241.3889.31753.34545693010110
HydraHead 7:1202.3730.31434.34546690310046
HydraHead 9:1183.0651.01275.04582702310278
HypeNet-2b-gdn246.9918.91814.9438464129117
Qwen3-1.7B899.53587.57171.54608855613820

Compared with Qwen3-1.7B. At 64K, HydraHead reduces prefill latency by 38.9–47.5%, total cache by 64.0–82.2%, and peak memory by 22.1–27.3%. The 5:1, 7:1, and 9:1 configurations also achieve faster decoding at this context length.

Compared with layer-wise hybridization. HydraHead 3:1 achieves decoding latency comparable to HypeNet at 32K and 64K. At these lengths, HydraHead 5:1 has a similar cache footprint and comparable prefill latency, while providing faster decoding. At 64K, it uses 3.4% less cache than HypeNet and achieves 1.50× decoding throughput, demonstrating improved decoding efficiency at a similar cache budget.

HydraHead retains quadratic prefill complexity, as do layer-wise hybrids with global FA layers. Layer-wise hybrids reduce the number of FA layers; HydraHead reduces the number of FA heads. The benefit is a smaller FA computation and storage footprint, rather than a change in the asymptotic order.

AuthorsOpenReview ↗

Response [2/3]

【Q2】Generalization across tasks, models and attention mechanisms.

Framework applicability and adaptation. HydraHead is a mechanism-agnostic head-wise framework: its allocation and fusion principles are not tied to the FA–GDN pair. Other attention or recurrent mechanisms can serve as hybrid units through appropriate head-level interfaces and causal implementations, including variants such as PolaFormer++ and FLatten and combinations such as Mamba–DeltaNet or GQA–Mamba. For pairs such as Mamba–DeltaNet that do not retain Qwen3's original GQA-based FA mechanism, the transfer training scheme may benefit from appropriate adjustments. We will add a discussion subsection in the revised manuscript explaining these interfaces, causal adaptations, and transfer considerations.

Alternative attention mechanisms. We additionally evaluate FA–Lightning Attention within the same 3:1 head-wise framework:

Head-wise variant (3:1)Single (native)Single (extended)Multi-key (native)Multi-key (extended)General (hard)General (easy)
FA + GDN98.4787.4937.1027.3731.0362.12
FA + Lightning Attention89.9384.5145.6533.5726.0562.07

Lightning Attention improves native and extended multi-key retrieval by 8.55 and 6.20 percentage points, respectively, whereas GDN performs better on single-key retrieval and the hard general-task set. Different mechanisms can therefore be incorporated successfully, but their relative strengths depend on the task and their interaction within the hybrid. HypeNet's mixer ablation (Chen et al., 2026, Figure 5 and Section 6.3) likewise reports stronger length generalization with Lightning Attention than with more recent recurrent variants. These findings motivate evaluating each hybrid combination rather than assuming that rankings in fully linear models carry over unchanged; branch complementarity and training adaptation may also affect stability and final performance.

Cross-task transfer of head selection. We use RULER retrieval tasks as controlled probes to identify heads important for locating information in long contexts, aligning selection with the retrieval capability we seek to preserve. To test whether this localization transfers, we applied the fixed RULER Top-8 heads to held-out BABILong examples in the original Qwen3-1.7B source model. Patching these heads with counterfactual prefill activations reduced accuracy more than patching layer-matched random heads:

BABILong taskOriginalRULER Top-8 patchedLayer-matched random patched
QA1, 2K100%68.4%95.7%
QA2, 1K100%63.0%91.4%

We report story-averaged first-token accuracy on examples with correct clean and counterfactual predictions. The random control averages 16 masks with matching per-layer head counts. The larger accuracy drop shows that RULER-selected heads remain functionally important on BABILong, supporting their transferability to long-context tasks beyond RULER.

Cross-task transfer in the converted model. Following Reviewer vj77's suggestion, we evaluate the 3:1 HydraHead model built using RULER-based head selection on BABILong and report the mean score across qa1–qa5 at each context length. We compare it with layer-wise HypeNet using the same training configuration and data:

Model4K16K32K64K128K
HypeNet-2B (layer-wise)0.1480.0560.0440.0200.008
HydraHead 3:1 (head-wise)0.4200.2120.1040.0840.092

HydraHead maintains an advantage over the layer-wise baseline at all five evaluated lengths from 4K to 128K. The model built using RULER-selected heads therefore also performs favorably on BABILong, providing further evidence for the cross-task transferability of RULER-based head selection.

Generalization to other models. We have converted Qwen3-4B using the same head-wise pipeline. Due to the limited rebuttal time, we adopted the optimized 5.8B-token transfer-training strategy (Table 23) without adjusting the data scale or training configuration for the larger model:

ModelSingle (native)Single (extended)Multi-key (native)Multi-key (extended)General (hard)General (easy)
Qwen3-4B10033.1199.7023.8072.7867.16
HydraHead (from Qwen3-4B)99.9089.6988.9043.3063.2467.31

HydraHead retains a clear advantage on extended-context retrieval, while native-context retrieval and easy general reasoning remain comparable. On the hard general-reasoning set, a gap remains (63.24 vs. 72.78). We attribute this gap to the training data scale and configuration, which were tuned for the 1.7B setting and not adjusted for the 4B model. We will update the results once the recipe is adapted to the larger model.

AuthorsOpenReview ↗

Response [3/3]

【Q3】Individual contributions of RMSNorm and learnable scaling.

To separate the contributions of RMSNorm and learnable scaling, we provide the complete four-way ablation below. It combines the three settings in Table 13 with the additional scaling-only result.

Fusion settingSingle (native)Single (extended)Multi-key (native)Multi-key (extended)Reasoning (hard)Reasoning (easy)
Neither RMSNorm nor extra scaling85.0771.3641.9023.9330.6262.58
RMSNorm only98.7081.7338.3525.9331.7062.68
Extra scaling only93.2377.3843.1025.8330.0862.51
RMSNorm + extra scaling98.4787.4937.1027.3731.0362.12

RMSNorm improves extended single-key and multi-key retrieval by 10.37 and 2.00 percentage points, respectively. Adding scaling on top of RMSNorm provides further gains of 5.76 and 1.44 points. Normalization therefore accounts for a substantial part of the improvement, while learnable scaling provides an additional benefit for extended retrieval. The combined setting achieves the best extended retrieval scores among the four configurations, with small decreases on the native-context and reasoning metrics relative to RMSNorm alone.

We will include the complete four-way ablation and the corresponding analysis in the revised manuscript to clarify the individual contributions of RMSNorm and learnable scaling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.