- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
HydraHead targets long-context retrieval under reduced full-attention (FA) budgets through head-level computation allocation. Intervention-based importance scores determine which heads retain FA and which convert to Gated DeltaNet, supported by per-head normalization, scaling, and three-stage distillation. Experiments on Qwen3-1.7B examine allocation strategies, fusion components, FA-budget trade-offs, and general capabilities.
Critical strengths
- Translating intervention-based head analysis into exclusive FA–GDN allocation meaningfully connects interpretability with architecture conversion. Global selection substantially improves retrieval over fixed allocation, and a 7:1 configuration approaches the retrieval average of a denser 3:1 layerwise hybrid.
- Calibration sensitivity, ranking stability, intervention controls, shared baseline improvements, and component ablations support the method’s empirical motivation.
Critical weaknesses
- Incomplete and training-unmatched flagship experiment: The emphasized approximately 15B-token results lack a complete checkpoint-specific recipe, while the disclosed schedule accounts for only 5.8B tokens. Reproducibility and attribution require complete training and evaluation configurations, including 512K evaluation, plus an FA continuation baseline matched for data, training budget, and appropriate positional adaptation.
- Unestablished resource-equivalent efficiency: Relative KV-cache usage is 0.35 for HydraHead versus 0.25 for the layerwise baseline, so equal FA-head proportions do not establish equal resource budgets.
- Global-Interp changes both head identities and per-layer FA allocation, while layer coverage alone produces substantial retrieval gains. Converted-model controls must preserve its exact per-layer quotas while randomizing head identities across multiple seeds with uncertainty estimates, original-model interventions do not isolate this contribution.
Questions for authors
See weaknesses.
6 comments
Response [2/3]
【Q2】Resource budgets and the efficiency–quality trade-off.
Thank you for raising this point. Although both methods use a 3:1 mixing ratio, this ratio is defined over query heads; GQA grouping leads to a higher retained KV cache fraction in our head-wise configuration. We report measured inference latency and cache-memory costs against the layer-wise baseline below.
| Model | Prefill 32K (ms) | Prefill 64K (ms) | Decode 32K (ms/token) | Decode 64K (ms/token) | Total cache 32K (MiB) | Total cache 64K (MiB) |
|---|---|---|---|---|---|---|
| HypeNet, layer-wise 3:1 | 772.7 | 1760.5 | 23.27 | 41.72 | 918.9 | 1814.9 |
| HydraHead, head-wise 3:1 | 840.5 | 1895.1 | 23.32 | 39.61 | 1303.2 | 2583.2 |
| HydraHead, head-wise 5:1 | 802.7 | 1743.2 | 17.83 | 27.82 | 889.3 | 1753.3 |
| HydraHead, head-wise 7:1 | 785.7 | 1671.4 | 14.74 | 21.26 | 730.3 | 1434.3 |
At 3:1. HydraHead uses more cache, but inference-latency differences are modest: prefill latency is 8.8% and 7.6% higher than the measured layer-wise baseline at 32K and 64K, respectively, while decoding is essentially matched at 32K and 5.1% faster at 64K.
At lower FA fractions. The 5:1 configuration provides a closer cache match: at 64K, total cache is 3.4% smaller, prefill latency is comparable, and decoding throughput is 1.50× higher. At 7:1, total cache is 21.0% smaller at 64K, with 5.1% lower prefill latency; decoding throughput reaches 1.58× at 32K and 1.96× at 64K. Thus, reducing the FA fraction yields larger cache and decoding efficiency gains, particularly at longer contexts. Practical efficiency also depends on implementation and kernel optimization.
Separately, Table 1 shows stronger task performance at 3:1 and broadly comparable retrieval performance at 7:1 relative to the layer-wise baseline, supporting a favorable performance–efficiency trade-off.
Response [3/3]
【Q3】Head selection, per-layer allocation, layer coverage, and uncertainty.
Thank you for suggesting this more rigorous comparison. Table 2 already provides complementary evidence: Layer-Rand (b) and Layer-Interp (d) both maintain a 3:1 ratio in every layer, differing only in random versus interpretability-guided head selection within each layer. Layer-Interp improves native single-key retrieval from 87.07 to 91.13, native multi-key retrieval from 27.70 to 35.10, and hard reasoning from 27.91 to 31.93. These gains support the benefit of interpretability-guided selection when the per-layer ratio is controlled.
To directly isolate the contribution of head identity, we compare Global-Interp with three randomized models at an overall 3:1 ratio. Each randomized model preserves Global-Interp's exact per-layer FA head counts and layer coverage, but randomly selects heads within each layer using a different seed. The remaining training and evaluation settings are unchanged. Evaluation is now complete for all six metrics. The random-seed mean and sample standard deviation (std) are reported in separate rows.
| Model | Single (native) | Single (extended) | Multi-key (native) | Multi-key (extended) | Reasoning (hard) | Reasoning (easy) |
|---|---|---|---|---|---|---|
| Global-Interp | 98.70 | 81.73 | 38.35 | 25.93 | 31.70 | 62.68 |
| Random within layers (3 seeds): mean | 58.09 | 39.04 | 20.37 | 10.79 | 27.51 | 62.51 |
| Random within layers (3 seeds): std | 6.92 | 11.48 | 5.55 | 3.16 | 0.78 | 0.23 |
Global-Interp outperforms all three random runs on all four retrieval metrics and hard reasoning. Relative to the random-seed mean, single-key retrieval improves by 40.61 and 42.69 percentage points at native and extended lengths, respectively; multi-key retrieval improves by 17.98 and 15.14 points. Hard reasoning also improves (31.70 vs. 27.51), while easy reasoning remains comparable (62.68 vs. 62.51). Because per-layer allocation and layer coverage are fixed, these converted-model controls demonstrate that head identity contributes to performance beyond allocation or coverage alone, with consistent retrieval gains across all three seeds. We will include the complete three-seed comparison and the corresponding analysis in the revised manuscript.
Response [1/3]
Thank you for recognizing the connection between intervention-based head analysis and architecture conversion, and the supporting ablations. Your comments highlight important requirements for reproducibility and attribution. We focus below on training details, actual resource budgets, and controls that distinguish head identity from per-layer allocation and coverage.
【Q1】Training recipe, matched baselines, and reproducibility of the 512K evaluation.
Thank you for pointing out the need to clarify the training and evaluation details. The 2.3B-token configuration in Table 6 is the default training setting for our ablation studies; the 5.8B-token configuration in Table 16 is used for the training-configuration optimization ablation; and the 15B-token configuration is used for the context-length extrapolation experiment.
For the 15B-token context-length extrapolation experiment, we provide the detailed three-stage training configuration below. The selected heads and the 3:1 configuration remain unchanged across all three stages.
For the 512K evaluation, we use the model trained with the above configuration as our final model and evaluate it using the default configuration of lm-harness. All models are evaluated with the same parameters.
Matched pure-FA continuation. We have completed two pure-FA controls initialized from Qwen3-1.7B: one follows the default training configuration used in Table 1 (approximately 2.3B tokens), and the other follows the extended 15B training schedule above. Both use the same FA-side gate and NoPE/scaling configuration as HydraHead, without introducing LA, and are matched to the corresponding HydraHead model in training data, schedule, and evaluation settings.
The FA-only controls recover little retrieval capability at either budget: even after extended training, extended single-key retrieval is 2.24 versus 97.93 for HydraHead, and extended multi-key retrieval is 0.00 versus 67.83. Hard reasoning is also lower (40.71 versus 50.62), while the easy-reasoning gap is smaller. Thus, the FA modifications and additional training alone do not reproduce HydraHead's performance; the LA-containing hybrid structure is important under this shared FA configuration.
We adopt NoPE in the FA branch of the hybrid specifically to improve context-length extrapolation. The contrast with the FA-only controls suggests that the effectiveness of this positional configuration depends on the architecture; LA's order-sensitive representations offer a plausible explanation for why it works better within the hybrid. This interpretation is consistent with HypeNet's HyPE design, which retains RoPE in RNN layers and uses NoPE with attention scaling in FA layers (Chen et al., 2026, Section 5.1). We will include the training and evaluation details, these matched controls, and the corresponding analysis in the revised manuscript.