- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed a new perspective on hybrid attention. They split every head in each layer into a softmax attention head and a linear attention head. The linear attention head does not have KV cache, so it helps with memory saving.
Critical strengths
- The idea is new. The information in each attention head can be divided into two parts: outlier is harder to preserve and retrieve, so it needs full attention; others can be compressed to the linear attention.
Critical weaknesses
- The main goal is to reduce the memory usage from the full attention layer. Only when the context length is very large is there a need to reduce the memory usage. But when doing the pre-training of this hybrid model, context length only reaches 1K.
- is not adaptively selected. I think each layer may have an optimal , which can help balance the efficiency and performance better.
- Both ablation and main results should have stronger experimental evidence on HOFA's strength.
Questions for authors
I think HOFA is an interesting architecture. But I have the question: why should we use HOFA-like architecture rather than some popular layer-wise hybrid attention architecture (qwen3.5, qwen3-next ...)? I think the author should at least compare the strength and weakness of these two different architecture.
2 comments
Thank you for the clarification. I now have a better understanding of the pre-training context length.
The results with adaptive r look promising. I encourage the authors to further refine the methodology and consider whether incorporating adaptive r into the main method could strengthen the paper's contributions.
I will maintain my original score.
Response to Reviewer R3Rv
Thank you for your feedback on long-context scaling, per-layer r selection, and layer-wise hybrids. We updated the paper to address each point:
[R3Rv: Pre-training context length only reaches 1K, while memory reduction matters most at very large context lengths.]
We agree that testing well past N=1024 is essential. In Section 3.3 and Appendix E (Figures 8-9), we evaluated 125M models from 1k to 64k tokens (64× extension) on PG-19 [1] with YaRN [2] across 54 books, both zero-shot and after matched continual pretraining at N=16384 (100 steps, 26.2M tokens). Without continual pretraining, base scalar HOFA beats base MHA at 64k (63.73 vs. 89.94 PPL) and maintains stable perplexity across pre-softmax temperature scaling (Figure 9), since Softmax temperature scaling applies only to the r outlier channels while the j inlier channels remain invariant. The scalar HOFA continual pretraining result motivated us to test channel-specific effective memory horizons. Standard GLA uses dense data-dependent vector gating [3], while our parameterization λt,c=λtβh,c (βh∈R>0j, initialized to 1) provides channel-specific decay through j learned exponents per head without a full vector-gate projection. HOFA (βh) achieves lower perplexity than MHA and scalar HOFA at every length from 1k to 64k, both zero-shot (56.24 vs. 89.94 at 64k) and after CPT (28.45 vs. 30.00 at 64k).
[R3Rv: r is not adaptively selected; each layer may have an optimal r.]
To study layer-wise capacity allocation, we measured per-head causal channel power Ec and Shannon effective dimension deff(ℓ)=exp(−∑cpcln(pc+10−12)) across eight pretrained models from 124M to 8B (Figure 6, Appendix C), which shows systematic depth-wise variation in channel concentration. While adaptive or learnable per-layer rℓ allocation is left to future work (Section 4.1), using a fixed per-layer rℓ schedule enables contiguous [B,H,N,rℓ] key storage and fixed-shape Triton kernels without dynamic routing overhead. Under a matched 256-channel budget (rˉ=32) at 70M scale (Table 4, Figure 7), moving from uniform Flat HOFA (r=32, 30.96 PPL, 38.72% zero-shot average) to non-uniform schedules improves both metrics: boundary-heavy U-shaped HOFA reaches 30.65 PPL and the highest zero-shot average (39.15%), while Increasing HOFA (r=[16,16,16,16,32,48,48,64]) achieves the lowest validation perplexity (30.35 PPL, 38.85% average).
[R3Rv: Both ablation and main results need stronger evidence; why use HOFA rather than layer-wise hybrids like Qwen3.5 or Qwen3-Next?]
Layer-wise hybrids such as Qwen3-Next [4] and Qwen3.5 [5] interleave Gated DeltaNet layers with Gated Attention layers in a 3:1 linear-to-full-attention layout, placing full-attention KV caching in one quarter of the layers before accounting for additional GQA compression (and Jamba [6] similarly alternates attention and SSM layers across depth). HOFA makes a different tradeoff: it retains a reduced-dimensional exact Softmax pathway in every layer (r≪dh keys, with a 50% value-cache floor as r→0 and 75% total cache at rˉ=dh/2), and composes orthogonally with GQA/MLA [7, 8] (Appendix I) or depth hybridization itself.
Our 70M depth-axis controls are controlled MHA+GLA baselines rather than direct reproductions or head-to-head evaluations of Qwen3-Next or Qwen3.5; they isolate layer placement under our matched architecture and 256-channel key-budget constraints (a 6-MHA : 2-GLA 75%-cache control and direct comparisons to those production architectures remain future work). In Table 4 (Appendix D), placing full MHA at boundary layers improves the 50%-MHA depth control from 31.54 to 30.86 PPL (38.03% to 38.92% avg), while U-shaped HOFA (30.65 PPL, 39.15%) and Increasing HOFA (30.35 PPL, 38.85%) achieve lower perplexity, and Flat HOFA outperforms the 75%-cache-matched Reduced-QK MHA control [9, 10] (30.96 vs. 30.99 PPL, 38.72% vs. 38.27% avg). At the 350M main scale (14B tokens, Table 1, Figure 5), HOFA matches MHA validation perplexity (15.87 vs. 15.88) and outperforms matched 350M MHA across all six zero-shot reasoning benchmarks.
[1] Rae et al., PG-19, ICLR 2020. [2] Peng et al., YaRN, ICLR 2024. [3] Yang et al., GLA, arXiv:2312.06635. [4] Qwen Team, Qwen3-Next, 2025. [5] Qwen Team, Qwen3.5, 2026. [6] AI21 Labs, Jamba, 2024. [7] Ainslie et al., GQA, EMNLP 2023. [8] DeepSeek-AI, DeepSeek-V3, 2024. [9] Singhania et al., Loki, NeurIPS 2024. [10] Xu et al., ThinK, ICLR 2025.