acceptodds
Under review as a conference paper at ICLR 2027

HOFA: Channel-Wise Hybrid Attention via Outlier Factorization

Abstract

KV cache memory remains a primary bottleneck during long-context autoregressive decoding. While prior hybrid architectures mitigate this by alternating exact and linear attention across macro structural axes (depth, width, or sequence), they impose a coarse tradeoff between decoding efficiency and fine-grained associative recall. We introduce Hybrid Outlier-Factorized Attention (HOFA), a channel-wise intra-head decomposition. Inside every attention head, HOFA statically allocates a small -dimensional outlier subspace to exact causal Softmax attention while routing the remaining inlier channels to a recurrent Gated Linear Attention (GLA) state. Hardware profiling demonstrates that this intra-head split reduces the decoding KV-cache footprint by 43.7% and yields up to a +75.9% peak decoding throughput speedup over standard MHA. Pretraining across 70M, 125M, and 350M parameter scales establishes that HOFA matches MHA perplexity and achieves competitive zero-shot reasoning accuracy. We show that this factorization naturally concentrates global associative matching into the outlier channels while the linear recurrence absorbs background context, with retrieval limits closely following a proposed theoretical extreme-value capacity model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid attention, KV-cache, linear attention, gated linear attention, long-context models, associative recall, feature outliers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_R3RvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a new perspective on hybrid attention. They split every head in each layer into a softmax attention head and a linear attention head. The linear attention head does not have KV cache, so it helps with memory saving.

Critical strengths
  • The idea is new. The information in each attention head can be divided into two parts: outlier is harder to preserve and retrieve, so it needs full attention; others can be compressed to the linear attention.
Critical weaknesses
  • The main goal is to reduce the memory usage from the full attention layer. Only when the context length is very large is there a need to reduce the memory usage. But when doing the pre-training of this hybrid model, context length only reaches 1K.
  • rr is not adaptively selected. I think each layer may have an optimal rr, which can help balance the efficiency and performance better.
  • Both ablation and main results should have stronger experimental evidence on HOFA's strength.
Questions for authors

I think HOFA is an interesting architecture. But I have the question: why should we use HOFA-like architecture rather than some popular layer-wise hybrid attention architecture (qwen3.5, qwen3-next ...)? I think the author should at least compare the strength and weakness of these two different architecture.

2 comments
AuthorsOpenReview ↗

Response to Reviewer R3Rv

Thank you for your feedback on long-context scaling, per-layer rr selection, and layer-wise hybrids. We updated the paper to address each point:

[R3Rv: Pre-training context length only reaches 1K, while memory reduction matters most at very large context lengths.]

We agree that testing well past N=1024N=1024 is essential. In Section 3.3 and Appendix E (Figures 8-9), we evaluated 125M models from 1k to 64k tokens (64×64\times extension) on PG-19 [1] with YaRN [2] across 54 books, both zero-shot and after matched continual pretraining at N=16384N=16384 (100 steps, 26.2M tokens). Without continual pretraining, base scalar HOFA beats base MHA at 64k (63.73 vs. 89.94 PPL) and maintains stable perplexity across pre-softmax temperature scaling (Figure 9), since Softmax temperature scaling applies only to the rr outlier channels while the jj inlier channels remain invariant. The scalar HOFA continual pretraining result motivated us to test channel-specific effective memory horizons. Standard GLA uses dense data-dependent vector gating [3], while our parameterization λt,c=λtβh,c\lambda_{t,c} = \lambda_t^{\beta_{h,c}} (βh∈R>0j\beta_h \in \mathbb{R}_{>0}^j, initialized to 1) provides channel-specific decay through jj learned exponents per head without a full vector-gate projection. HOFA (βh\beta_h) achieves lower perplexity than MHA and scalar HOFA at every length from 1k to 64k, both zero-shot (56.24 vs. 89.94 at 64k) and after CPT (28.45 vs. 30.00 at 64k).

[R3Rv: r is not adaptively selected; each layer may have an optimal r.]

To study layer-wise capacity allocation, we measured per-head causal channel power EcE_c and Shannon effective dimension deff(ℓ)=exp⁡(−∑cpcln⁡(pc+10−12))d_{\text{eff}}(\ell) = \exp(-\sum_c p_c \ln(p_c + 10^{-12})) across eight pretrained models from 124M to 8B (Figure 6, Appendix C), which shows systematic depth-wise variation in channel concentration. While adaptive or learnable per-layer rℓr_\ell allocation is left to future work (Section 4.1), using a fixed per-layer rℓr_\ell schedule enables contiguous [B,H,N,rℓ][B, H, N, r_\ell] key storage and fixed-shape Triton kernels without dynamic routing overhead. Under a matched 256-channel budget (rˉ=32\bar{r}=32) at 70M scale (Table 4, Figure 7), moving from uniform Flat HOFA (r=32r=32, 30.96 PPL, 38.72% zero-shot average) to non-uniform schedules improves both metrics: boundary-heavy U-shaped HOFA reaches 30.65 PPL and the highest zero-shot average (39.15%), while Increasing HOFA (r=[16,16,16,16,32,48,48,64]r=[16,16,16,16,32,48,48,64]) achieves the lowest validation perplexity (30.35 PPL, 38.85% average).

[R3Rv: Both ablation and main results need stronger evidence; why use HOFA rather than layer-wise hybrids like Qwen3.5 or Qwen3-Next?]

Layer-wise hybrids such as Qwen3-Next [4] and Qwen3.5 [5] interleave Gated DeltaNet layers with Gated Attention layers in a 3:1 linear-to-full-attention layout, placing full-attention KV caching in one quarter of the layers before accounting for additional GQA compression (and Jamba [6] similarly alternates attention and SSM layers across depth). HOFA makes a different tradeoff: it retains a reduced-dimensional exact Softmax pathway in every layer (r≪dhr \ll d_h keys, with a 50% value-cache floor as r→0r \to 0 and 75% total cache at rˉ=dh/2\bar{r}=d_h/2), and composes orthogonally with GQA/MLA [7, 8] (Appendix I) or depth hybridization itself.

Our 70M depth-axis controls are controlled MHA+GLA baselines rather than direct reproductions or head-to-head evaluations of Qwen3-Next or Qwen3.5; they isolate layer placement under our matched architecture and 256-channel key-budget constraints (a 6-MHA : 2-GLA 75%-cache control and direct comparisons to those production architectures remain future work). In Table 4 (Appendix D), placing full MHA at boundary layers improves the 50%-MHA depth control from 31.54 to 30.86 PPL (38.03% to 38.92% avg), while U-shaped HOFA (30.65 PPL, 39.15%) and Increasing HOFA (30.35 PPL, 38.85%) achieve lower perplexity, and Flat HOFA outperforms the 75%-cache-matched Reduced-QK MHA control [9, 10] (30.96 vs. 30.99 PPL, 38.72% vs. 38.27% avg). At the 350M main scale (14B tokens, Table 1, Figure 5), HOFA matches MHA validation perplexity (15.87 vs. 15.88) and outperforms matched 350M MHA across all six zero-shot reasoning benchmarks.

[1] Rae et al., PG-19, ICLR 2020. [2] Peng et al., YaRN, ICLR 2024. [3] Yang et al., GLA, arXiv:2312.06635. [4] Qwen Team, Qwen3-Next, 2025. [5] Qwen Team, Qwen3.5, 2026. [6] AI21 Labs, Jamba, 2024. [7] Ainslie et al., GQA, EMNLP 2023. [8] DeepSeek-AI, DeepSeek-V3, 2024. [9] Singhania et al., Loki, NeurIPS 2024. [10] Xu et al., ThinK, ICLR 2025.

Thank you for the clarification. I now have a better understanding of the pre-training context length.

The results with adaptive r look promising. I encourage the authors to further refine the methodology and consider whether incorporating adaptive r into the main method could strengthen the paper's contributions.

I will maintain my original score.

Reviewer_JG3uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Outlier-Factorized Attention, which splits query and key channels within each attention head between a small exact Softmax branch and a recurrent Gated Linear Attention branch. Both branches process full-dimensional values, and a learned gate combines their outputs. The paper evaluates language models from 70M to 350M parameters, synthetic associative recall tasks, and attention-layer hardware performance. It reports perplexity comparable to standard multi-head attention, improved downstream scores at 350M, and reduced cache memory and decoding latency in isolated attention benchmarks. Additional analyses examine channel norms, pathway behavior, and a capacity-inspired model of retrieval accuracy.

Critical strengths
  1. Splitting exact and recurrent attention within each head is an interesting design. It reduces key-cache storage while retaining an exact retrieval pathway.

  2. The paper includes actual pretraining experiments beyond synthetic tasks. The 350M model improves over the corresponding MHA baseline on all six downstream benchmarks while maintaining comparable perplexity.

Critical weaknesses
  1. The efficiency claims need clarification. The reported 17% training FLOP reduction is difficult to explain using Equation 7 and the stated configuration. The main decoding speedup also comes from a single attention layer with identity projections, leaving the practical benefit for a complete model unclear.

  2. The comparisons do not fully isolate channel-wise splitting. Appendix G treats equal exact-attention channel counts as equal compute, although HOFA retains full-dimensional values in its exact branch. The depth baseline also deliberately excludes exact attention from boundary layers, where HOFA allocates more capacity. The gains could therefore partly reflect differences in layer placement.

Questions for authors
  1. Can you provide the exact parameter counts and forward/backward FLOP breakdown behind Figure 5? With equal P=350P=350M, N=1024N=1024, and the stated layer schedule, Equation 7 appears to give roughly 0.7% forward FLOP savings. Does the reported 17% come from faster convergence, different model configurations, or another accounting method?
  2. How is FLOP parity enforced in Appendix G? At dh=64d_h=64 and average r=32r=32, the stated formulas give HOFA a quadratic attention term 1.5 times that of a hybrid with 50% full attention. Does the advantage remain under measured compute matching and a depth baseline that also places exact attention at boundary layers?
  3. Can you report end-to-end decoding speed, total cache memory, and retrieval quality for the trained models at longer contexts? The current language-model training uses 1,024-token sequences, while the hardware benchmarks extend to 128k tokens.
2 comments
AuthorsOpenReview ↗

Response to Reviewer JG3u

Thank you for your detailed review and for checking Equation (7). We updated the paper to fix the FLOP accounting, add the missing controls, and include 64k evaluations:

[JG3u: Exact parameter counts and forward/backward FLOP breakdown behind Figure 5, plus single-layer scope of decoding speedup.]

We confirm the Equation (7) calculation: the 17% figure did not come from faster convergence or different model configurations, but from a plotting bug that scaled exact attention by rr instead of (r+dh)(r+d_h) and omitted dense 2PN2PN compute. At 350M (N=1024,L=24,H=8,dh=128,rˉ=22.67N=1024, L=24, H=8, d_h=128, \bar{r}=22.67; 353.8M unique parameters = 302.3M blocks + 51.5M tied vocabulary head), forward FLOPs per sequence are: projections+MLPs 619.2G (618.9G MHA), vocabulary head 105.4G, QK⊤QK^\top 4.6G (25.8G MHA), AVAV 25.8G, and GLA 15.9G, totaling 770.8G (HOFA) vs. 775.9G (MHA), a 0.65% reduction (0.69% at nominal P=350MP=350\text{M} in Eq. 7, where dense 2PN2PN is 93.4% of MHA's forward FLOPs; backward is ≈2×\approx 2\times forward, giving 2.31T vs. 2.33T training FLOPs/seq). At N=128kN=128\text{k}, Equation (7) estimates a 36.90% full-model reduction where attention is 90.2% of MHA's forward FLOPs (distinct from Fig. 1's 43.75% attention-core reduction). Since per-token FLOPs differ by under 0.7% at N=1024N=1024, we dropped the compute claim and plot Figure 5 vs. tokens: at 14B tokens, 350M HOFA matches MHA perplexity (15.87 vs. 15.88) and scores higher on all six zero-shot tasks in Table 1.

Our Section 3.1 benchmark (Figure 1) allocates compact [B,H,N,r][B, H, N, r] keys and measures peak GPU memory at B=1B=1 on an RTX 5090. Peak KV memory drops 43.6% and decode speed reaches 1.74x MHA at 128k, crossing over near N≈10kN \approx 10\text{k} (below which dual-branch launch adds overhead). Section 3.1 states explicitly that this isolates one block with identity projections, while full-model inference also includes projection, MLP, and vocabulary-head compute.

[JG3u: FLOP parity in Appendix G (1.5x quadratic term at d_h=64, r=32) and boundary-matched depth control.]

In Table 4 (now Appendix D), the 70M models share 76.7M stored parameters (51.0M unique weight-tied) and total forward FLOPs per sequence of 111.6G (HOFA) vs. 109.4G (4-layer MHA depth/width, +2.0%), 106.5G (Reduced-QK MHA), and 112.9G (MHA). Because HOFA retains full dh=64d_h=64 values, rˉ=32\bar{r}=32 has a 1.5×1.5\times (r+dh)(r+d_h) exact attention-core term and caches 75% of MHA's KV cache vs. 50% for 4-layer MHA hybrids. We added two controls to separate layer placement from (r+dh)(r+d_h) cost and 75% cache size. First, placing full MHA at boundary layers improves the 50%-MHA depth control from 31.54 to 30.86 PPL (38.03% to 38.92% zero-shot avg), while U-shaped HOFA (30.65, 39.15%) and Increasing HOFA (30.35, 38.85%) keep lower PPL. Second, Reduced-QK MHA [1, 2] (dQK=32d_{QK}=32, full dh=64d_h=64 values, no recurrence) matches HOFA's (r+dh)(r+d_h) quadratic term and 75% KV cache, getting 30.99 PPL (38.27% avg) vs. Flat HOFA's 30.96 (38.72%).

[JG3u: End-to-end decoding speed, total cache memory, and long-context quality for trained models.]

For trained models, heterogeneous 125M and 350M schedules yield 39.6% and 41.1% theoretical KV savings (full-model 64k decode speed and retrieval remain future work). On PG-19 [3] up to 64k (64×64\times extension) with YaRN [4] (Figure 8, Appendix E), zero-shot scalar HOFA beats MHA at 64k (63.73 vs. 89.94 PPL) and maintains stable perplexity across pre-softmax temperature scaling (Figure 9), since temperature scaling applies only to the rr outlier channels while the jj inlier channels remain invariant. The scalar HOFA continual pretraining result motivated us to test channel-specific memory horizons. Standard GLA uses dense data-dependent vector gating [5], while λt,c=λtβh,c\lambda_{t,c} = \lambda_t^{\beta_{h,c}} provides channel-specific decay through jj learned exponents βh∈R>0j\beta_h \in \mathbb{R}_{>0}^j per head without a full vector-gate projection, achieving lower perplexity than MHA and scalar HOFA across 1k-64k both zero-shot (56.24 vs. 89.94 at 64k) and after CPT (28.45 vs. 30.00 at 64k).

[1] Singhania et al., Loki, NeurIPS 2024. [2] Xu et al., ThinK, ICLR 2025. [3] Rae et al., PG-19, ICLR 2020. [4] Peng et al., YaRN, ICLR 2024. [5] Yang et al., GLA, arXiv:2312.06635.

AuthorsOpenReview ↗

Full 350M Decoding Benchmarks and 6:2 (75% KV) Depth Control

To complete our earlier response on Questions 2 and 3, we ran the remaining two experiments and added them to the updated paper:

  • Full 350M decoding speed and memory up to 128k (Section 3.1, Appendix J, Figure 13): We profiled the complete 24-layer 350M U-shaped HOFA model (rˉ=22.67\bar{r}=22.67, 17.7% exact keys; Table 3) against 350M MHA on an RTX 5090 (32 GB, bfloat16) inside PyTorch CUDA Graphs across N∈[512,128k]N \in [512, 128\text{k}]:

  • At B=1B=1, reading the 0.70 GB weights dominates below N≈13kN \approx 13\text{k}. At 128k, HOFA cuts measured KV and state memory by 40.8% (12.10 GB to 7.16 GB) and reaches +43.1% throughput (141.2 vs. 98.7 tok/s, 1.43x).

  • At B=4B=4, sharing the 0.70 GB weight reads across 4 sequences moves the crossover down to N≈3.5kN \approx 3.5\text{k}, reaching +48.8% throughput (332.6 vs. 223.6 tok/s, 1.49x) and -40.7% memory (24.09 GB to 14.28 GB) at 64k. At 128k (B=4B=4), MHA runs out of memory (>32 GB, ~48 GB needed), while HOFA continues decoding at 191.0 tok/s in 28.41 GB.

  • 6:2 Depth-Axis control with 75% KV cache at 70M (Section 3.4, Appendix D, Table 4): Alongside the 4:4 Boundary control (256 key channels, 50% KV cache, 30.86 PPL, 38.92% zero-shot avg), we trained a 6:2 Depth-Axis control (6 MHA + 2 GLA layers, r=[64,64,0,64,64,0,64,64]r=[64, 64, 0, 64, 64, 0, 64, 64]) matching HOFA's 75% total KV cache (768 elements/head) and forward FLOPs (111.6G/seq). Even with 1.5x more exact key channels (384 vs. 256) by dropping the value cache in 2 layers, 6:2 (30.44 PPL, 38.86% zero-shot avg) still trails Increasing HOFA on validation perplexity (30.35 vs. 30.44) and Canonical U-Shaped HOFA on zero-shot average (39.15% vs. 38.86%).

We hope these 6:2 and 350M decoding results answer Questions 2 and 3, and we would appreciate it if you could reconsider your score.

Reviewer_QNkbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HOFA, which partitions the query and key channels within each attention head into a small exact Softmax branch and a recurrent gated linear-attention branch. Both branches use full-width values and their outputs are dynamically blended. The paper evaluates the architecture through attention-kernel profiling, synthetic associative-recall tasks, and language-model pretraining up to 350M parameters. It reports competitive perplexity and downstream accuracy, alongside reduced KV storage and decoding cost. It also proposes an empirical retrieval-capacity model motivated by the Welch bound and extreme-value statistics.

Critical strengths
  1. The paper provides a concrete intra-head design with explicit exact and recurrent pathways, and explains why full-dimensional values impose a nonzero cache floor. The implementation includes custom kernels and training/evaluation entry points.

  2. The paired pretraining comparison is useful: at 350M, HOFA improves over the matched MHA model on all six reported downstream tasks. The ablation suite also attempts to separate routing geometry, layer allocation, and mixing gates.

Critical weaknesses
  1. The main empirical motivation for r≈16 measures the wrong axis. Appendix F describes cumulative feature-channel mass, but benchmark_K_eff.py sorts and sums the key-token axis of attention matrices of shape [B,H,N,N]. The plotting code then divides this token count by d_model and labels it a routing-dimension ratio. This establishes token sparsity, not concentration in 16 Q/K channels. Please replace this analysis with a precisely defined channel-level statistic and a channel-ablation test, or remove its use as evidence for the r-schedule.

  2. The reported training-compute saving and strict FLOP parity controls need correction. The plotting code charges the exact branch proportionally to r, although its AV multiplication uses d_h-dimensional values; Eq. (7) correctly contains r+d_h. Moreover, an average r=d_h/2 gives HOFA a quadratic attention term 1.5 times that of a 50%-MHA depth/width hybrid, before common dense costs. The depth baseline also deliberately excludes attention at boundary layers favored by HOFA. Please recompute Figure 5, report actual parameter/compute budgets, and include a boundary-matched depth control before attributing the advantage to the hybridization axis.

  3. The memory reduction is not measured by the supplied decode benchmark. It reports an r-dimensional key-cache formula but allocates a full d_h-dimensional key cache for HOFA. The throughput experiment also covers one block at fixed r=16, whereas the trained models use heterogeneous schedules and are evaluated at a 1024-token training context. Please implement compact key storage, measure allocated and peak memory, and report full-model generation throughput and quality at a longer context. These measurements are necessary to connect the kernel result to the paper’s long-context model claim.

Questions for authors
  1. Is a separate channel-level implementation available for Figure 9? If the submitted script generated it, can you replace it with per-head Q/K channel contributions and compare fixed-r settings under matched training budgets? This would distinguish an architectural bottleneck learned during training from evidence of pre-existing outlier channels.

  2. After accounting separately for QK, AV, GLA, projections, and the vocabulary head, what is the corrected compute reduction at matched validation perplexity? For the 70M suite, does the result persist against a depth schedule that also places full attention at the boundaries, with actual FLOPs and parameters reported?

  3. Which equations generated Figure 4b, and how were γ and σ estimated?

3 comments
AuthorsOpenReview ↗

Response to Reviewer QNkb

Thank you for pointing the important axis issue, the FLOP error, and the uncompacted key cache. We updated the paper to fix all three:

[QNkb: Figure 9/K_eff axis, channel statistic, and fixed-r comparison.]

The original script sorted along the key-token axis instead of feature channels. We replaced it (now Figure 6, Appendix C) with per-head causal channel power Ec=1dh∑i=1NQi,c2∑j≤iKj,c2E_c = \frac{1}{d_h}\sum_{i=1}^N Q_{i,c}^2 \sum_{j \le i} K_{j,c}^2, normalized energy pc=Ec/∑c′Ec′p_c = E_c / \sum_{c'} E_{c'}, cumulative power C(r)\mathcal{C}(r), and Shannon effective dimension deff(ℓ)=exp⁡(−∑cpcln⁡(pc+10−12))d_{\text{eff}}(\ell) = \exp(-\sum_c p_c \ln(p_c + 10^{-12})) across 8 models (124M-8B). Following your suggestion, we removed Figure 6's use as evidence of rr-schedule optimality and clarified in Sections 1 and 3.2 that HOFA's [0:r][0:r] split is a fixed architectural partition, while optimization concentrates parameter norms within that assigned subspace (shown by recall sweeps for r∈{8,10,16}r \in \{8, 10, 16\} in Figure 4a and weight-norm cliffs at index r−1r-1 for r=8,16r=8, 16 in Figure 4c); Figure 6 only motivates testing depth schedules. Additional uniform rˉ\bar{r} or channel-zeroing sweeps at 70M+ remain future work; under a 256-channel budget (rˉ=32\bar{r}=32, Table 4), Flat HOFA (r=32r=32) gets 30.96 PPL (38.72% avg), U-shaped gets 30.65 (39.15%), and Increasing gets 30.35 (38.85%).

[QNkb: Figure 5 FLOPs, separated parameter/FLOP budgets, and boundary depth control.]

The 17% claim came from scaling exact attention by rr instead of (r+dh)(r+d_h) and omitting dense 2PN2PN compute. At 350M (N=1024,L=24,H=8,dh=128,rˉ=22.67N=1024, L=24, H=8, d_h=128, \bar{r}=22.67; 353.8M unique parameters = 302.3M blocks + 51.5M tied vocabulary head), forward FLOPs per sequence are: projections+MLPs 619.2G (618.9G MHA), vocabulary head 105.4G, QK⊤QK^\top 4.6G (25.8G MHA), AVAV 25.8G, and GLA 15.9G, totaling 770.8G (HOFA) vs. 775.9G (MHA), or -0.65% (-0.69% at nominal P=350MP=350\text{M} in Eq. 7; training ≈3×\approx 3\times forward: 2.31T vs. 2.33T FLOPs/seq; Eq. 7 estimates -36.90% full-model at N=128kN=128\text{k} vs. Fig. 1's 43.75% attention-core reduction). We dropped the compute claim and plot Figure 5 vs. tokens (15.87 vs. 15.88 PPL at 14B tokens).

At 70M (N=1024,L=8,H=8,dh=64,rˉ=32N=1024, L=8, H=8, d_h=64, \bar{r}=32, Table 4, Appendix D), models have 76.7M stored parameters (51.0M unique tied) and total forward FLOPs per sequence of 111.6G (HOFA), 112.9G (MHA), 109.4G (4-layer MHA depth/width), and 106.5G (Reduced-QK MHA). Boundary-matched 50%-MHA improves from 31.54 to 30.86 PPL (38.03% to 38.92% avg), while U-shaped (30.65, 39.15%) and Increasing (30.35, 38.85%) keep lower PPL. Matching HOFA's 1.5×1.5\times (r+dh)(r+d_h) term and 75% KV cache, Reduced-QK MHA [1, 2] (dQK=32d_{QK}=32, no recurrence) gets 30.99 PPL (38.27% avg) vs. Flat HOFA's 30.96 (38.72%).

[QNkb: Compact key storage, peak memory, and long-context quality.]

Our decode benchmark allocates compact [B,H,N,r][B, H, N, r] keys and measures peak GPU memory at B=1B=1 (r=16,dh=128r=16, d_h=128, RTX 5090). At 128k, peak KV memory drops 43.6% and decode speed reaches 1.74x MHA, crossing over near 10k. Section 3.1 notes this isolates one block with identity projections (125M/350M schedules give 39.6%/41.1% theoretical KV savings; full-model 64k speed and retrieval remain future work). On PG-19 [3] up to 64k (64×64\times extension) with YaRN [4] (Figure 8, Appendix E), zero-shot scalar HOFA beats MHA at 64k (63.73 vs. 89.94 PPL) and stays stable across pre-softmax temperatures (Figure 9, Appendix E). The scalar HOFA continual pretraining result motivated us to test channel-specific memory horizons. Standard GLA uses dense data-dependent vector gating [5], while λt,c=λtβh,c\lambda_{t,c}=\lambda_t^{\beta_{h,c}} provides channel-specific decay via jj learned exponents βh∈R>0j\beta_h \in \mathbb{R}_{>0}^j per head without a dense gate projection, beating MHA and scalar HOFA across 1k-64k both zero-shot (56.24 vs. 89.94 at 64k) and after CPT (28.45 vs. 30.00 at 64k).

[QNkb: Equations and fit for Figure 4b.]

Figure 4b fits Equation (11) (from [6, 7], Equations 16-17, Appendix B): P(correct)=100⋅Φ((μ−bn)/σ)P(\text{correct}) = 100\cdot\Phi((\mu - b_n)/\sigma), n=cN−1n=cN-1, bn=2ln⁡nb_n=\sqrt{2\ln n}, μ=γr/(cN−r)\mu=\sqrt{\gamma r / (cN-r)}, r=8r=8, c=1/32c=1/32. Least squares across 30 checkpoints (N∈[640,1200]N \in [640, 1200]) gives γ^=16.11\hat{\gamma}=16.11 and σ^=0.170\hat{\sigma}=0.170.

[1] Singhania et al., Loki, NeurIPS 2024. [2] Xu et al., ThinK, ICLR 2025. [3] Rae et al., PG-19, ICLR 2020. [4] Peng et al., YaRN, ICLR 2024. [5] Yang et al., GLA, arXiv:2312.06635. [6] Welch, 1974. [7] Fisher & Tippett, 1928.

Thanks for the detailed response and revision. The authors have fixed the three issues I previously pointed out (the incorrect axis for KeffK_{\text{eff}} statistics, the 17% training FLOPs calculation discrepancy, and the uncompressed Key cache in single-layer decoding) and added the 70M ablation and 64k PG-19 experiments, effectively improving the rigor of the paper.

However, I believe the existing experiments are still insufficient to support the core conclusions, mainly for the following reasons:

In the 70M comparison (Table 4), after restoring full attention to the boundary layers, the inter-layer hybrid baseline with a 50% KV cache (30.86 PPL, 38.92% zero-shot average) outperforms Flat HOFA with a 75% cache (30.96 PPL, 38.72%) and is very close to U-shaped HOFA with a 75% cache (30.65 PPL, 39.15%), while a fair inter-layer baseline with the same 75% cache (6 MHA + 2 GLA) is still missing. In addition, the gap between Flat HOFA and Reduced-QK MHA with the GLA branch removed is minimal (30.96 vs. 30.99 PPL), indicating that intra-head channel splitting does not show a clear advantage over simpler approaches. Furthermore, decoding speed and measured memory usage are still limited to single-layer projection-free microbenchmarks, and verification of long-context decoding efficiency and associative recall on fully trained models (125M/350M) is still lacking.

In summary, I believe the current content of the paper is still insufficient to support acceptance, and therefore I lean toward maintaining my original score (Rating: 2).

AuthorsOpenReview ↗

Follow-Up: 70M 6:2 (75% KV) Control and Full 350M Decoding Benchmarks

We ran the remaining experiments you asked for and updated the paper:

  • Full 350M decoding speed and memory up to 128k (Section 3.1, Appendix J, Figure 13): We profiled the complete 24-layer 350M U-shaped HOFA model (rˉ=22.67\bar{r}=22.67, 17.7% exact keys; Table 3) against 350M MHA on an RTX 5090 (32 GB, bfloat16) inside PyTorch CUDA Graphs across N∈[512,128k]N \in [512, 128\text{k}]:

  • At B=1B=1, reading the 0.70 GB weights dominates below N≈13kN \approx 13\text{k}. At 128k, HOFA cuts measured KV and state memory by 40.8% (12.10 GB to 7.16 GB) and reaches +43.1% throughput (141.2 vs. 98.7 tok/s, 1.43x).

  • At B=4B=4, sharing the 0.70 GB weight reads across 4 sequences moves the crossover down to N≈3.5kN \approx 3.5\text{k}, reaching +48.8% throughput (332.6 vs. 223.6 tok/s, 1.49x) and -40.7% memory (24.09 GB to 14.28 GB) at 64k. At 128k (B=4B=4), MHA runs out of memory (>32 GB, ~48 GB needed), while HOFA continues decoding at 191.0 tok/s in 28.41 GB.

  • 6:2 Depth-Axis control with 75% KV cache at 70M (Section 3.4, Appendix D, Table 4, Figure 7): We trained the 6:2 Depth-Axis control (6 MHA + 2 GLA layers, r=[64,64,0,64,64,0,64,64]r=[64, 64, 0, 64, 64, 0, 64, 64]) for 2.8B tokens with the same 75% total KV cache (768 elements/head) and forward FLOPs (111.6G/seq) as HOFA. Because 6:2 drops the value cache in its 2 GLA layers and puts that memory into full keys in 6 MHA layers, it gets 1.5x more exact key channels than HOFA (384 vs. 256). Even so, 6:2 gets 30.44 Val PPL and 38.86% zero-shot average (ARC-E 42.1%, ARC-C 23.5%, PIQA 58.9%, Wino 51.2%, OBQA 29.0%, Hella 28.4%). Increasing HOFA still gets lower perplexity (30.35 vs. 30.44), and Canonical U-Shaped HOFA still gets a higher zero-shot average (39.15% vs. 38.86%, leading on 4 of 6 tasks including PIQA +1.3% and OBQA +1.8%).

  • Why 70M uses 75% KV cache, and the role of the GLA branch (Table 3, Figure 9, Figure 11): At 70M (L=8,dh=64L=8, d_h=64), boundary layers make up half the network, so rˉ/dh=50.0%\bar{r}/d_h = 50.0\% (-25.0% KV cache). As the model scales to 125M and 350M (L=24,dh=128L=24, d_h=128), exact keys drop to 20.8% (-39.6% KV) and 17.7% (-41.1% KV in Table 3, saving 40.8% measured memory in Figure 13), while HOFA's zero-shot lead over MHA grows from 4/6 tasks (+0.54% avg) at 70M to 6/6 tasks (+1.62% avg) at 350M (Table 1). Finally, removing the GLA branch (Reduced-QK MHA) drops 70M zero-shot accuracy by -0.45% vs. Flat HOFA (-2.9% on WinoGrande, -2.4% on OBQA) and -0.88% vs. U-Shaped HOFA. HOFA also stays flat across pre-softmax temperature scaling where MHA spikes (Figure 9), since only the rr outlier channels undergo Softmax temperature scaling while the GLA branch is invariant, and on the trained 350M checkpoint (Figure 11) the 17.7% outlier Softmax subspace preserves full long-range effective attention distance across all 24 layers (257.1 vs. 255.9 tokens for 350M MHA).

We hope these 6:2 and 350M decoding results answer your remaining questions. Please let us know if any other experiments are needed, and we would appreciate it if you could reconsider your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

Thank you to all three reviewers for your helpful feedback. We uploaded a revised paper addressing each point raised in the reviews:

  1. Long-context scaling to 64k with YaRN and per-channel decay (Section 3.3, Appendix E, Figures 8-9): We tested 125M models up to 64k64\text{k} (64×64\times extension) on PG-19 using YaRN, both zero-shot and after continual pretraining at N=16384N=16384 (100 steps, 26.2M tokens). Without continual pretraining, base scalar HOFA already achieves lower perplexity than base MHA at 64k under zero-shot YaRN (63.73 vs. 89.94 PPL) and maintains stable perplexity across pre-softmax temperature scaling 1/t∈[1.0,1.5]1/\sqrt{t} \in [1.0, 1.5] (Figure 9), since Softmax temperature scaling applies only to the rr outlier channels while the jj inlier channels remain invariant. The scalar HOFA continual pretraining result motivated us to test channel-specific effective memory horizons. Standard GLA uses dense data-dependent vector gating [Yang et al., 2023], while our parameterization λt,c=λtβh,c\lambda_{t,c} = \lambda_t^{\beta_{h,c}} (βh∈R>0j\beta_h \in \mathbb{R}_{>0}^j, initialized to 1) provides channel-specific decay through jj learned exponents per head without a full vector-gate projection. With βh\beta_h, 125M HOFA achieves lower perplexity than MHA and scalar HOFA at every length from 1k to 64k, both zero-shot (56.24 vs. 89.94 at 64k) and after N=16384N=16384 continual pretraining (28.45 vs. 30.00 at 64k).

  2. Compact key storage in hardware profiling (Figure 1, Section 3.1): Our single-layer decode benchmark now allocates compact [B,H,N,r][B, H, N, r] keys and measures peak allocated GPU memory at B=1B=1 on an RTX 5090 (updated from our earlier RTX 4060 run). At 128k, peak KV memory drops by 43.6% and decode throughput reaches 1.74x MHA. Table 2 also lists the exact batch-size, learning-rate, and warmup hyperparameters used across our training runs.

  3. New 70M controls (Appendix D, Table 4, Figure 7): We trained a boundary-matched depth-axis control (full MHA at boundary layers) and a Reduced-QK MHA control (dQK=32d_{QK}=32, full dh=64d_h=64 values) under the same 2.8B-token and 256-channel key budgets. Appendix D reports parameter counts (76.7M stored / 51.0M unique weight-tied parameters) and states the 75%-vs-50% total KV-cache difference from keeping full dhd_h values.

  4. Feature-channel power diagnostic (Figure 6, Appendix C): We replaced the key-token sparsity calculation with per-head causal feature-channel power Ec=1dh∑i=1NQi,c2∑j≤iKj,c2E_c = \frac{1}{d_h}\sum_{i=1}^N Q_{i,c}^2 \sum_{j \le i} K_{j,c}^2, normalized energy pcp_c, and Shannon effective dimension deff(ℓ)=exp⁡(−∑cpcln⁡(pc+10−12))d_{\text{eff}}(\ell) = \exp(-\sum_c p_c \ln(p_c + 10^{-12})) across eight pretrained models (124M to 8B). Figure 6 serves as post-hoc motivation to evaluate depth-wise rr-schedules rather than evidence of optimality.

  5. Equation (7) FLOP accounting and Figure 5 (Section 2.2, Section 3.3): The earlier 17% training FLOP claim came from a plotting bug that scaled exact attention by rr instead of (r+dh)(r + d_h) and omitted dense 2PN2PN compute. At N=1024N=1024, the dense term 2PN2PN (including projections, MLPs, and the vocabulary head) accounts for 93.4% of the matched MHA model's total forward FLOPs, so HOFA reduces forward FLOPs per token by ~0.7%. Equation (7) yields an equation-derived full-model estimate of 36.90% reduction at N=128kN=128\text{k}, where attention accounts for 90.2% of MHA's forward FLOPs (distinct from Figure 1's 43.75% attention-core reduction, which excludes dense model compute). Since per-token FLOPs at N=1024N=1024 differ by under 0.7%, we dropped the training-FLOP claim and plot Figure 5 against processed tokens, where 350M HOFA matches MHA (15.87 vs. 15.88 PPL after 14B tokens) and scores higher on all six zero-shot tasks (Table 1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.