- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes HLA, which incorporates query-dependent, chunk-level routing into the GDN (Gated Dynamic Network) layer. Each completed chunk is precisely represented as an affine transition . A sigmoid gate is then computed using representative keys—derived via self-attentive pooling—to interpolate between the original transition and an identity mapping. During training, two regularization terms—budget and effective-support—are introduced to encourage sparse routing.
Critical strengths
The formulation is concise. It leverages the affine structure of the Gated Linear Unit (GLU/GDN) to generate precise chunk summaries, with the gate simultaneously modulating both the additive memory and the transformation of the previous state. When all , the model strictly reduces to a standard GDN.
In from-scratch training experiments, the model demonstrates significant gains in length extrapolation. For instance, the S1 metric improves from 11.2 to 67.4 when evaluated at a sequence length of 32K; this finding is further supported by the distance analysis presented in Figures 5–6.
Critical weaknesses
-
It loses the core advantage of linear attention. Each chunk requires storing a matrix , resulting in a cache size of that grows linearly with . The decoding cost, , also increases with . As shown in Fig. 3c, with , the cache size remains 50% of a standard KV cache; essentially, it functions as a KV cache with a compression ratio of . Given this, the authors should provide a direct comparison with block-sparse attention.
-
Key baselines and related work are missing. Chunk-routing sparse attention methods—such as MoBA, NSA, and InfLLM—are conceptually highly similar to HLA. Log-Linear Attention (Guo et al., 2025) also utilizes a gated-dynamic-network (GDN) approach to maintain multiple chunk-based or hierarchical states, while requiring only memory. The "from-scratch" training experiments lack a softmax-based or hybrid-architecture baseline.
-
The "from-scratch" results are not particularly compelling. The absolute scores are very low (an average of 24.6 at 4K), and the overall performance gains are primarily driven by the S1 component. Metrics such as CWE, FWE, and VT show significant declines—for instance, FWE drops from 17.28 to 2.72 at 8K—yet the paper offers no analysis of why this occurs. The absence of perplexity evaluations and short-context tasks makes it impossible to determine whether the model’s general-purpose capabilities have been compromised.
-
The efficiency analysis is incomplete. The actual sparsity during inference—specifically the average number of selected chunks, —is not reported; yet, this is the fundamental basis for the speedup achieved via the threshold. Latency is measured only up to a sequence length of 4K, at which point the model is already 14–19% slower; no scaling curves showing performance as a function of sequence length are provided. During training, continuous weights are used, and no chunks are skipped. The cost of the "reverse scan" operation is —quadratic in terms of —but this is not discussed in the paper.
-
There are inconsistencies in the reported numerical values. In Fig. 3, the value for "Native GDN" is 22.66, whereas in Table 3, the value for the 0.8B model is 22.86. The default configuration for the ablation study () yields 29.62, while the 0.8B HLA model in Table 3 shows 28.43—a figure that happens to match the result for in the ablation study. The paper should clarify which specific model and configuration were used for the ablation experiments. A pooling window size of is used during training, while is used for evaluation; this represents a train/test mismatch that requires explanation.
Questions for authors
How does HLA compare to methods like MoBA/NSA or Log-Linear GDN under the same memory budget?
Can the Qwen3.5 experiments be conducted in settings where the sequence length exceeds 32K and the softmax layer becomes a bottleneck?
What is the average value of during inference? What are the latency and memory usage at 32K and 128K sequence lengths?
5 comments
Response to Reviewer 2iU6(2/2)
Q3. What is the actual inference sparsity, and how do latency and memory scale to longer contexts?
A3. We provide additional inference measurements on Qwen3.5-0.8B at 4K, 32K, and 128K contexts. The 4K latency results for Native GDN and sparse HLA are consistent with Table 4 of our submission. We have incorporated the additional latency, memory, and routing sparsity measurements into Appendix A.4.
| Context | Method | Prefill (ms) ↓ | Decode (ms/tok) ↓ | Cache (MiB) ↓ | Peak Mem. (GiB) ↓ | Accessed Chunks (Prefill / Decode) ↓ |
|---|---|---|---|---|---|---|
| 4K | Native GDN | 40.36 | 5.90 | 66.84 | 1.78 | – |
| 4K | HLA Dense | 69.49 | 13.12 | 227.09 | 1.96 | 8.50 / 17.00 |
| 4K | HLA Sparse | 46.00 | 7.01 | 227.09 | 1.96 | 3.20 / 4.32 |
| 32K | Native GDN | 677.60 | 13.74 | 402.84 | 3.93 | – |
| 32K | HLA Dense | 1,231.51 | 104.64 | 1,607.09 | 9.78 | 64.50 / 129.00 |
| 32K | HLA Sparse | 713.90 | 16.52 | 1,607.09 | 9.78 | 8.46 / 9.13 |
| 128K | Native GDN | 4,797.97 | 14.96 | 1,554.84 | 11.30 | – |
| 128K | HLA Dense | 8,752.89 | 213.61 | 7,141.09 | 51.28 | 256.50 / 513.00 |
| 128K | HLA Sparse | 5,132.88 | 19.96 | 7,141.09 | 51.34 | 10.41 / 10.23 |
-
Sparse routing substantially reduces historical computation. At 128K, the average number of accessed chunks decreases from 513 to 10.23 during decoding, yielding a 10.70× speedup over dense HLA. The corresponding speedup at 32K is 6.33×.
-
Sparse HLA maintains moderate inference overhead relative to native GDN. Decoding overhead is 18.8%, 20.2%, and 33.4% at 4K, 32K, and 128K, respectively. Prefill overhead is 14.0%, 5.4%, and 7.0%, respectively.
-
Sparse routing reduces computation without reducing persistent cache storage. HLA retains historical affine summaries for future queries, resulting in higher memory consumption than native GDN. Nevertheless, sparse routing substantially reduces the computational cost of query-dependent historical composition.
Q4. Continuous training visits all historical chunks and has quadratic sequence-length complexity.
A4. We clarify that HLA introduces additional training complexity from query-dependent historical transition composition. Since training uses continuous routing over all historical chunks, the composition and scoring costs are and (O(TNPd)), respectively, resulting in quadratic complexity with respect to sequence length. The reverse readout avoids materializing per-query state matrices but does not eliminate this asymptotic cost.
To quantify the practical overhead, we additionally profile the training efficiency of 1.3B GDN and HLA models across different context lengths.
| Context | Method | Throughput (tokens/s) ↑ | Peak Memory (GiB) ↓ |
|---|---|---|---|
| 4K | GDN | 22,949.82 | 15.56 |
| 4K | HLA | 21,953.71 | 21.32 |
| 8K | GDN | 13,797.72 | 28.80 |
| 8K | HLA | 13,423.16 | 35.12 |
| 16K | GDN | 14,492.21 | 55.27 |
| 16K | HLA | 14,009.13 | 68.74 |
HLA retains 95.7%–97.3% of native GDN training throughput, with a 21.9%–37.0% increase in peak memory. These results demonstrate that the additional training overhead remains manageable within the evaluated context lengths, despite the higher asymptotic complexity. We have added the training throughput and memory analysis to Appendix A.5.
Q5. The reported configurations are inconsistent, and Qwen3.5 is not tested at sufficiently long contexts.
A5. We clarify these two concerns as follows.
-
The configuration inconsistencies have been corrected. We have corrected the pooling-window typo and the numerical discrepancies in Figure 3, as detailed in our response to Reviewer 5Gzk (Q4).
-
We extend Qwen3.5 evaluation to substantially longer contexts. We additionally evaluate Qwen3.5-0.8B on RULER at 8K, 16K, and 32K, where HLA consistently outperforms Native GDN and MHLA, with gains persisting even after excluding all Single Needle tasks (see Reviewer 5Gzk, Q2). We also report LongBench-V2 results with a 32K input budget and additional inference measurements at 32K and 128K, including latency, memory consumption, and routing sparsity (see Q3).
Thank you for the detailed response and the additional experiments. The new efficiency measurements at longer contexts, the training throughput analysis, the perplexity and short-context results, and the extended Qwen3.5 evaluations are helpful, and I appreciate the corrections to the reported configurations.
I also appreciate the clarification in Appendix A.2 that HLA targets the progressive attenuation of information in recurrent memory, rather than token-level KV selection as in MoBA/NSA, and I agree that the mechanisms differ. That said, from a practitioner’s perspective, these methods are alternative ways to spend a growing memory budget on improving long-range retrieval. The same applies to closely related approaches such as Log-Linear Attention and Native Hybrid Attention, which also augment linear attention with additional historical states. An empirical comparison with some of them under a comparable memory budget would therefore still be valuable for understanding where HLA stands, even if the mechanisms are not directly analogous.
Response to Reviewer 2iU6(1/2)
Q1. How does HLA differ from MoBA, NSA, InfLLM, and Log-Linear Attention? Are memory-matched comparisons available?
A1. We clarify the motivation and key distinctions of HLA as follows.
HLA addresses a fundamental limitation of recurrent linear attention: the progressive forgetting of distant information. In GDN, early information undergoes successive state transformations that can attenuate important historical signals. HLA addresses this problem through query-dependent composition of complete affine transitions, allowing relevant information to bypass interfering updates.
HLA differs from existing methods in how historical information is accessed and composed. MoBA [1], NSA [2], and InfLLM [3] selectively access token-level KV representations rather than modifying recurrent state transitions. Log-Linear Attention [4] is more closely related but maintains a predefined hierarchy of recurrent states and uses input-dependent weights to combine their readouts. In contrast, HLA applies query-dependent gates to complete affine transitions before composition, controlling both each chunk's contribution and how it transforms earlier memory.
HLA prioritizes improved recurrent attention expressivity while introducing an explicit memory–efficiency trade-off. Unlike native GDN, HLA maintains chunk-level summaries and therefore does not retain constant-size historical memory. We acknowledge this additional cost and focus on the trade-off between long-range retrieval, memory consumption, and inference efficiency.
References
[1] Lu et al. MoBA: Mixture of Block Attention for Long-Context LLMs. 2025. arXiv
[2] Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. 2025. arXiv
[3] Xiao et al. InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory. 2024. arXiv
[4] Guo et al. Log-Linear Attention. 2025. arXiv
Q2. The from-scratch scores are low and gains are concentrated in S1. Are general language-modeling capabilities preserved?
A2. We clarify the following points. We have added these results and the corresponding analysis to Appendix A.3 of the revised manuscript.
The from-scratch experiments are designed for controlled architectural comparison rather than achieving state-of-the-art absolute performance. We follow the publicly released GDN implementation and train both models under the same 100B-token budget and 4K training context. Our evaluation focuses on the relative improvements over the matched GDN baseline under identical conditions.
HLA preserves and improves general language-modeling capabilities. We additionally evaluate both from-scratch checkpoints on held-out perplexity and standard short-context benchmarks:
HLA achieves lower perplexity and higher accuracy across all three benchmarks, demonstrating that its long-range retrieval improvements do not come at the expense of general language-modeling performance.
The observed regressions are task-specific rather than evidence of a general capability degradation. As clarified in our response to Reviewer 5Gzk (Q1), improvements are not limited to S1 across our evaluations.