acceptodds
Under review as a conference paper at ICLR 2027

HLA: Expressive Hybrid Linear Attention via Chunk-Wise Dynamic Mixing

Abstract

Linear attention enables efficient long-context autoregressive decoding by compressing history into recurrent states, but this compression can make selective access to sparse and distant information difficult. Existing chunk-based extensions increase memory capacity, yet learned chunk-mixing coefficients may remain fixed with respect to input content and therefore cannot adapt historical access to each query. We introduce Hybrid Linear Attention (HLA), a query-dependent chunk-level attention mechanism for Gated DeltaNet (GDN). HLA represents each completed chunk as an exact affine state transition and computes content-dependent routing gates from compact, self-attentively pooled representatives. Each gate interpolates the corresponding historical transition with the identity map, controlling both the chunk's additive memory and its transformation of earlier states. Effective-support regularization further encourages concentrated routing for sparse inference. We evaluate HLA under both pretrained adaptation and from-scratch training. Across Qwen3.5 models from 0.8B to 9B, HLA consistently improves over native GDN and fixed chunk mixing, with gains of up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER. In a controlled from-scratch 1.3B setting trained for 100B tokens with a 4K context, HLA also improves RULER performance from 4K to 32K, with gains increasing from 0.83 points at 4K to 4.22 points at 32K. These results demonstrate that query-dependent composition of recurrent memory improves long-context modeling and remains effective beyond the training context while using compact per-chunk affine summaries.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear Attention, Long-Context Modeling, Efficient Inference, Long-Range Retrieval, Gated DeltaNet

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2iU6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HLA, which incorporates query-dependent, chunk-level routing into the GDN (Gated Dynamic Network) layer. Each completed chunk is precisely represented as an affine transition (Aj,Bj)(A_j, B_j). A sigmoid gate is then computed using representative keys—derived via self-attentive pooling—to interpolate between the original transition and an identity mapping. During training, two regularization terms—budget and effective-support—are introduced to encourage sparse routing.

Critical strengths

The formulation is concise. It leverages the affine structure of the Gated Linear Unit (GLU/GDN) to generate precise chunk summaries, with the gate simultaneously modulating both the additive memory and the transformation of the previous state. When all w=1w=1, the model strictly reduces to a standard GDN.

In from-scratch training experiments, the model demonstrates significant gains in length extrapolation. For instance, the S1 metric improves from 11.2 to 67.4 when evaluated at a sequence length of 32K; this finding is further supported by the distance analysis presented in Figures 5–6.

Critical weaknesses
  • It loses the core advantage of linear attention. Each chunk requires storing a d×dd \times d matrix AjA_j, resulting in a cache size of O(Nd2)O(N d^2) that grows linearly with TT. The decoding cost, O(NPd+Md2)O(NPd + Md^2), also increases with NN. As shown in Fig. 3c, with L=256L=256, the cache size remains 50% of a standard KV cache; essentially, it functions as a KV cache with a compression ratio of L/dL/d. Given this, the authors should provide a direct comparison with block-sparse attention.

  • Key baselines and related work are missing. Chunk-routing sparse attention methods—such as MoBA, NSA, and InfLLM—are conceptually highly similar to HLA. Log-Linear Attention (Guo et al., 2025) also utilizes a gated-dynamic-network (GDN) approach to maintain multiple chunk-based or hierarchical states, while requiring only O(log⁡T)O(\log T) memory. The "from-scratch" training experiments lack a softmax-based or hybrid-architecture baseline.

  • The "from-scratch" results are not particularly compelling. The absolute scores are very low (an average of 24.6 at 4K), and the overall performance gains are primarily driven by the S1 component. Metrics such as CWE, FWE, and VT show significant declines—for instance, FWE drops from 17.28 to 2.72 at 8K—yet the paper offers no analysis of why this occurs. The absence of perplexity evaluations and short-context tasks makes it impossible to determine whether the model’s general-purpose capabilities have been compromised.

  • The efficiency analysis is incomplete. The actual sparsity during inference—specifically the average number of selected chunks, MM—is not reported; yet, this is the fundamental basis for the speedup achieved via the θ\theta threshold. Latency is measured only up to a sequence length of 4K, at which point the model is already 14–19% slower; no scaling curves showing performance as a function of sequence length are provided. During training, continuous weights are used, and no chunks are skipped. The cost of the "reverse scan" operation is O(TNd2)O(TNd^2)—quadratic in terms of TT—but this is not discussed in the paper.

  • There are inconsistencies in the reported numerical values. In Fig. 3, the value for "Native GDN" is 22.66, whereas in Table 3, the value for the 0.8B model is 22.86. The default configuration for the ablation study (L=256,p=32L=256, p=32) yields 29.62, while the 0.8B HLA model in Table 3 shows 28.43—a figure that happens to match the result for p=64p=64 in the ablation study. The paper should clarify which specific model and configuration were used for the ablation experiments. A pooling window size of p=16p=16 is used during training, while p=32p=32 is used for evaluation; this represents a train/test mismatch that requires explanation.

Questions for authors

How does HLA compare to methods like MoBA/NSA or Log-Linear GDN under the same memory budget?

Can the Qwen3.5 experiments be conducted in settings where the sequence length exceeds 32K and the softmax layer becomes a bottleneck?

What is the average value of MM during inference? What are the latency and memory usage at 32K and 128K sequence lengths?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2iU6(1/2)

Q1. How does HLA differ from MoBA, NSA, InfLLM, and Log-Linear Attention? Are memory-matched comparisons available?

A1. We clarify the motivation and key distinctions of HLA as follows.

  1. HLA addresses a fundamental limitation of recurrent linear attention: the progressive forgetting of distant information. In GDN, early information undergoes successive state transformations that can attenuate important historical signals. HLA addresses this problem through query-dependent composition of complete affine transitions, allowing relevant information to bypass interfering updates.

  2. HLA differs from existing methods in how historical information is accessed and composed. MoBA [1], NSA [2], and InfLLM [3] selectively access token-level KV representations rather than modifying recurrent state transitions. Log-Linear Attention [4] is more closely related but maintains a predefined hierarchy of recurrent states and uses input-dependent weights to combine their readouts. In contrast, HLA applies query-dependent gates to complete affine transitions before composition, controlling both each chunk's contribution and how it transforms earlier memory.

  3. HLA prioritizes improved recurrent attention expressivity while introducing an explicit memory–efficiency trade-off. Unlike native GDN, HLA maintains chunk-level summaries and therefore does not retain constant-size historical memory. We acknowledge this additional cost and focus on the trade-off between long-range retrieval, memory consumption, and inference efficiency.

References

[1] Lu et al. MoBA: Mixture of Block Attention for Long-Context LLMs. 2025. arXiv

[2] Yuan et al. Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention. 2025. arXiv

[3] Xiao et al. InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory. 2024. arXiv

[4] Guo et al. Log-Linear Attention. 2025. arXiv

Q2. The from-scratch scores are low and gains are concentrated in S1. Are general language-modeling capabilities preserved?

A2. We clarify the following points. We have added these results and the corresponding analysis to Appendix A.3 of the revised manuscript.

  1. The from-scratch experiments are designed for controlled architectural comparison rather than achieving state-of-the-art absolute performance. We follow the publicly released GDN implementation and train both models under the same 100B-token budget and 4K training context. Our evaluation focuses on the relative improvements over the matched GDN baseline under identical conditions.

  2. HLA preserves and improves general language-modeling capabilities. We additionally evaluate both from-scratch checkpoints on held-out perplexity and standard short-context benchmarks:

    ModelPPL ↓PIQA ↑WinoGrande ↑ARC-Easy ↑
    GDN-1.3B17.96270.6751.7845.79
    HLA-1.3B17.02772.1452.0947.22

    HLA achieves lower perplexity and higher accuracy across all three benchmarks, demonstrating that its long-range retrieval improvements do not come at the expense of general language-modeling performance.

  3. The observed regressions are task-specific rather than evidence of a general capability degradation. As clarified in our response to Reviewer 5Gzk (Q1), improvements are not limited to S1 across our evaluations.

AuthorsOpenReview ↗

Response to Reviewer 2iU6(2/2)

Q3. What is the actual inference sparsity, and how do latency and memory scale to longer contexts?

A3. We provide additional inference measurements on Qwen3.5-0.8B at 4K, 32K, and 128K contexts. The 4K latency results for Native GDN and sparse HLA are consistent with Table 4 of our submission. We have incorporated the additional latency, memory, and routing sparsity measurements into Appendix A.4.

ContextMethodPrefill (ms) ↓Decode (ms/tok) ↓Cache (MiB) ↓Peak Mem. (GiB) ↓Accessed Chunks (Prefill / Decode) ↓
4KNative GDN40.365.9066.841.78–
4KHLA Dense69.4913.12227.091.968.50 / 17.00
4KHLA Sparse46.007.01227.091.963.20 / 4.32
32KNative GDN677.6013.74402.843.93–
32KHLA Dense1,231.51104.641,607.099.7864.50 / 129.00
32KHLA Sparse713.9016.521,607.099.788.46 / 9.13
128KNative GDN4,797.9714.961,554.8411.30–
128KHLA Dense8,752.89213.617,141.0951.28256.50 / 513.00
128KHLA Sparse5,132.8819.967,141.0951.3410.41 / 10.23
  1. Sparse routing substantially reduces historical computation. At 128K, the average number of accessed chunks decreases from 513 to 10.23 during decoding, yielding a 10.70× speedup over dense HLA. The corresponding speedup at 32K is 6.33×.

  2. Sparse HLA maintains moderate inference overhead relative to native GDN. Decoding overhead is 18.8%, 20.2%, and 33.4% at 4K, 32K, and 128K, respectively. Prefill overhead is 14.0%, 5.4%, and 7.0%, respectively.

  3. Sparse routing reduces computation without reducing persistent cache storage. HLA retains historical affine summaries for future queries, resulting in higher memory consumption than native GDN. Nevertheless, sparse routing substantially reduces the computational cost of query-dependent historical composition.

Q4. Continuous training visits all historical chunks and has quadratic sequence-length complexity.

A4. We clarify that HLA introduces additional training complexity from query-dependent historical transition composition. Since training uses continuous routing over all historical chunks, the composition and scoring costs are O(TNd2)O(TNd^2) and (O(TNPd)), respectively, resulting in quadratic complexity with respect to sequence length. The reverse readout avoids materializing per-query state matrices but does not eliminate this asymptotic cost.

To quantify the practical overhead, we additionally profile the training efficiency of 1.3B GDN and HLA models across different context lengths.

ContextMethodThroughput (tokens/s) ↑Peak Memory (GiB) ↓
4KGDN22,949.8215.56
4KHLA21,953.7121.32
8KGDN13,797.7228.80
8KHLA13,423.1635.12
16KGDN14,492.2155.27
16KHLA14,009.1368.74

HLA retains 95.7%–97.3% of native GDN training throughput, with a 21.9%–37.0% increase in peak memory. These results demonstrate that the additional training overhead remains manageable within the evaluated context lengths, despite the higher asymptotic complexity. We have added the training throughput and memory analysis to Appendix A.5.

Q5. The reported configurations are inconsistent, and Qwen3.5 is not tested at sufficiently long contexts.

A5. We clarify these two concerns as follows.

  1. The configuration inconsistencies have been corrected. We have corrected the pooling-window typo and the numerical discrepancies in Figure 3, as detailed in our response to Reviewer 5Gzk (Q4).

  2. We extend Qwen3.5 evaluation to substantially longer contexts. We additionally evaluate Qwen3.5-0.8B on RULER at 8K, 16K, and 32K, where HLA consistently outperforms Native GDN and MHLA, with gains persisting even after excluding all Single Needle tasks (see Reviewer 5Gzk, Q2). We also report LongBench-V2 results with a 32K input budget and additional inference measurements at 32K and 128K, including latency, memory consumption, and routing sparsity (see Q3).

Thank you for the detailed response and the additional experiments. The new efficiency measurements at longer contexts, the training throughput analysis, the perplexity and short-context results, and the extended Qwen3.5 evaluations are helpful, and I appreciate the corrections to the reported configurations.

I also appreciate the clarification in Appendix A.2 that HLA targets the progressive attenuation of information in recurrent memory, rather than token-level KV selection as in MoBA/NSA, and I agree that the mechanisms differ. That said, from a practitioner’s perspective, these methods are alternative ways to spend a growing memory budget on improving long-range retrieval. The same applies to closely related approaches such as Log-Linear Attention and Native Hybrid Attention, which also augment linear attention with additional historical states. An empirical comparison with some of them under a comparable memory budget would therefore still be valuable for understanding where HLA stands, even if the mechanisms are not directly analogous.

Reviewer_HSPuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a chunk-wise linear-attention architecture and evaluates it under both pretrained adaptation and from-scratch training. The core idea is to retain representative keys and chunk-level affine state transitions for each chunk. During decoding, the current query is used to compute the relevance of each chunk and assign it a routing weight. Chunks with higher weights exert a stronger influence through their state transitions, whereas those with lower weights have a weaker influence and may be skipped entirely.

The approach shares a similar intuition with sparse attention methods for Transformer architectures, such as NSA, but focuses on improving the token-mixing capability of linear modules, particularly their ability to retrieve distant information. Although the method improves model performance, it introduces additional computation and storage costs. In particular, retaining representative keys breaks the property of recurrent linear-attention models that their inference cache does not grow with sequence length.

Critical strengths

The proposed method effectively improves the long-range information retrieval capability of linear token-mixing modules, at the cost of additional KV-cache storage. The paper evaluates both performance and efficiency, demonstrating the effectiveness of the method.

Critical weaknesses
  1. The method departs from the property of linear token-mixing modules that they do not retain a KV cache. With a chunk size of 256, HLA’s affine states require 50% of the storage of a KV cache. This is a non-negligible storage overhead, particularly for longer sequences.

  2. Since HLA introduces additional storage costs, I believe it should be compared with a broader range of sparse attention methods, as many such methods can reduce KV-cache storage to less than 50%, or even substantially lower, while maintaining performance.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Response to Reviewer HSPu

Q1. HLA loses constant-memory recurrence and introduces substantial historical storage.

A1. We acknowledge that HLA introduces additional historical storage and clarify this design trade-off as follows.

  1. The additional memory directly addresses a fundamental limitation of recurrent linear attention. GDN compresses history into a fixed-size state, where important early information can be progressively attenuated by subsequent updates. HLA preserves historical affine transitions and enables query-dependent recomposition, allowing relevant information to bypass interfering updates.

  2. The additional storage provides substantial gains in long-range retrieval. At 32K, HLA improves RULER S1 accuracy from 11.2% to 67.4% over native GDN. We further characterize this trade-off by combining the chunk-size ablation in Figure 3(a) with the corresponding historical-memory analysis in Figure 3(c). These results show that an intermediate chunk size of 256 achieves the highest accuracy among the evaluated configurations while requiring less historical-state storage than finer chunking.

Q2. Additional sparse-attention comparisons are needed under comparable memory budgets.

A2. We clarify that HLA addresses a different architectural problem from existing sparse-attention methods. We have added a discussion in Appendix A.2 to clarify the distinction between HLA and existing sparse-attention methods, including the role of inference-time sparsification and the associated memory trade-off. The changes are highlighted in blue.

  1. HLA targets the progressive forgetting of distant information in recurrent linear attention. In GDN, historical information undergoes successive state transformations that may attenuate important signals. HLA addresses this limitation through query-dependent composition of complete affine transitions, improving the expressivity and long-range retrieval capability of recurrent linear attention.

  2. Existing sparse-attention methods focus on a different mechanism. MoBA, NSA, and InfLLM selectively access token-level KV representations to improve softmax attention efficiency. In contrast, HLA modifies how recurrent memory is composed, jointly controlling each historical chunk's additive contribution and its transformation of earlier information. Our comparisons with GDN and MHLA therefore aim to isolate this architectural contribution.

  3. Sparse selection is an inference optimization rather than the core contribution of HLA. Our central innovation is the query-dependent recomposition of historical GDN transitions to mitigate information loss from intervening updates. Inference-time thresholding simply allows low-weight transitions to be skipped, reducing historical state access. The proposed mechanism remains applicable without sparse selection. Thus, HLA should not be viewed as another sparse-attention method, but as an approach to improving recurrent linear attention expressivity.

Thanks for your reply. My fundamental assessment of this paper remains unchanged, and I will maintain my score.

AuthorsOpenReview ↗

Follow-up Clarification

We thank the reviewer for the follow-up. To directly address the remaining concerns about memory efficiency and comparisons with related methods, we conducted an additional comparison with Log-Linear GDN.

We acknowledge that the original chunk size (L=256L=256) was relatively small, primarily chosen for a controlled comparison with MHLA under the same chunk size. To evaluate HLA under a smaller memory budget, we retrained HLA following the Log-Linear GDN setup with the same model scale, training dataset, 50B-token budget, and 16K training context, using L=4096L=4096 and p=32p=32.

We report the results on the full 13-task RULER benchmark at 16K:

MethodS1S2S3MK1MK2MK3MQMVVTCWEFWESQuADHotpotAvg.Excl. S1
Log-Linear GDN100.020.88.616.20.20.013.7517.252.801.662.679.4013.6015.928.91
HLA100.020.212.217.80.20.014.619.74.925.9214.4010.8013.2018.0011.16
Persistent inference cacheLog-Linear GDNHLA
Per layer, single sequence25.35 MiB18.11 MiB
Full model, single sequence532.30 MiB380.26 MiB
Full model, batch size 168.32 GiB5.94 GiB

HLA achieves 2.08 percentage points higher average accuracy with 28.6% less persistent inference cache. The improvement also holds when excluding S1 (+2.25 points).

These results provide direct evidence that HLA can achieve a favorable accuracy–memory trade-off without requiring the fine-grained chunking used in our original experiments. We hope this additional comparison helps address the reviewer's remaining concerns.

Reviewer_5GzkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hybrid Linear Attention (HLA), a query-dependent chunk-routing extension to Gated DeltaNet (GDN). Each completed chunk is represented by an exact affine transition, while self-attentively pooled keys enable dynamic routing. Unlike conventional chunk weighting, HLA modulates both additive memory and the transformation of earlier states.

The authors evaluate HLA by adapting pretrained Qwen3.5 models (0.8B–9B) with frozen backbones and sparsity-regularized routing, as well as training a 1.3B model from scratch on 100B tokens. HLA improves LongBench-V2 and RULER averages over GDN and fixed chunk mixing. The from-scratch model also shows gains up to 32K contexts, though improvements are concentrated in single-needle retrieval. These benefits come with additional computation and context-dependent memory costs from caching and scoring historical chunks.

Critical strengths
  • Query-conditioned affine transition control. HLA modulates both memory writes and transformations of earlier states, reducing interference from irrelevant chunks. The reverse-readout algorithm also avoids materializing per-query state matrices.

  • Broad experimental validation.Results across four Qwen3.5 scales and from-scratch training (Tables 1–3) demonstrate consistent average improvements. Additional analyses support stronger long-range single-needle retrieval.

Critical weaknesses
  • Limited evidence for broad long-context generalization. Table 1's gains beyond 4K are largely driven by S1 retrieval. At 16K, S1 improves from 23.80 to 97.00, while the average of the other 12 tasks drops from 6.66 to 3.89. Several tasks also regress. Moreover, Qwen3.5 evaluations use a 4K input budget, so Table 3 does not establish generalization beyond 4K.

  • Missing ablations and unclear experimental configurations. The budget and support losses are not ablated separately, leaving their contributions to sparse routing unclear. Adaptation also uses different pooling windows during training (p=16) and evaluation (p=32), while Figure 3 reports HLA/Native GDN scores of 29.62/22.66, versus 28.43/22.86 for the 0.8B model in Table 3, without explaining configuration differences.

Questions for authors
  • Why are long-context gains concentrated in single-needle retrieval? Excluding S1, HLA underperforms GDN at 8K–32K (Table 1). What explains the regressions on VT, FWE, and other tasks? Can these regressions be mitigated without sacrificing S1 gains?

  • Does pretrained HLA generalize beyond 4K? Qwen3.5 evaluations use a 4K input budget. Could the authors report results at 8K, 16K, and 32K to verify length generalization?

  • What are the individual contributions of the routing regularizers? Could the authors separately ablate the budget and support losses, reporting their effects on accuracy and routing sparsity?

  • What explains the configuration and result discrepancies? Why does Qwen3.5 adaptation use p=16 for training but p=32 for evaluation? How does this affect accuracy, and why do Figure 3 and Table 3 report different scores?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5Gzk(1/2)

Q1. Long-context gains are dominated by S1, while other tasks regress. What explains this, and can the regressions be mitigated?

A1. We clarify these concerns with additional experiments.

  1. The dominance of S1 is specific to the from-scratch extrapolation results, rather than a general limitation of HLA. In pretrained Qwen3.5 experiments (Table 2), S1 reaches 100% for both GDN and HLA across all four model scales, so all improvements come from the remaining tasks. Moreover, at 8K, 16K, and 32K, HLA improves over Native GDN by 0.89, 1.94, and 2.54 points, respectively, even after excluding all three Single Needle tasks. These results demonstrate that HLA's benefits extend beyond single-needle retrieval.

  2. The from-scratch regressions can be substantially mitigated by adjusting inference-time routing. HLA primarily targets sparse long-range retrieval, while tasks requiring broader historical integration may benefit from retaining more state transitions. To investigate this trade-off, we reduce the inference threshold from 0.1 to 0.02 using the same from-scratch checkpoint and evaluate on RULER at 16K.

MethodS1 ↑Avg. excl. S1 ↑Full Avg. ↑VT ↑CWE ↑FWE ↑
GDN23.806.667.9822.646.089.80
HLA (θ=0.1\theta=0.1)97.003.8911.051.802.940.20
HLA (θ=0.02\theta=0.02)62.406.4110.7214.745.707.40

Reducing the threshold improves the average over the remaining 12 tasks from 3.89 to 6.41, approaching GDN's 6.66. Meanwhile, S1 remains substantially higher than GDN (62.40 vs. 23.80), despite decreasing from 97.00. These results show that the task-specific regressions can be substantially mitigated while retaining a considerable long-range retrieval advantage, revealing a controllable trade-off between sparse retrieval and broader historical integration.

Q2. Does pretrained HLA generalize beyond 4K?

A2. We extend Qwen3.5-0.8B evaluation to 8K, 16K, and 32K without further training. We report the full 13-task RULER average, together with averages excluding S1 and all three Single Needle tasks.

ContextMethodRULER Avg. ↑Excl. S1 ↑Excl. Single ↑
8KNative GDN80.7479.1475.01
8KMHLA81.0279.4875.37
8KHLA81.4479.9075.90
16KNative GDN76.9575.0370.06
16KMHLA78.0176.1971.45
16KHLA78.4476.6572.00
32KNative GDN72.1569.8364.36
32KMHLA74.1472.0166.51
32KHLA74.5272.4066.90

HLA consistently outperforms both Native GDN and MHLA across all evaluated context lengths. Compared with Native GDN, the full RULER gains increase from 0.70 points at 8K to 2.37 points at 32K. Importantly, improvements persist after excluding all Single Needle tasks, demonstrating that the benefits extend beyond sparse single-needle retrieval.

We further evaluate LongBench-V2 with a 32K input budget:

MethodLongBench-V2 ↑
Native GDN25.45
MHLA25.84
HLA26.84

HLA improves over Native GDN and MHLA by 1.39 and 1.00 percentage points, respectively. Together, these results demonstrate that the pretrained HLA adaptation remains effective at context lengths substantially beyond the 4K adaptation setting. We have incorporated these additional analysis into Appendix A.6.

Q3. What are the individual contributions of the budget and effective-support regularizers?

A3. We provide an additional ablation to evaluate the individual contributions of the two regularizers under identical training and inference settings. We have incorporated this ablation and the corresponding analysis into Section A.1 of the revised manuscript.

ObjectiveLongBench-V2 ↑RULER ↑Gate mass ↓Effective support ↓Accessed chunks ↓
LM only27.6382.684.66.27.6
LM + budget27.8382.842.65.77.4
LM + support28.0383.012.53.14.4
LM + budget + support28.4383.122.22.73.8

The budget term substantially reduces total routing mass (4.6 → 2.6) but has limited impact on actual chunk access. In contrast, the support term encourages more concentrated routing, reducing effective support (6.2 → 3.1) and accessed chunks (7.6 → 4.4). Combining both regularizers reduces accessed chunks by 50% while improving LongBench-V2 and RULER performance. These results demonstrate their complementary benefits for accuracy and routing sparsity.

AuthorsOpenReview ↗

Response to Reviewer 5Gzk(2/2)

Q4. Why use different pooling windows during training and evaluation, and why do Figure 3 and Table 3 disagree?

A4. We clarify these two points as follows.

  1. Pooling window. The reported training window of p=16p=16 in Section 4.1 is a typo. Both training and evaluation use p=32p=32 for the pretrained Qwen3.5 experiments. We have corrected this in the revised manuscript.

  2. Numerical discrepancies. We thank the reviewer for identifying this inconsistency. We have rechecked the experimental records and corrected the ablation results in Figure 3 and the corresponding discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.