acceptodds
Under review as a conference paper at ICLR 2027

The Entropy Bias of Hybrid Attention: Training Models to Be Sparse-Ready

Abstract

Hybrid language models improve long-context efficiency by concentrating global information exchange into a small number of full-attention layers. We find that these layers exhibit a systematic entropy bias: compared with full-attention Transformers, their attention is more diffuse, leaving less mass recoverable under a fixed sparse budget and making them less robust to sparsification. Importantly, this behavior is not an inherent limitation of hybrid architectures, but an optimization outcome that can be shaped during training. Through controlled pretraining experiments, we show that an architecture-aware concentration prior reduces entropy and substantially improves sparse long-context performance. The gains extend across hybrid architectures, model sizes, and longer contexts, and remain evident with learned QK gains and trained sparse indexers. Entropy saturation further offers a practical guide to prior strength. Together, these findings establish architecture-aware attention concentration as a key training principle for sparse-ready hybrid models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Hybrid Attention, Sparse Attention, Attention Entropy

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C26MOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why the six full-attention layers in 3:1 SWA/GDN hybrid Transformers are less sparse-ready than a 24-layer full-attention Transformer. In controlled pretraining, the hybrids develop higher attention entropy and lower oracle top-k retained mass, while a fixed QK concentration scale applied during pretraining improves 16K and 64K sparse RULER performance and preserves short-context ability; the SWA experiments also show gains with a trained sparse indexer. The problem is important and the mechanism is clearly presented, with the main remaining questions concerning compute-matched attribution and the transfer from oracle diagnostics to practical sparse serving.

Critical strengths
  1. The paper asks a concrete and relevant systems/representation question and supports it with matched Full, SWA-hybrid, and GDN-hybrid pretraining runs. The entropy, retained-mass, layer/head retrieval, and 16K/64K RULER measurements form a coherent characterization of the reported bias.
  2. The mechanism is unusually transparent. The retained-mass identity and the fixed-logit entropy/top-k derivatives explain why concentration should reduce the local sparse-approximation error, while the post-hoc-versus-training comparison shows that adapting the model during pretraining matters beyond simply sharpening an existing checkpoint.
  3. The ablations cover scale strength, application scope, model size, introduction time, context extension, learned QK gains, and a trained indexer. The consistent SWA improvements at tight oracle budgets and at 64K, together with the preservation of answer mass and NLL under the trained prior, make the proposed intervention easy to interpret.
Critical weaknesses
  1. The architectural attribution is not compute-matched. The Full baseline has 24 full-attention layers whereas each hybrid has six, yet all models consume the same token budget; this changes the amount of global-attention computation and the optimization path. Without an equal-FLOP or otherwise compute-matched control, the entropy gap is not isolated as an effect of hybridization itself.
  2. Most sparse-robustness results use oracle per-head top-k selection, which requires computing the dense attention scores being approximated. The practical indexer study covers only the SWA model at 64K with one shared top-2,048 indexer, and the paper reports no end-to-end latency, memory, or indexer-overhead measurements; the LongBench changes are also small and mixed. Thus the evidence does not yet quantify the serving benefit implied by “sparse-ready.”
  3. The generality and statistical support are narrower than the headline wording. The study uses one corpus and one 3:1 pattern with two efficient components at roughly 0.15–0.85B parameters; model-size, learned-QK, context-extension, and trained-indexer results are largely SWA-only, with no alternate ratios or windows. Each training condition appears to be a single run, RULER has 50 examples per task, and uncertainty intervals are not reported for the main RULER/LongBench comparisons.

Justification of Recommendation

I recommend weak acceptance because the coherent entropy and retained-mass analysis, interpretable training-time intervention, and supporting ablations establish a useful contribution to sparse robustness. The decisive limitations are the unmatched global-attention compute, reliance on oracle top-k evaluation, and lack of uncertainty estimates for the main comparisons. I would rate the paper higher with compute-matched architectural controls, repeated-run uncertainty estimates, and end-to-end measurements of the trained indexer’s latency, memory use, and overhead.

Questions for authors
  1. Can the authors add a compute-matched Full-versus-hybrid control (for example, equal training FLOPs or matched global-attention interactions) and report whether the entropy and top-k gaps persist? This would clarify which part of the effect is attributable to the hybrid architecture rather than its different attention-compute budget.
  2. For the trained sparse indexer, what are the end-to-end prefill/decode latency, peak KV memory, and indexer cost at 64K across the reported budgets? Can the same measurement protocol be run for GDN or another hybrid component so that oracle retained mass can be connected to an actual deployment trade-off?
  3. Can the authors report multiple-seed or paired-bootstrap uncertainty for the key benchmark deltas and select the concentration scale on a calibration split before evaluating held-out natural long-context tasks? How stable are the chosen scales and gains under unseen context lengths, hybrid ratios, or window sizes?
4 comments
AuthorsOpenReview ↗

Response to Reviewer C26M Part I

W1 + Q1. Do the entropy and retained-mass gaps persist at matched training compute?

R: In the main 24-layer comparison, we counted the query–key pairs allowed by causal attention to estimate attention FLOPs, then selected the archived Full checkpoint whose cumulative training FLOPs were closest to those of the final SWA hybrid. The Full checkpoint had reached 98.6% of the hybrid's estimated training FLOPs. At matched layer positions, the SWA hybrid still had 0.445 nats higher mean entropy and retained 7.69 percentage points less attention mass at Top-128. Both gaps therefore persist at nearly equal estimated training compute.

W2.1 + Q2. Indexer latency and memory costs versus dense global attention

R: Using the same 24-layer SWA hybrid checkpoint, we compare native FlashAttention 2 in the six global layers with learned Indexer sparsification at K=2048K=2048. At 64K, 128K, and 256K, sparsification reduces prefill latency by 28.2%, 49.1%, and 65.2%, respectively, including Indexer overhead. Decode latency falls by 35.2%, 52.7%, and 67.6%, respectively.

ContextPrefill, dense → sparsePrefill reductionDecode, dense → sparseDecode reductionIndexer time included in sparse prefill
64K329.8 → 236.7 ms28.2%616.8 → 399.4 ms35.2%35.9 ms (15.2%)
128K1,021.4 → 519.4 ms49.1%878.7 → 416.1 ms52.7%117.7 ms (22.7%)
256K3,502.4 → 1,218.3 ms65.2%1,406.2 → 455.7 ms67.6%416.3 ms (34.2%)

We measure prefill at batch size 1 and time 128 decode steps at batch size 16, starting from each path’s prefilled KV cache. Each number is the median of six warmed-up CUDA graph replays on the same GPU. Indexer-based sparse attention still requires the full main KV cache at runtime, so the main cache uses the same amount of GPU memory as in the dense hybrid. Each of the six global layers stores one 32-dimensional Indexer key per token. Each of the 24 layers stores four 96-dimensional key heads and four 96-dimensional value heads per token in the main cache. At the same cache length, batch size, and precision, the additional Indexer K cache increases total cache memory by 6×1×3224×2×4×96≈1.04%\frac{6\times 1\times 32}{24\times 2\times 4\times 96}\approx 1.04\%.

We also benchmarked the GDN hybrid under the same timing protocol at K=2048K=2048. Both paths use the same GDN checkpoint and recurrent kernels. The sparse path uses an untrained, randomly initialized Indexer to score and select keys in its six global layers. This experiment simulates the runtime cost of Indexer-based sparsification on GDN. The measurements address efficiency alone.

GDN contextPrefill, dense → sparsePrefill reductionDecode, dense → sparseDecode reductionIndexer time included in sparse prefill
64K359.0 → 263.0 ms26.7%553.0 → 336.5 ms39.2%34.8 ms (13.2%)
128K1,078.4 → 571.6 ms47.0%816.4 → 352.8 ms56.8%114.4 ms (20.0%)
256K3,621.0 → 1,326.6 ms63.4%1,336.7 → 388.9 ms70.9%405.0 ms (30.5%)

W2.2. LongBench gains after matched QA adaptation

R: To understand the small LongBench gap in the manuscript, we manually inspected the original generations. Our sub-billion-parameter base models often continued generating after an answer, sometimes starting a new question. Unrelated continuation lowers token-overlap F1 even when the answer is already present. In our matched rerun on 1,350 examples across seven short-answer QA tasks, 99.2% of the g=1.44g=1.44 base outputs and 99.0% of the g=2.56g=2.56 base outputs reached the maximum generation length.

For a fair comparison, we adapted both checkpoints with rank-16 LoRA using the same Dolly closed-QA and information-extraction examples, screened for lexical overlap with LongBench. We used identical adapter initialization and training settings, with 262,144 supervised tokens per model. The loss covered the answer and its stop token. We kept the backbone and existing Indexer weights frozen. After adaptation, the rates of hitting the length cap fell to 26.8% and 19.9%, respectively.

We evaluated both checkpoints before and after LoRA on all 21 LongBench v1 tasks, with the same trained Indexer budget (K=2048K=2048), prompts, decoding settings, and official scoring.

Official scoreg=1.44g=1.44 Base → LoRAg=2.56g=2.56 Base → LoRA
LongBench AVG2121.75 → 23.4721.80 → 25.25
Short-answer QA (7 tasks)9.94 → 18.4410.06 → 20.06
Multi-document QA (4 tasks)9.68 → 13.929.87 → 16.91

The AVG21 gap grows from +0.05 to +1.77 points after matched adaptation. In Appendix H of the revised manuscript, we retain the original base-model results and add a separate table of LoRA-adapted scores by task category, along with the output analysis and adaptation protocol.

AuthorsOpenReview ↗

Response to Reviewer C26M Part II

W3 + Q3. Benchmark uncertainty and transfer of the calibrated scale

R: 1. Benchmark uncertainty. For the fixed-scale comparisons at K=2048K=2048, we computed 95% confidence intervals using a paired bootstrap that resamples examples within each task and recomputes the benchmark macro-average. The RULER result comes from the original evaluation, and the LongBench results come from the matched rerun. The intervals quantify example-level uncertainty conditional on the evaluated checkpoints.

Benchmark and settingg=2.56g=2.56 minus g=1.44g=1.44 (points)95% CI
RULER, original evaluation+9.10[6.38, 11.83]
LongBench AVG21, before LoRA+0.05[−0.48, 0.57]
LongBench AVG21, after LoRA+1.77[1.07, 2.47]

2. Scale calibration and held-out RULER. On a separate RULER calibration split, we chose the smallest candidate gg that retained at least 50% of the attention mass at Top-128 and increased the NLL on the last 256 input tokens by at most 0.10 nats/token relative to g=1.44g=1.44 at the same training progress. This rule selected g=1.96g=1.96 at both 3.3% and 16.3% of pretraining. After freezing the choice, we evaluated the final 3:1 SWA checkpoints on 50 disjoint RULER examples. Relative to g=1.44g=1.44, the selected scale improved the oracle Top-128 and Top-512 scores by 9.10 and 12.37 points, respectively.

3. Held-out natural QA. We applied the frozen choice to 72 LongBench v1 questions outside the calibration split, with 24 questions each from Qasper, MultiFieldQA-en, and HotpotQA. We evaluated both checkpoints with FP32 dense attention and per-head oracle Top-128 on the same inputs. In this evaluation, 282 of 288 outputs reached the generation limit, and the base models often started another question after answering. That continuation can lower token-overlap F1. For the results below, we applied the same rule to all four conditions, cutting each output before the next question marker when one appeared and then applying the standard QA F1 metric.

Task (F1, truncated before the next question)g=1.44g=1.44 FP32 denseg=1.44g=1.44 Top-128g=1.96g=1.96 FP32 denseg=1.96g=1.96 Top-128
Qasper23.0522.1724.7923.31
MultiFieldQA-en37.9630.1033.1135.49
HotpotQA13.154.8216.4017.40
Three-task macro-average24.7219.0324.7725.40

At Top-128, the calibrated scale improved F1 on all three tasks and raised the macro-average by 6.37 points, from 19.03 to 25.40. At this budget, g=1.96g=1.96 also preserved its dense macro-average.

4. Hybrid ratios and context length. At fixed g=1.44g=1.44 and window size, the entropy gap is positive at all four SWA-to-full layer ratios (1:1, 2:1, 3:1, and 5:1), and the Top-128 retained-mass gap widens as full-attention layers become less frequent (Appendix G). In the separate context-extension experiment (Section 5.4), after continued pretraining to 64K, the Top-128 RULER score is 2.12 at g=1.00g=1.00 and 32.76 at g=2.56g=2.56. Taken together, these results give us reason to expect the concentration prior to be useful across hybrid layer ratios and at retrieval distances beyond those tested. Appendix F.5 describes how to probe a few candidate scales early in pretraining, using Full as a reference. This offers a low-cost way to identify a suitable scale for different model sizes and architectural configurations, including different layer ratios and window sizes.

We hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process.

Dear Reviewer C26M,

Thank you again for your thoughtful review and for highlighting the evidence that would inform your recommendation. In our rebuttal, we focused on the concerns you raised about compute matching for the global-attention comparison, uncertainty across repeated runs. We also addressed the trained indexer’s end-to-end latency, memory use, and overhead.

As the discussion period draws to a close, we would appreciate hearing whether our response addresses your main concerns. If any questions remain, we would be happy to clarify. If the additional evidence changes your assessment, we would appreciate an update to your evaluation.

Best regards,

The Authors

Reviewer_C26MOpenReview ↗

Thank you for the response

Thank you for your response. My concerns have largely been addressed, and I will maintain my score.

Reviewer_ESj9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores the sparse readiness of hybrid language models. It finds that the full-attention layers of the hybrid models show more diffuse attention than matched layers in a full-attention Transformer, retaining less attention mass under a fixed sparse budget. The authors propose a simple training-time intervention: applying a fixed logit scale to the hybrid's full-attention layers. Controlled experiments show substantial improvements in sparse long-context retrieval, with further evaluations across model sizes, context extension, learned QK gains, and trained sparse indexers.

Critical strengths

I particularly appreciate the pretraining-from-scratch experiments. The matched comparisons control the corpus, example order, token budget, and training schedule, making the evidence more convincing than comparisons between independently developed pretrained models. The models receive approximately 100 training tokens per non-embedding parameter; the largest model is pretrained on 64.5B tokens before an additional 34.9B-token context-extension stage. These are substantial budgets for the model sizes studied, although they do not establish convergence or generalization to larger-scale training.

The intervention is simple, and the gains are substantial, particularly for SWA hybrids. The authors also address several important alternative explanations through learned-QK and post-hoc scaling comparisons, and go beyond oracle selection by evaluating trained indexers. These experiments make the contribution more compelling than a temperature-scaling observation alone.

Critical weaknesses
  1. Selecting the scale in a realistic pretraining run remains unclear.

Performance varies substantially with g, and the best reported settings differ across hybrid architectures and evaluation objectives. In SWA, g=2.56 gives the best dense and top-128 RULER scores among the tested scales, whereas general ability is highest at g=1.44. While entropy saturation is proposed as a guide, its reliability as an early selection criterion is not established. This limits the practical guidance, since sweeping several full pretraining runs would be expensive at production scale and the paper favors introducing the prior early.

  1. The origin and generality of the entropy bias remain open.

The empirical observation is well supported within the tested setup, but the explanation for why hybrid architectures develop higher entropy is less conclusive. Broader retrieval activity across heads does not directly explain more diffuse attention within a head. I appreciate the existing size ablations, but they primarily establish the benefit of the intervention rather than directly test the hybrid-versus-Full entropy gap across scales. The models also remain below 1B parameters and use a fixed hybrid pattern within one pretraining mixture. A complete causal explanation is not necessary for the contribution to be valuable, but the evidence for a general training principle remains limited to this experimental setting.

  1. The incremental practical benefit of sparsifying an already-hybrid model is not quantified.

Hybridization already reduces the number of full-attention layers. These remaining layers may still dominate attention cost at long contexts. However, the paper does not quantify the additional end-to-end benefit relative to the same hybrid with dense global layers. The trained-indexer results establish accuracy under learned token selection, but not the corresponding efficiency gains after accounting for indexer overhead.

Questions for authors
  1. Can a short pilot run or smaller model identify a useful g that transfers to a larger run and unseen downstream tasks? In particular, does entropy saturation early in training predict a suitable scale for the final checkpoint?

  2. Which aspects of the architecture or training setup do you expect to drive the entropy bias? Do the available checkpoints provide evidence that the hybrid-versus-Full entropy gap remains stable across model sizes or as the training budget increases?

  3. What incremental latency and memory benefits does learned-indexer sparsification provide over the same hybrid with dense global layers? Representative measurements at different context lengths, including indexer overhead and separate prefill/decode results, would be helpful.

  4. How do you interpret the substantial NIAH gains alongside the limited and mixed LongBench improvements? Can you provide uncertainty estimates for the main benchmark comparisons?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ESj9 Part I

Thank you for your detailed review and valuable feedback! Below, we address the comments you raised:

W1 + Q1. Choosing a logit scale without a full training sweep

R: As discussed in Appendix F, the SWA hybrid exhibits substantially higher attention entropy than Full early in pretraining at the same logit scale. We now compare the SWA hybrid's entropy throughout pretraining across all five values of gg. The table shows mean full-attention entropy in nats at selected points.

Pretraining progressg=1.00g=1.00g=1.44g=1.44g=1.96g=1.96g=2.56g=2.56g=3.24g=3.24
3.3%8.1547.2576.4236.0515.734
16.3%7.9026.7585.9545.5765.291
100%7.8486.5535.8275.4185.298

Diminishing returns are already visible at 3.3% of pretraining. Raising gg from 1.44 to 1.96 lowers entropy by 0.83 nats, while the larger increase from 2.56 to 3.24 lowers it by 0.32 nats. The 0.83-nat gap also exceeds the 0.704-nat drop in the g=1.44g=1.44 run over the rest of training. At the same early point, the g=1.96g=1.96 hybrid is close to Full in entropy (6.423 vs. 6.596 nats), making g=1.96g=1.96 a reasonable candidate. Short pilots at a few scales can use Full as a reference to narrow the choice of gg before full pretraining. Appendix F.5 and Figure 6 in the revised manuscript provide the full trajectories and measurement details.

W2 + Q2. Origins of the entropy bias and its persistence across model sizes

R: Appendix F is devoted to when the entropy bias emerges and what might explain it. In the main SWA comparison, the entropy gap is already visible at 3.3% of pretraining and remains at the final checkpoint. We extended the matched Full–Hybrid comparison across model sizes. At the end of pretraining, the 12-, 16-, 20-, and 24-layer SWA hybrids have 0.386, 0.350, 0.318, and 0.486 nats higher mean entropy, respectively, than their Full counterparts at matched layer positions, using the same 16K inputs and logit scale (g=1.00g=1.00). The gap varies over training and is positive at the final checkpoint for all four sizes.

Appendices F.1–F.2 examine whether weaker Q/K learning could explain SWA's higher entropy. From mid-training to the final checkpoint, SWA's Q projection changes more than Full's at all six matched layers. On a fixed retrieval probe, SWA's final normalized Q and K answer-loss gradients are 1.41 and 1.44 times those of Full, respectively, weakening this explanation. Appendix F.3 shows that the relationship between retrieval activity and entropy depends on the architecture: retrieval-active heads are more concentrated than other heads in the same layer in SWA and more diffuse in GDN. The proportion of full-attention layers may also shape how long-range retrieval is distributed across the network. In Appendix G, the entropy gap is positive at all four SWA-to-full layer ratios, and the Top-128 retained-mass gap widens as full-attention layers become less frequent. These results offer indirect evidence that the layer ratio may contribute to the entropy bias. Identifying the full mechanism will require further investigation.

W3 + Q3. Indexer latency and memory costs versus dense global attention

R: Using the same 24-layer SWA hybrid checkpoint, we compare native FlashAttention 2 in the six global layers with learned Indexer sparsification at K=2048K=2048. At 64K, 128K, and 256K, sparsification reduces prefill latency by 28.2%, 49.1%, and 65.2%, respectively, including Indexer overhead. Decode latency falls by 35.2%, 52.7%, and 67.6%, respectively.

ContextPrefill, dense → sparsePrefill reductionDecode, dense → sparseDecode reductionIndexer time included in sparse prefill
64K329.8 → 236.7 ms28.2%616.8 → 399.4 ms35.2%35.9 ms (15.2%)
128K1,021.4 → 519.4 ms49.1%878.7 → 416.1 ms52.7%117.7 ms (22.7%)
256K3,502.4 → 1,218.3 ms65.2%1,406.2 → 455.7 ms67.6%416.3 ms (34.2%)

We measure prefill at batch size 1 and time 128 decode steps at batch size 16, starting from each path’s prefilled KV cache. Each number is the median of six warmed-up CUDA graph replays on the same GPU. Indexer-based sparse attention still requires the full main KV cache at runtime, so the main cache uses the same amount of GPU memory as in the dense hybrid. Each of the six global layers stores one 32-dimensional Indexer key per token. Each of the 24 layers stores four 96-dimensional key heads and four 96-dimensional value heads per token in the main cache. At the same cache length, batch size, and precision, the additional Indexer K cache increases total cache memory by 6×1×3224×2×4×96≈1.04%\frac{6\times 1\times 32}{24\times 2\times 4\times 96}\approx 1.04\%.

AuthorsOpenReview ↗

Response to Reviewer ESj9 Part II

Q4. NIAH–LongBench performance differences and benchmark uncertainty

R: To understand the small LongBench gap in the manuscript, we manually inspected the original generations. Our sub-billion-parameter base models often continued generating after an answer, sometimes starting a new question. Unrelated continuation lowers token-overlap F1 even when the answer is already present. In our matched rerun on 1,350 examples across seven short-answer QA tasks, 99.2% of the g=1.44g=1.44 base outputs and 99.0% of the g=2.56g=2.56 base outputs reached the maximum generation length.

For a fair comparison, we adapted both checkpoints with rank-16 LoRA using the same Dolly closed-QA and information-extraction examples, screened for lexical overlap with LongBench. We used identical adapter initialization and training settings, with 262,144 supervised tokens per model. The loss covered the answer and its stop token. We kept the backbone and existing Indexer weights frozen. After adaptation, the rates of hitting the length cap fell to 26.8% and 19.9%, respectively.

We evaluated both checkpoints before and after LoRA on all 21 LongBench v1 tasks, with the same trained Indexer budget (K=2048K=2048), prompts, decoding settings, and official scoring.

Official scoreg=1.44g=1.44 Base → LoRAg=2.56g=2.56 Base → LoRA
LongBench AVG2121.75 → 23.4721.80 → 25.25
Short-answer QA (7 tasks)9.94 → 18.4410.06 → 20.06
Multi-document QA (4 tasks)9.68 → 13.929.87 → 16.91

The AVG21 gap grows from +0.05 to +1.77 points after matched adaptation. The widening is 1.73 points. In Appendix H of the revised manuscript, we retain the original base-model results and add a separate table of LoRA-adapted scores by task category, along with the output analysis and adaptation protocol.

For the fixed-scale comparisons at K=2048K=2048, we computed 95% confidence intervals using a paired bootstrap that resamples examples within each task and recomputes the benchmark macro-average. The RULER result comes from the original evaluation, and the LongBench results come from the matched rerun. The intervals quantify example-level uncertainty conditional on the evaluated checkpoints.

Benchmark and settingg=2.56g=2.56 minus g=1.44g=1.44 (points)95% CI
RULER, original evaluation+9.10[6.38, 11.83]
LongBench AVG21, before LoRA+0.05[−0.48, 0.57]
LongBench AVG21, after LoRA+1.77[1.07, 2.47]

We hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process.

Dear Reviewer ESj9,

Thank you again for your valuable comments on our submission. As the discussion phase draws to a close, we would like to kindly ask whether our responses have adequately addressed your concerns. If any questions remain or if further clarification would be helpful, please do not hesitate to let us know; we would be happy to provide additional details.

If you find our responses satisfactory, we would greatly appreciate it if you would consider adjusting your evaluation accordingly.

We sincerely look forward to your feedback.

Best regards,

The Authors

Reviewer_67JBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of sparsifying full attention in hybrid architectures (SWA/GDN + Full Attention). In initial experiments, the authors show that both SWA and GDN hybrids have full attention layers with a high entropy and that retain less probability mass from an oracle top-k. This is correlated to lower retrieval performance. As a fix, the authors propose to introduce a scaler (aka inverse temperature) to (normalized) QK logits during training. Experiments by sweeping and learning scalers shows substantial improvements in RULER, particularly on NIAH tasks. Gains on LongBench are more modest.

Critical strengths
  • I really liked the idea of investigating how full attention layers in hybrid models behave under sparsification. This kind of analysis is novel to the best of my knowledge.
  • The paper includes two strong representative hybrid models: Attention + SWA and Attention + GDN.
  • The analysis in term of entropy, retained-mass, and sparse evaluation provides an interesting view of this problem.
  • The experiments are simple to define and seem to produce substantial signal differences on downstream task quality.
  • The post-hoc scaling experiment is also useful: it basically shows that reducing entropy alone is not enough to achieve the same benefits of training with a good scale.
Critical weaknesses

While I really liked the empirical motivation of this paper, I found several shortcomings:

  • The conclusions are architecture-specific and I believe the paper needs a clearer positioning against prior work. For example, the proposed concentration prior is a fixed inverse temperature, while the supporting temperature and truncation results are already established (e.g., Velickovic et al., 2025 and Nakanishi, 2025).

  • The main novelty in this paper is on the hybrid-specific analysis and its consequences for sparsification. However, the paper does not vary the hybrid ratio, which can greatly impact the results. For example, full attention models have 24 global attention layers, while the hybrids have only 6. I think this would help clarifying how much the robustness difference comes from having fewer retrieval opportunities.

  • Furthermore, I think the paper would be much stronger if comparisons with related approaches, such as SSMax and ASEntmax, were included in the paper. ASEntmax is a strong baseline as it also deals with sparsity and long-context generalization.

  • The most appealing practical gains are on retrieval, making the broader "sparse-readiness" conclusion less convincing to me.

  • With trained indexers, increasing the fixed scale improves RULER from 26.57 to 35.67, but LongBench changes only from 21.74 to 21.81. This concerns me as to what exactly the fixed scale is improving in this work.

Questions for authors
  • Can you provide the same logit-scale sweep for the Full Attention baseline? Does scaling improve hybrids more than Full Attention?

  • How do the entropy gap and sparsification robustness change with the hybrid ratio?

  • Can you compare with SSMax and/or ASEntmax under the same training setup? What exactly the fixed-scale method adds beyond these approaches?

  • Why are the gains much larger on NIAH than on LongBench and non-NIAH RULER tasks? Here I'm interested to see what happens when many tokens must remain relevant (rather than just a small number of needles).

  • Is it possible to choose the scale without running a full training sweep?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 67JB Part I

Thank you for your detailed review and valuable feedback! Below, we address the comments you raised:

W1 + W3 + Q3. Novelty relative to prior length-scaling methods

R: We thank the reviewer for the suggestions. The cited methods address a different problem. As discussed in the Introduction (line 46) and §2.3, SSMax and ASEntmax aim to preserve effective attention as the context grows longer. Their scaling factors depend on log⁡n\log n, where nn is the number of visible keys, to adjust attention concentration or support as nn increases.

Our study asks how hybrid architecture changes the entropy of its full-attention layers and how that shift affects sparsification quality. The Hybrid and Full models process exactly the same context length, and matched queries see the same number of keys in their full-attention layers. The entropy shift appears when the architecture changes at fixed nn.

We agree that using inverse temperature as a concentration prior is an established technique. Our contribution is to identify the entropy shift in hybrid full-attention layers and show through controlled pretraining ablations that a simple, fixed concentration prior improves task performance under the sparse budget.

W2 + Q2. How hybrid ratio affects entropy and sparse robustness

R: We pretrained four 24-layer SWA hybrids with different ratios of sliding-window to full-attention layers, using the same g=1.44g=1.44 and training token budget. We measure entropy and retained mass on the same RULER prompts at 16K, matching each hybrid's full-attention layer positions to the corresponding layers in Full. The entropy gap is positive at all four hybrid ratios, and the Top-128 mass gap widens as full-attention layers become less frequent.

Model (SWA:Full)Full-attention layersEntropy gap (nats)Top128 mass gap (pp)Top128 retrained RULER
Full24——40.85
Hybrid 1:112+0.374−6.6736.58
Hybrid 2:18+0.796−10.8720.66
Hybrid 3:16+0.762−12.0324.14
Hybrid 5:14+0.840−12.7026.43

Entropy and retained-mass gaps are Hybrid minus Full at matched layer positions. We have added Appendix G to the revised manuscript to describe the experimental setup and discuss these results.

W4 + W5 + Q4. Do retrieval gains extend to natural QA?

R: To understand the small LongBench gap in the manuscript, we manually inspected the original generations. Our sub-billion-parameter base models often continued generating after an answer, sometimes starting a new question. Unrelated continuation lowers token-overlap F1 even when the answer is already present. In our matched rerun on 1,350 examples across seven short-answer QA tasks, 99.2% of the g=1.44g=1.44 base outputs and 99.0% of the g=2.56g=2.56 base outputs reached the maximum generation length.

For a fair comparison, we adapted both checkpoints with rank-16 LoRA using the same Dolly closed-QA and information-extraction examples, screened for lexical overlap with LongBench. We used identical adapter initialization and training settings, with 262,144 supervised tokens per model. The loss covered the answer and its stop token. We kept the backbone and existing Indexer weights frozen. After adaptation, the rates of hitting the length cap fell to 26.8% and 19.9%, respectively.

We evaluated both checkpoints before and after LoRA on all 21 LongBench v1 tasks, with the same trained Indexer budget (K=2048K=2048), prompts, decoding settings, and official scoring.

Official scoreg=1.44g=1.44 Base → LoRAg=2.56g=2.56 Base → LoRA
LongBench AVG2121.75 → 23.4721.80 → 25.25
Short-answer QA (7 tasks)9.94 → 18.4410.06 → 20.06
Multi-document QA (4 tasks)9.68 → 13.929.87 → 16.91

The AVG21 gap grows from +0.05 to +1.77 points after matched adaptation. In Appendix H of the revised manuscript, we retain the original base-model results and add a separate table of LoRA-adapted scores by task category, along with the output analysis and adaptation protocol.

AuthorsOpenReview ↗

Response to Reviewer 67JB Part II

Q1. Does logit scaling benefit Hybrid more than Full?

R: Increasing the training-time scale from g=1.44g=1.44 to g=2.56g=2.56 improves Full Attention at Top128, with a substantially larger gain in the SWA hybrid. We pretrained a 24-layer Full Transformer from scratch with g=2.56g=2.56 and reevaluated both Full checkpoints on the original 16K RULER benchmark. The SWA scores are from our original scale ablation on the same examples.

Oracle budgetFull: g=1.44g=1.44 → g=2.56g=2.56Full gain3:1 SWA: g=1.44g=1.44 → g=2.56g=2.56SWA gain
Top12839.78 → 43.78+4.0025.95 → 45.42+19.47
Top51241.66 → 43.72+2.0628.38 → 49.02+20.65

Q5. Can a short pilot identify a useful logit scale?

R: As discussed in Appendix F, the SWA hybrid exhibits substantially higher attention entropy than Full early in pretraining at the same logit scale. We now compare the SWA hybrid's entropy throughout pretraining across all five values of gg. The table shows mean full-attention entropy in nats at selected points.

Pretraining progressg=1.00g=1.00g=1.44g=1.44g=1.96g=1.96g=2.56g=2.56g=3.24g=3.24
3.3%8.1547.2576.4236.0515.734
16.3%7.9026.7585.9545.5765.291
100%7.8486.5535.8275.4185.298

At 3.3% of pretraining, the SWA hybrid's entropy is 0.834 nats lower at g=1.96g=1.96 than at g=1.44g=1.44, a larger difference than the 0.704-nat drop in the g=1.44g=1.44 run over the rest of training. At the same point, the g=1.96g=1.96 hybrid is close to Full in entropy (6.423 vs. 6.596 nats), making g=1.96g=1.96 a reasonable candidate. Short pilots at a few scales can use Full as a reference to narrow the choice of gg before full pretraining. Appendix F.5 and Figure 6 in the revised manuscript provide the full trajectories and measurement details.

We hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process.

Dear Reviewer 67JB,

Thank you again for your valuable comments on our submission. As the discussion phase draws to a close, we would like to kindly ask whether our responses have adequately addressed your concerns. If any questions remain or if further clarification would be helpful, please do not hesitate to let us know; we would be happy to provide additional details.

If you find our responses satisfactory, we would greatly appreciate it if you would consider adjusting your evaluation accordingly.

We sincerely look forward to your feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion Deadline Imminent — We Would Appreciate Your Feedback on Our Rebuttal

Dear Reviewers,

Thank you for your careful and constructive reviews. We appreciate your recognition of the question motivating this work. As more hybrid models sparsify their remaining global attention layers, understanding how those layers respond to sparsification becomes increasingly important. We are also grateful that you highlighted the value of our controlled pretraining-from-scratch experiments.

In the rebuttal, we added experiments across hybrid ratios, evaluated an early scale-selection rule on held-out tasks, and reported paired-bootstrap confidence intervals for RULER and LongBench. We also measured prefill and decode latency for the trained indexer, including its overhead, and quantified its KV-cache cost.

With about one day left in the discussion period, we have yet to receive a follow-up on our rebuttal. If you have time to revisit our responses, we would be grateful to hear whether the new evidence addresses your main concerns and which questions remain open. Even a brief reply would help us see where the paper needs further clarification or evidence and make the final manuscript more complete. We would be glad to discuss any remaining questions before the deadline.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.