- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies why the six full-attention layers in 3:1 SWA/GDN hybrid Transformers are less sparse-ready than a 24-layer full-attention Transformer. In controlled pretraining, the hybrids develop higher attention entropy and lower oracle top-k retained mass, while a fixed QK concentration scale applied during pretraining improves 16K and 64K sparse RULER performance and preserves short-context ability; the SWA experiments also show gains with a trained sparse indexer. The problem is important and the mechanism is clearly presented, with the main remaining questions concerning compute-matched attribution and the transfer from oracle diagnostics to practical sparse serving.
Critical strengths
- The paper asks a concrete and relevant systems/representation question and supports it with matched Full, SWA-hybrid, and GDN-hybrid pretraining runs. The entropy, retained-mass, layer/head retrieval, and 16K/64K RULER measurements form a coherent characterization of the reported bias.
- The mechanism is unusually transparent. The retained-mass identity and the fixed-logit entropy/top-k derivatives explain why concentration should reduce the local sparse-approximation error, while the post-hoc-versus-training comparison shows that adapting the model during pretraining matters beyond simply sharpening an existing checkpoint.
- The ablations cover scale strength, application scope, model size, introduction time, context extension, learned QK gains, and a trained indexer. The consistent SWA improvements at tight oracle budgets and at 64K, together with the preservation of answer mass and NLL under the trained prior, make the proposed intervention easy to interpret.
Critical weaknesses
- The architectural attribution is not compute-matched. The Full baseline has 24 full-attention layers whereas each hybrid has six, yet all models consume the same token budget; this changes the amount of global-attention computation and the optimization path. Without an equal-FLOP or otherwise compute-matched control, the entropy gap is not isolated as an effect of hybridization itself.
- Most sparse-robustness results use oracle per-head top-k selection, which requires computing the dense attention scores being approximated. The practical indexer study covers only the SWA model at 64K with one shared top-2,048 indexer, and the paper reports no end-to-end latency, memory, or indexer-overhead measurements; the LongBench changes are also small and mixed. Thus the evidence does not yet quantify the serving benefit implied by “sparse-ready.”
- The generality and statistical support are narrower than the headline wording. The study uses one corpus and one 3:1 pattern with two efficient components at roughly 0.15–0.85B parameters; model-size, learned-QK, context-extension, and trained-indexer results are largely SWA-only, with no alternate ratios or windows. Each training condition appears to be a single run, RULER has 50 examples per task, and uncertainty intervals are not reported for the main RULER/LongBench comparisons.
Justification of Recommendation
I recommend weak acceptance because the coherent entropy and retained-mass analysis, interpretable training-time intervention, and supporting ablations establish a useful contribution to sparse robustness. The decisive limitations are the unmatched global-attention compute, reliance on oracle top-k evaluation, and lack of uncertainty estimates for the main comparisons. I would rate the paper higher with compute-matched architectural controls, repeated-run uncertainty estimates, and end-to-end measurements of the trained indexer’s latency, memory use, and overhead.
Questions for authors
- Can the authors add a compute-matched Full-versus-hybrid control (for example, equal training FLOPs or matched global-attention interactions) and report whether the entropy and top-k gaps persist? This would clarify which part of the effect is attributable to the hybrid architecture rather than its different attention-compute budget.
- For the trained sparse indexer, what are the end-to-end prefill/decode latency, peak KV memory, and indexer cost at 64K across the reported budgets? Can the same measurement protocol be run for GDN or another hybrid component so that oracle retained mass can be connected to an actual deployment trade-off?
- Can the authors report multiple-seed or paired-bootstrap uncertainty for the key benchmark deltas and select the concentration scale on a calibration split before evaluating held-out natural long-context tasks? How stable are the chosen scales and gains under unseen context lengths, hybrid ratios, or window sizes?
4 comments
Response to Reviewer C26M Part II
W3 + Q3. Benchmark uncertainty and transfer of the calibrated scale
R: 1. Benchmark uncertainty. For the fixed-scale comparisons at , we computed 95% confidence intervals using a paired bootstrap that resamples examples within each task and recomputes the benchmark macro-average. The RULER result comes from the original evaluation, and the LongBench results come from the matched rerun. The intervals quantify example-level uncertainty conditional on the evaluated checkpoints.
| Benchmark and setting | minus (points) | 95% CI |
|---|---|---|
| RULER, original evaluation | +9.10 | [6.38, 11.83] |
| LongBench AVG21, before LoRA | +0.05 | [−0.48, 0.57] |
| LongBench AVG21, after LoRA | +1.77 | [1.07, 2.47] |
2. Scale calibration and held-out RULER. On a separate RULER calibration split, we chose the smallest candidate that retained at least 50% of the attention mass at Top-128 and increased the NLL on the last 256 input tokens by at most 0.10 nats/token relative to at the same training progress. This rule selected at both 3.3% and 16.3% of pretraining. After freezing the choice, we evaluated the final 3:1 SWA checkpoints on 50 disjoint RULER examples. Relative to , the selected scale improved the oracle Top-128 and Top-512 scores by 9.10 and 12.37 points, respectively.
3. Held-out natural QA. We applied the frozen choice to 72 LongBench v1 questions outside the calibration split, with 24 questions each from Qasper, MultiFieldQA-en, and HotpotQA. We evaluated both checkpoints with FP32 dense attention and per-head oracle Top-128 on the same inputs. In this evaluation, 282 of 288 outputs reached the generation limit, and the base models often started another question after answering. That continuation can lower token-overlap F1. For the results below, we applied the same rule to all four conditions, cutting each output before the next question marker when one appeared and then applying the standard QA F1 metric.
| Task (F1, truncated before the next question) | FP32 dense | Top-128 | FP32 dense | Top-128 |
|---|---|---|---|---|
| Qasper | 23.05 | 22.17 | 24.79 | 23.31 |
| MultiFieldQA-en | 37.96 | 30.10 | 33.11 | 35.49 |
| HotpotQA | 13.15 | 4.82 | 16.40 | 17.40 |
| Three-task macro-average | 24.72 | 19.03 | 24.77 | 25.40 |
At Top-128, the calibrated scale improved F1 on all three tasks and raised the macro-average by 6.37 points, from 19.03 to 25.40. At this budget, also preserved its dense macro-average.
4. Hybrid ratios and context length. At fixed and window size, the entropy gap is positive at all four SWA-to-full layer ratios (1:1, 2:1, 3:1, and 5:1), and the Top-128 retained-mass gap widens as full-attention layers become less frequent (Appendix G). In the separate context-extension experiment (Section 5.4), after continued pretraining to 64K, the Top-128 RULER score is 2.12 at and 32.76 at . Taken together, these results give us reason to expect the concentration prior to be useful across hybrid layer ratios and at retrieval distances beyond those tested. Appendix F.5 describes how to probe a few candidate scales early in pretraining, using Full as a reference. This offers a low-cost way to identify a suitable scale for different model sizes and architectural configurations, including different layer ratios and window sizes.
We hope this response has adequately addressed your concerns. We look forward to the opportunity for additional discussion during the review process.
Dear Reviewer C26M,
Thank you again for your thoughtful review and for highlighting the evidence that would inform your recommendation. In our rebuttal, we focused on the concerns you raised about compute matching for the global-attention comparison, uncertainty across repeated runs. We also addressed the trained indexer’s end-to-end latency, memory use, and overhead.
As the discussion period draws to a close, we would appreciate hearing whether our response addresses your main concerns. If any questions remain, we would be happy to clarify. If the additional evidence changes your assessment, we would appreciate an update to your evaluation.
Best regards,
The Authors
Thank you for the response
Thank you for your response. My concerns have largely been addressed, and I will maintain my score.
Response to Reviewer C26M Part I
W1 + Q1. Do the entropy and retained-mass gaps persist at matched training compute?
R: In the main 24-layer comparison, we counted the query–key pairs allowed by causal attention to estimate attention FLOPs, then selected the archived Full checkpoint whose cumulative training FLOPs were closest to those of the final SWA hybrid. The Full checkpoint had reached 98.6% of the hybrid's estimated training FLOPs. At matched layer positions, the SWA hybrid still had 0.445 nats higher mean entropy and retained 7.69 percentage points less attention mass at Top-128. Both gaps therefore persist at nearly equal estimated training compute.
W2.1 + Q2. Indexer latency and memory costs versus dense global attention
R: Using the same 24-layer SWA hybrid checkpoint, we compare native FlashAttention 2 in the six global layers with learned Indexer sparsification at K=2048. At 64K, 128K, and 256K, sparsification reduces prefill latency by 28.2%, 49.1%, and 65.2%, respectively, including Indexer overhead. Decode latency falls by 35.2%, 52.7%, and 67.6%, respectively.
We measure prefill at batch size 1 and time 128 decode steps at batch size 16, starting from each path’s prefilled KV cache. Each number is the median of six warmed-up CUDA graph replays on the same GPU. Indexer-based sparse attention still requires the full main KV cache at runtime, so the main cache uses the same amount of GPU memory as in the dense hybrid. Each of the six global layers stores one 32-dimensional Indexer key per token. Each of the 24 layers stores four 96-dimensional key heads and four 96-dimensional value heads per token in the main cache. At the same cache length, batch size, and precision, the additional Indexer K cache increases total cache memory by 24×2×4×966×1×32≈1.04%.
We also benchmarked the GDN hybrid under the same timing protocol at K=2048. Both paths use the same GDN checkpoint and recurrent kernels. The sparse path uses an untrained, randomly initialized Indexer to score and select keys in its six global layers. This experiment simulates the runtime cost of Indexer-based sparsification on GDN. The measurements address efficiency alone.
W2.2. LongBench gains after matched QA adaptation
R: To understand the small LongBench gap in the manuscript, we manually inspected the original generations. Our sub-billion-parameter base models often continued generating after an answer, sometimes starting a new question. Unrelated continuation lowers token-overlap F1 even when the answer is already present. In our matched rerun on 1,350 examples across seven short-answer QA tasks, 99.2% of the g=1.44 base outputs and 99.0% of the g=2.56 base outputs reached the maximum generation length.
For a fair comparison, we adapted both checkpoints with rank-16 LoRA using the same Dolly closed-QA and information-extraction examples, screened for lexical overlap with LongBench. We used identical adapter initialization and training settings, with 262,144 supervised tokens per model. The loss covered the answer and its stop token. We kept the backbone and existing Indexer weights frozen. After adaptation, the rates of hitting the length cap fell to 26.8% and 19.9%, respectively.
We evaluated both checkpoints before and after LoRA on all 21 LongBench v1 tasks, with the same trained Indexer budget (K=2048), prompts, decoding settings, and official scoring.
The AVG21 gap grows from +0.05 to +1.77 points after matched adaptation. In Appendix H of the revised manuscript, we retain the original base-model results and add a separate table of LoRA-adapted scores by task category, along with the output analysis and adaptation protocol.