- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper routes each token to of KV groups, while every group caches all tokens, trading cache storage for fewer decode reads. Routing reuses the sigmoid router with auxiliary loss free balancing from feedforward MoE. GDN plus attention hybrids are trained from scratch at 220M and 3B scale. At 3B model size level, reports 4.59 at LongPPL, versus 4.71 for dense KV4 and 4.58 for dense KV8, with 1.60 to 1.94 times the peak decode throughput of KV8. The paper also gives a FlexAttention query packing scheme for parallel training.
Critical strengths
- The general idea looks good for me.
- Statistical reporting is careful. This paper uses paired bootstrap intervals with stated limits and includes a parameter matched wide FFN control.
Critical weaknesses
I think the technical contribution is limited. For the main idea, I think: (1) token level head routing exists in MoA and MoH, and the router and balancing come unchanged from Wang et al. (2024); (2) Polar Sparsity already performs selective KV group access with full histories for batched decoding. For the experimental setup: (1) training from scratch and enlarging are also configuration choices, not new mechanisms. (2)The FlexAttention packing applies the existing BlockMask interface. There is no ablation that separates added KV capacity from learned routing.
Another shortcoming is the lack of supporting evidence. (1) Long context quality rests solely on LongPPL over 50 GovReport documents of at most 32K tokens. There is no RULER, needle retrieval, or LongBench evaluation. (2) At 3B, the sparse model does not beat dense KV8 and is significantly worse than dense KV16. At , LongPPL worsens as grows. (3) There are no baselines at equal cache memory, such as MLA or KV quantization.
Questions for authors
- How does decoding batch tokens that select different groups? Is router cost timed, and at which batch size does each 3B peak occur?
- What are the RULER or needle retrieval results at 32K and beyond?
- How to explain the LongPPL trend at ?
- How does KV12/A4 compare with MLA or dense baselines at equal cache memory?
7 comments
"For the experimental setup: (1) training from scratch and enlarging G are also configuration choices, not new mechanisms"
We disagree that these are simply configuration choices. The performance degradation of polar sparsity indicate that training from scratch is important as the sparsity increases. Furthermore, without training from scratch the pre training attention compute will scale with the number of total KV groups, rather than the number of active KV groups. Also, enlarging the number of KV groups is a previously unexplored axis of model design. MoA requires a single KV head, MoH fixes kv_heads = model_dim // head_dim in the from scratch experiments, and Polar Sparsity relies on a pretrained model with a fixed number of KV groups. Instead, the central hypothesis of our work depends on the relationship between the number of KV groups and model quality, and exploring that relationship is a large part of the contribution.
"(2)The FlexAttention packing applies the existing BlockMask interface."
We agree. We initially developed a custom kernel to efficiently implement the head sparse attention training on A100 GPUs, but moved to flex attention to enable drop in compatibility on a wide range of hardware, and we see the use of pytorch's flex attention as a positive.
"There is no ablation that separates added KV capacity from learned routing."
We perform an ablation in which the learned routing of the 220M class sparse attention A4 models is replaced by uniform random routing at inference time (keeping router weight gating), and report the results here:
| Model | Routing | LongPPL | Validation CE |
|---|---|---|---|
| Sparse KV8/A4 | Learned | 9.33 | 2.3565 |
| Sparse KV8/A4 | Uniform random | 17.01 | 2.4483 |
| Sparse KV12/A4 | Learned | 9.05 | 2.3517 |
| Sparse KV12/A4 | Uniform random | 26.37 | 2.4900 |
| Sparse KV16/A4 | Learned | 9.03 | 2.3503 |
| Sparse KV16/A4 | Uniform random | 30.24 | 2.5069 |
We observe that the random routing moderately degrades short context ppl but seriously harms longppl results. This is strong evidence that KV groups specialize over the course of training, from scratch random routing experiments are still being trained at the time of writing but will be complete shortly.
" (1) Long context quality rests solely on LongPPL over 50 GovReport documents of at most 32K tokens."
We agree. We found that at the compute budget available, the results of task based evals like RULER were very noisy, with accuracy range of 45.73% across 4 seeds for a dense MHA KV8 model config, preventing us from reliably replicating the RULER results of Bertsch et al (https://allenai.org/papers/olmpool) at the 220M parameter scale with a reasonable number of seeds. However, we found that longppl results were much less noisy, aligning with the finding of Madaan et al (https://arxiv.org/abs/2406.10229) that perplexity based evaluations can be more reliable than task/accuracy based evaluations, especially for small training budgets, and decided to keep longppl as the primary long context evaluation metric for the 220M class of models. For the 3B models, we expand our evaluation pipeline to include HELMET, which combines results from multiple long context benchmarks including RULER. We also perform context extension training on the 3B models by training on an additional 4.1 billion tokens at a 64K sequence length. We report the results in the table below:
HELMET average success rate, higher is better
| Context | Dense KV4 | Dense KV8 | Sparse KV12/A4 | Dense KV16 |
|---|---|---|---|---|
| 8K | 21.15 | 23.72 | 23.99 | 22.57 |
| 16K | 19.04 | 21.74 | 21.15 | 21.01 |
| 32K | 18.26 | 18.23 | 18.60 | 20.36 |
| 64K | 15.29 | 15.56 | 16.44 | 17.24 |
(Note, standard HELMET evaluations us a GPT 4o LLM judge on a portion of the tasks, due to the increased API cost of using an older model we use GLM 5.3 instead)
These results are consistent with the conclusions of our paper. At long context performance the KV12A4 model consistently outperforms dense KV4 and is comparable to KV8. We will update the manuscript to include this table.
"(2) At 3B, the sparse model does not beat dense KV8 and is significantly worse than dense KV16."
This is accurate, but we do not claim that the sparse model with 12 total heads and 4 active heads should be expected to beat either the dense KV8 or KV16. The sparse model (before context extension) achieves 0.4% higher longPPL than the dense KV8 (within bootstrapped CI, not statistically significant difference), and beats it in the HELMET sequence lengths above, with just half of the decode time KV bandwidth. (post context extension difference is 0.65%, still not statistically significant). We include the KV16 checkpoint as a strong baseline that should be expected to act as an upper bound on long context quality, and we should not expect the other models consistently outperform it. Qualitatively we observe a similar pattern as is seen in MoE models, where an MoE model may underperform dense models with the same number of total parameters but significantly outperfirm dense models with equal active parameters and fewer total parameters. Predicting exactly how a given head-sparse attention configuration should perform relative to dense baselines requires expensive scaling studies and is beyond the scope of this paper.
"At A=2, LongPPL worsens as G grows."
We noticed this trend as well. We do not claim that increasing total KV capacity will monotonically improve performance ceteris paribus. While this result does not contradict the claims of our paper (that long context quality can be improved by having more total KV groups than active), it is a notable artifact. We suspect it is related to the very high sparsity ratio of the A2 G16, A2G12 configs, perhaps combined with the use of load balancing hyperparameters optimized for the the sparse KV8A4 model, and we note that similar behavior has been observed in MoE models in which very sparse models become unstable, leading to lower performance. Isolating the precise cause of this trend remains a promising avenue of future work.
" (3) There are no baselines at equal cache memory, such as MLA or KV quantization."
We have 220M scale models with 8 total KV groups with 4 and 2 active groups, which perform similarly to the dense 220M models with 8 kv groups. This gives a comparison against dense baselines with equal cache memory. Methods like MLA/GLA and KV cache quantization are orthogonal to our work, they should not be seen as competing baselines to compare against, since it is possible to train sparse models using both of these methods. To demonstrate this, we train a three new GLA models at the 220M scale: a pair of dense GLA models with 4 and 8 KV groups respectively, and a sparse model with 8 total GLA groups and 4 active, this model performs routing and load balancing in the same way that the GQA based sparse models do. The results are given here:
| Model | LongPPL | WikiText PPL | LAMBADA PPL |
|---|---|---|---|
| Dense GLA, 4 groups | 12.14 | 35.44 | 51.60 |
| Dense GLA, 8 groups | 11.67 | 35.04 | 45.29 |
| Sparse GLA, 8 groups / 4 active | 11.60 | 34.90 | 50.50 |
We also include a table of paired bootstrapped confidence intervals for the longPPL difference:
| Comparison | ΔLongPPL | 95% CI |
|---|---|---|
| Dense GLA8 − Dense GLA4 | −0.4708 | [−0.8037, −0.1788] |
| Sparse GLA8/A4 − Dense GLA4 | −0.5411 | [−0.8649, −0.2916] |
| Sparse GLA8/A4 − Dense GLA8 | −0.0704 | [−0.4131, +0.2349] |
We find that the capacity-quality relationship found in Bertsch et al (https://allenai.org/papers/olmpool) and our work extends from GQA to the GLA case, and larger KV cache memory is associated with better long context performance. Furthermore, GLA architectures drop in seamlessly to our sparsely routed implementation and we replicate the statistically significant long context gains from sparse models in this case as well. (We note that the GLA models tend to perform moderately worse than equivalent cache sized GQA models from our earlier experiments. For the GLA results we re used the hyperparameters that were optimized for dense GQA, and we suspect this explains the difference)
Thank you for your thoughtful review. To clarify, the primary contribution of our work is not the exact combination of routing mechanism and load balancing, or the idea of selective head or KV group access.
Instead, our primary contribution is:
Taken together, these demonstrate that models with head-sparse attention can achieve high quality with low inference cost, replacing a difficult tradeoff decision between quality and efficiency with an easy one.
Responding to the weaknesses outlined in your review:
"(1) token level head routing exists in MoA and MoH, and the router and balancing come unchanged from Wang et al. (2024)"
We agree that token level head routing exists in MoA and MoH, and in other prior work, and we do not claim to have introduced it. MoA does routing over query heads, and forces a shared KV projection to be used for all query experts. The MoH paper describes routing over QKV heads in an MHA setting, but does not discuss routing over KV groups. In their released code, their MoH implementation on the llama model, which uses GQA, does routing over individual query heads regardless of KV head access. This aligns with the focus of the MoH paper, which is on reduced self attention flops relative to dense models with the same number of total query heads, but it does nothing to reduce KV bandwidth unless every query head of a KV group goes unselected. Furthermore, the GQA setting in the MoH paper use a straight through estimator for routing gradients (https://arxiv.org/html/2410.11842v3#S4.SS4), instead of the router weight scaling standard in MoE models that we use in our work. We agree that our implementation uses standard load balancing and routing methods from the MoE literature, as we discuss in the paper. Our goal is to explore the hypothesis that total KV capacity can improve model quality without hurting decode efficiency, not to research novel routing or load balancing mechanisms, so we used the standard methods from the MoE literature, where mature solutions to these problems have already been tested. Exploring these implementation mechanics remains a promising avenue for future work.
"(2) Polar Sparsity already performs selective KV group access with full histories for batched decoding."
We agree, and discuss polar sparsity as the most closely related work in our paper for this reason. However, the method use for polar sparsity 1: are not trained from scratch and require a dense already trained base model, and 2: leads to substantial model quality degradation with sparsity in models using GQA. Even if a model were trained with the intention of using polar sparsity, a model with n:k total:active KV groups and equal query heads per group to the dense baseline would have the training cost of a dense model with n KV groups, while our implementation has the training cost of a dense model with k KV groups. We apply the Polar Sparsity method to the attention layers of our dense 220M models, with the following results:
We trained the polar sparsity selector modules using data from our 32K sequence length prolong training corpus. We observe that for shorter context evaluations like wikitextppl and lambada ppl, the Polar Sparsity procedure leads to moderate performance degradation, substantial degradation in the sparser KV8A2 case, and in longer context evaluation Polar Sparsity leads to substantial drop in performance.