acceptodds
Under review as a conference paper at ICLR 2027

MIXTURE OF KV GROUPS: TRADING KV-CACHE CAPACITY FOR DECODE BANDWIDTH

Abstract

KV cache memory bandwidth has emerged as a major bottleneck to LLM inference and RL training, especially at long contexts. Grouped-query attention (GQA) has become a widely used method to accelerate decoding by reducing KV-cache traffic, but aggressive KV sharing can impair long-context quality. We study Mixture of KV Groups, which expands KV capacity while routing each token to a small subset of KV groups. Every group retains the complete sequence history, trading additional cache storage for sparse KV memory access. We train 220M-class and 3B hybrid language models from scratch, jointly learning KV representations and routing. At 220M scale and 3B parameter scales we find that Mixture of KV Groups reliably improves on the long context performance of GQA models while preserving much of their decode efficiency. We also develop an attention masking scheme to enable parallel sparse training with FlexAttention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, Transformer Architecture, Attention, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_BpkmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper routes each token to AA of GG KV groups, while every group caches all tokens, trading cache storage for fewer decode reads. Routing reuses the sigmoid router with auxiliary loss free balancing from feedforward MoE. GDN plus attention hybrids are trained from scratch at 220M and 3B scale. At 3B model size level, G=12,A=4G=12, A=4 reports 4.59 at LongPPL, versus 4.71 for dense KV4 and 4.58 for dense KV8, with 1.60 to 1.94 times the peak decode throughput of KV8. The paper also gives a FlexAttention query packing scheme for parallel training.

Critical strengths
  1. The general idea looks good for me.
  2. Statistical reporting is careful. This paper uses paired bootstrap intervals with stated limits and includes a parameter matched wide FFN control.
Critical weaknesses

I think the technical contribution is limited. For the main idea, I think: (1) token level head routing exists in MoA and MoH, and the router and balancing come unchanged from Wang et al. (2024); (2) Polar Sparsity already performs selective KV group access with full histories for batched decoding. For the experimental setup: (1) training from scratch and enlarging GG are also configuration choices, not new mechanisms. (2)The FlexAttention packing applies the existing BlockMask interface. There is no ablation that separates added KV capacity from learned routing.

Another shortcoming is the lack of supporting evidence. (1) Long context quality rests solely on LongPPL over 50 GovReport documents of at most 32K tokens. There is no RULER, needle retrieval, or LongBench evaluation. (2) At 3B, the sparse model does not beat dense KV8 and is significantly worse than dense KV16. At A=2A=2, LongPPL worsens as GG grows. (3) There are no baselines at equal cache memory, such as MLA or KV quantization.

Questions for authors
  1. How does decoding batch tokens that select different groups? Is router cost timed, and at which batch size does each 3B peak occur?
  2. What are the RULER or needle retrieval results at 32K and beyond?
  3. How to explain the LongPPL trend at A=2A=2?
  4. How does KV12/A4 compare with MLA or dense baselines at equal cache memory?
7 comments

Thank you for your thoughtful review. To clarify, the primary contribution of our work is not the exact combination of routing mechanism and load balancing, or the idea of selective head or KV group access.

Instead, our primary contribution is:

  1. the discovery that one of the findings of Bertsch et al (https://allenai.org/papers/olmpool), (that dense model KV capacity is associated with long context model quality, enforcing a tradeoff between long context quality and decode efficiency) can be extended to a relationship between total KV capacity and long context quality while keeping active KV access fixed in a way that enables accelerated decoding that is tied to the active KV size, not total. This allows us to shift the quality/efficiency tradeoff, all while keeping attention compute fixed.
  2. Demonstrating that while the sparse attention models with larger total KV capacity than dense baselines introduce a new trade (better model quality for larger KV cache memory, using more HBM), this trade is worth making: even at short sequence lengths the best achievable sparse model decode efficiency is closely tied to active KV rather than by total KV.

Taken together, these demonstrate that models with head-sparse attention can achieve high quality with low inference cost, replacing a difficult tradeoff decision between quality and efficiency with an easy one.

Responding to the weaknesses outlined in your review:

"(1) token level head routing exists in MoA and MoH, and the router and balancing come unchanged from Wang et al. (2024)"

We agree that token level head routing exists in MoA and MoH, and in other prior work, and we do not claim to have introduced it. MoA does routing over query heads, and forces a shared KV projection to be used for all query experts. The MoH paper describes routing over QKV heads in an MHA setting, but does not discuss routing over KV groups. In their released code, their MoH implementation on the llama model, which uses GQA, does routing over individual query heads regardless of KV head access. This aligns with the focus of the MoH paper, which is on reduced self attention flops relative to dense models with the same number of total query heads, but it does nothing to reduce KV bandwidth unless every query head of a KV group goes unselected. Furthermore, the GQA setting in the MoH paper use a straight through estimator for routing gradients (https://arxiv.org/html/2410.11842v3#S4.SS4), instead of the router weight scaling standard in MoE models that we use in our work. We agree that our implementation uses standard load balancing and routing methods from the MoE literature, as we discuss in the paper. Our goal is to explore the hypothesis that total KV capacity can improve model quality without hurting decode efficiency, not to research novel routing or load balancing mechanisms, so we used the standard methods from the MoE literature, where mature solutions to these problems have already been tested. Exploring these implementation mechanics remains a promising avenue for future work.

"(2) Polar Sparsity already performs selective KV group access with full histories for batched decoding."

We agree, and discuss polar sparsity as the most closely related work in our paper for this reason. However, the method use for polar sparsity 1: are not trained from scratch and require a dense already trained base model, and 2: leads to substantial model quality degradation with sparsity in models using GQA. Even if a model were trained with the intention of using polar sparsity, a model with n:k total:active KV groups and equal query heads per group to the dense baseline would have the training cost of a dense model with n KV groups, while our implementation has the training cost of a dense model with k KV groups. We apply the Polar Sparsity method to the attention layers of our dense 220M models, with the following results:

ModelLongPPLWikiText PPLLAMBADA PPL
Polar KV4/A214.7237.6654.23
Polar KV8/A227.2743.50105.18
Polar KV8/A414.9036.6756.36
Dense KV29.9034.0045.58
Dense KV49.8834.1747.56
Dense KV89.4433.6545.65
Sparse KV8/A29.4233.8145.62
Sparse KV12/A29.6633.7247.70
Sparse KV16/A29.8733.8047.88
Sparse KV8/A49.3333.7346.13
Sparse KV12/A49.0533.4446.63
Sparse KV16/A49.0333.3443.95

We trained the polar sparsity selector modules using data from our 32K sequence length prolong training corpus. We observe that for shorter context evaluations like wikitextppl and lambada ppl, the Polar Sparsity procedure leads to moderate performance degradation, substantial degradation in the sparser KV8A2 case, and in longer context evaluation Polar Sparsity leads to substantial drop in performance.

"For the experimental setup: (1) training from scratch and enlarging G are also configuration choices, not new mechanisms"

We disagree that these are simply configuration choices. The performance degradation of polar sparsity indicate that training from scratch is important as the sparsity increases. Furthermore, without training from scratch the pre training attention compute will scale with the number of total KV groups, rather than the number of active KV groups. Also, enlarging the number of KV groups is a previously unexplored axis of model design. MoA requires a single KV head, MoH fixes kv_heads = model_dim // head_dim in the from scratch experiments, and Polar Sparsity relies on a pretrained model with a fixed number of KV groups. Instead, the central hypothesis of our work depends on the relationship between the number of KV groups and model quality, and exploring that relationship is a large part of the contribution.

"(2)The FlexAttention packing applies the existing BlockMask interface."

We agree. We initially developed a custom kernel to efficiently implement the head sparse attention training on A100 GPUs, but moved to flex attention to enable drop in compatibility on a wide range of hardware, and we see the use of pytorch's flex attention as a positive.

"There is no ablation that separates added KV capacity from learned routing."

We perform an ablation in which the learned routing of the 220M class sparse attention A4 models is replaced by uniform random routing at inference time (keeping router weight gating), and report the results here:

ModelRoutingLongPPLValidation CE
Sparse KV8/A4Learned9.332.3565
Sparse KV8/A4Uniform random17.012.4483
Sparse KV12/A4Learned9.052.3517
Sparse KV12/A4Uniform random26.372.4900
Sparse KV16/A4Learned9.032.3503
Sparse KV16/A4Uniform random30.242.5069

We observe that the random routing moderately degrades short context ppl but seriously harms longppl results. This is strong evidence that KV groups specialize over the course of training, from scratch random routing experiments are still being trained at the time of writing but will be complete shortly.

" (1) Long context quality rests solely on LongPPL over 50 GovReport documents of at most 32K tokens."

We agree. We found that at the compute budget available, the results of task based evals like RULER were very noisy, with accuracy range of 45.73% across 4 seeds for a dense MHA KV8 model config, preventing us from reliably replicating the RULER results of Bertsch et al (https://allenai.org/papers/olmpool) at the 220M parameter scale with a reasonable number of seeds. However, we found that longppl results were much less noisy, aligning with the finding of Madaan et al (https://arxiv.org/abs/2406.10229) that perplexity based evaluations can be more reliable than task/accuracy based evaluations, especially for small training budgets, and decided to keep longppl as the primary long context evaluation metric for the 220M class of models. For the 3B models, we expand our evaluation pipeline to include HELMET, which combines results from multiple long context benchmarks including RULER. We also perform context extension training on the 3B models by training on an additional 4.1 billion tokens at a 64K sequence length. We report the results in the table below:

HELMET average success rate, higher is better

ContextDense KV4Dense KV8Sparse KV12/A4Dense KV16
8K21.1523.7223.9922.57
16K19.0421.7421.1521.01
32K18.2618.2318.6020.36
64K15.2915.5616.4417.24

(Note, standard HELMET evaluations us a GPT 4o LLM judge on a portion of the tasks, due to the increased API cost of using an older model we use GLM 5.3 instead)

These results are consistent with the conclusions of our paper. At long context performance the KV12A4 model consistently outperforms dense KV4 and is comparable to KV8. We will update the manuscript to include this table.

"(2) At 3B, the sparse model does not beat dense KV8 and is significantly worse than dense KV16."

This is accurate, but we do not claim that the sparse model with 12 total heads and 4 active heads should be expected to beat either the dense KV8 or KV16. The sparse model (before context extension) achieves 0.4% higher longPPL than the dense KV8 (within bootstrapped CI, not statistically significant difference), and beats it in the HELMET sequence lengths above, with just half of the decode time KV bandwidth. (post context extension difference is 0.65%, still not statistically significant). We include the KV16 checkpoint as a strong baseline that should be expected to act as an upper bound on long context quality, and we should not expect the other models consistently outperform it. Qualitatively we observe a similar pattern as is seen in MoE models, where an MoE model may underperform dense models with the same number of total parameters but significantly outperfirm dense models with equal active parameters and fewer total parameters. Predicting exactly how a given head-sparse attention configuration should perform relative to dense baselines requires expensive scaling studies and is beyond the scope of this paper.

"At A=2, LongPPL worsens as G grows."

We noticed this trend as well. We do not claim that increasing total KV capacity will monotonically improve performance ceteris paribus. While this result does not contradict the claims of our paper (that long context quality can be improved by having more total KV groups than active), it is a notable artifact. We suspect it is related to the very high sparsity ratio of the A2 G16, A2G12 configs, perhaps combined with the use of load balancing hyperparameters optimized for the the sparse KV8A4 model, and we note that similar behavior has been observed in MoE models in which very sparse models become unstable, leading to lower performance. Isolating the precise cause of this trend remains a promising avenue of future work.

" (3) There are no baselines at equal cache memory, such as MLA or KV quantization."

We have 220M scale models with 8 total KV groups with 4 and 2 active groups, which perform similarly to the dense 220M models with 8 kv groups. This gives a comparison against dense baselines with equal cache memory. Methods like MLA/GLA and KV cache quantization are orthogonal to our work, they should not be seen as competing baselines to compare against, since it is possible to train sparse models using both of these methods. To demonstrate this, we train a three new GLA models at the 220M scale: a pair of dense GLA models with 4 and 8 KV groups respectively, and a sparse model with 8 total GLA groups and 4 active, this model performs routing and load balancing in the same way that the GQA based sparse models do. The results are given here:

ModelLongPPLWikiText PPLLAMBADA PPL
Dense GLA, 4 groups12.1435.4451.60
Dense GLA, 8 groups11.6735.0445.29
Sparse GLA, 8 groups / 4 active11.6034.9050.50

We also include a table of paired bootstrapped confidence intervals for the longPPL difference:

ComparisonΔLongPPL95% CI
Dense GLA8 − Dense GLA4−0.4708[−0.8037, −0.1788]
Sparse GLA8/A4 − Dense GLA4−0.5411[−0.8649, −0.2916]
Sparse GLA8/A4 − Dense GLA8−0.0704[−0.4131, +0.2349]

We find that the capacity-quality relationship found in Bertsch et al (https://allenai.org/papers/olmpool) and our work extends from GQA to the GLA case, and larger KV cache memory is associated with better long context performance. Furthermore, GLA architectures drop in seamlessly to our sparsely routed implementation and we replicate the statistically significant long context gains from sparse models in this case as well. (We note that the GLA models tend to perform moderately worse than equivalent cache sized GQA models from our earlier experiments. For the GLA results we re used the hyperparameters that were optimized for dense GQA, and we suspect this explains the difference)

Reviewer_Eu5dOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Mixture of KV Groups: an attention layer that uses G KV groups, with a small router choosing A of these for each token. Each group provides K and V values for every token (meaning the cache ends up G/A times larger than GQA when A groups are active), but during decoding a token accesses only the A selected groups. The authors trained 220M and 3B hybrid (Gated DeltaNet plus attention) models from scratch, achieved better LongPPL than GQA at the same number of active groups and similar decoding speed, and also present a FlexAttention packing and masking technique that lets the sparse layer train in parallel.

Critical strengths

The idea is simple and the efficiency claim is measured end-to-end: at 3B, sparse KV12/A4 reaches 1.6-1.9x the peak decode throughput of dense KV8 at a similar LongPPL.

Critical weaknesses

1-Novelty seems incremental. It seemss that the innovation lies in applying MoE top-k routing over GQA groups with dense K/V writes, using DeepSeek-V3's auxiliary loss-free balancing. As far as I know, MoH (MoH: Multi-Head Attention as Mixture-of-Head Attention by Jin et al., 2024) already features the same cache semantics (each head's K/V is written, only the selected heads are read), and LLaMA-MoE v2 handles the routing of head groups; what is new in this case is the GQA grouping, the decode benchmark, and the training kernel. Other attention-based MoE or routed-KV work is also missing: SwitchHead (Csordas et al., NeurIPS 2024), mixSGA (Song et al., EMNLP 2025), and MoSA (Piekos et al., TMLR 2025).

2-There is no work on the decode phase. Some designs have reduced decode bandwidth without increasing the size of the cache. MLA (DeepSeek-V2, 2024) and GTA/GLA (Zadouri et al., 2025) are the standard solutions to the problem of 'GQA with few groups leading to a drop in quality', and in these cases the cache is shrunk rather than tripled. If one of these approaches had not been trained under the same budget, we would not be able to determine whether tripling the storage for bandwidth is a good trade-off. There is also no dense KV12 at 3B (with the same cache as the sparse model). According to Tables 2 and 4, dense KV16 performs better than sparse KV12 on LongPPL (the confidence interval excludes zero) and dense KV8 achieves the same result with only two-thirds of the cache, so on a per-stored-byte basis the sparse model is inferior to plain GQA. This point should be mentioned.

3-Quality improvement is not significant, especially at 3B.

4-One seed at 3B. All benchmark differences in Table 2 are within noise, and the sparse model is slightly worse than the cheapest baseline (dense KV4) on LAMBADA (accuracy and PPL), ARC-E, ARC-C and SIQA, while having around 5% more parameters (2.74B vs 2.6B).

5- There is No RULER//HELMET, and nothing beyond 32K for quality results, although throughput is shown up to 512K.

6-The paper says the sparse 3B model "nearly matches" dense KV8. Its own statistics (Table 4) do not support this. The measured gap between the two models is +0.017 LongPPL, and the 95% confidence interval runs from -0.081 to +0.125. That interval is about 0.2 wide, but the entire improvement from dense KV4 to dense KV8 is only 0.134. So the test is too noisy to tell whether the sparse model is as good as KV8, better, or worse. . The abstract also says the method "reliably" improves long-context quality. Table 4 does not back this up. For sparse KV16/A2 versus dense KV2, the interval runs from -0.303 to +0.265 and includes zero, which means "no improvement at all" cannot be ruled out. Worse, with two active groups (A=2), adding more total groups makes LongPPL worse, not better: 9.42 with 8 groups, 9.66 with 12 groups, and 9.87 with 16 groups (Table 1). This is the opposite of the paper's main idea, that storing more KV capacity helps, and the paper does not explain it.

7-It seems the parameter-matched control exists only at 220M and for one configuration; other 220M sparse models carry up to 26% more parameters (268M vs 213M).

8-Memory usage is not reported

9-The sparse decode kernel is not described (how per-sequence group selection is gathered, paged or not, FA4 or custom), and Table 3 is one run per config. The "same active parameters" argument (Sec. 3.1) hides that in batched decode all G groups' Q/O weights are read every step (Efficiently Scaling Transformer Inference", MLSys 2023), which is why A=2 drops to 83-87% TPS.

10-The training throughput figures given refer only to the attention operator and are provided only for the case where A=2 (the configuration mentioned in the headline having A=4). At context lengths of 2K to 8K the throughput is about 45 to 82 per cent of that of the dense case (see Fig. 4b). Since standard training recipes (Llama 3; ProLong, Gao et al., 2025) carry out most of their pretraining at short context lengths, the claim of only a 'modest overhead' is valid only when the context length is 64K. There is no information given regarding full-model training throughput and no data on prefill or TTFT.

Questions for authors

Please check the weakness.

10 comments

Thank you for your detailed review. At a high level, we would like to clarify that the primary contribution of the paper is not the novelty of the specific implementation of head sparse routed attention studied in this work, and we do not claim to have invented head level routing in the attention mechanism. Rather, it is the discovery that previous findings that link KV capacity to long context quality in dense models can be extended to a sparse setting (in which this has not been explored) and that by training models with more total KV groups than active KV groups, we can improve long context quality without increasing KV memory bandwidth. Even better, models can be trained sparsely enough that the KV bandwidth reduction offsets the maximum batch size reduction imposed by having a larger KV cache. We will address the points raised in order below:

"1-Novelty seems incremental. It seemss that the innovation lies in applying MoE top-k routing over GQA groups with dense K/V writes, using DeepSeek-V3's auxiliary loss-free balancing."

We agree that we apply existing methods from the MoE literature in our work, but the specific implementations of load balancing or router are not the primary or even secondary contributions of our work. Please see above.

"As far as I know, MoH (MoH: Multi-Head Attention as Mixture-of-Head Attention by Jin et al., 2024) already features the same cache semantics (each head's K/V is written, only the selected heads are read), and LLaMA-MoE v2 handles the routing of head groups; what is new in this case is the GQA grouping, the decode benchmark, and the training kernel."

This is incorrect. Most importantly, MoA and MoH may densely write to KV cache, but they do not read from it sparsely. MoA routes over query heads only, keeping KV access fixed regardless of the number of total or active query heads, and thus it has no impact on KV cache bandwidth. Similarly, MoH routes over query heads. The query head level routing in MoH induces an implicit routing over KV heads in MHA-like configurations in which each query head is matched to a single KV head, but the query head level routing becomes clear in their application to Llama models which use GQA, in which routing is done over all query heads without regard for KV group, which would only lead to KV cache reduction in cases when all query heads in a given group are coincidentally passed over by the router. Llama-MoE does have KV group level routing, but only writes new tokens to a KV group if that token also was selected to perform attention computation on that head. This formulation allows the Llama-MoE v2 method to be implemented using a python for loop over vanilla flash attention kernel calls with causal masking (https://github.com/OpenSparseLLMs/LLaMA-MoE-v2/blob/main/smoe/models/mixtral/modeling_mixtral.py). We describe in more detail (https://openreview.net/forum?id=z4MHMvdyJb&noteId=DI8UvPkrhE) the technical reasons why the Llama MoE v2 approach is simpler to implement that a true dense write sparse read KV architecture. We maintain that our dense KV write sparse KV read architecture is a useful contribution, although secondary to the contributions outlined above.

"Other attention-based MoE or routed-KV work is also missing: SwitchHead (Csordas et al., NeurIPS 2024), mixSGA (Song et al., EMNLP 2025), and MoSA (Piekos et al., TMLR 2025)."

Thank you pointing these out. SwitchHead performs sparse MoE-style selection of V/O projection weights but maintains dense KV read and write semantics. mixSGA uses a procedure that averages the KV vectors of different heads according to a selector module and explores a different axis for inference efficiency. MoSA performs a sparse attention by selecting tokens from the context to be included in the attention computation, which is a different axis of sparsity and not similar to our MoE style routing work. Still, all of these methods apply some form of token based selection mechanism to some part of the attention computation, so we will include citations for each.

"2-There is no work on the decode phase."

We are unsure what the intended meaning of this is. However, the central result of the paper is that sparse models enable faster decoding with high long context quality, the architecture we use is designed around drop in compatibility with high performance decode attention kernels, and we perform experiments measuring improvements in max decode throughput and decode TPS/sequence.

If this statement is intended to mean that the paper would benefit from more discussion of decode optimizations like MLA/GLA in the related work, we agree, and will include a paragraph on these techniques.

"Some designs have reduced decode bandwidth without increasing the size of the cache. MLA (DeepSeek-V2, 2024) and GTA/GLA (Zadouri et al., 2025) are the standard solutions to the problem of 'GQA with few groups leading to a drop in quality', and in these cases the cache is shrunk rather than tripled."

We agree GLA/MLA have in some cases been found to improve model performance, but are unaware of results showing that MLA/GLA are a solution to quality drops associated with small KV cache in long context settings. Regardless of the relative effectiveness of GLA and GQA, our work is orthogonal to this configuration decision, since GLA can be used as a drop in replacement of GQA in our method. To demonstrate this, we train a set of models from the 220M parameter family: a dense GLA model with 4 groups, a dense GLA model with 8 groups, and a sparse model with 8 total GLA groups of which four are active. The models maintain the same 8 active query heads and the GLA/MLA latent dimension is 256, giving equal KV cache size as the corresponding GQA models. The results are given here:

ModelLongPPLWikiText PPLLAMBADA PPL
Dense GLA, 4 groups12.1435.4451.60
Dense GLA, 8 groups11.6735.0445.29
Sparse GLA, 8 groups / 4 active11.6034.9050.50

We also include a table of paired bootstrapped confidence intervals for the longPPL difference:

ComparisonΔLongPPL95% CI
Dense GLA8 − Dense GLA4−0.4708[−0.8037, −0.1788]
Sparse GLA8/A4 − Dense GLA4−0.5411[−0.8649, −0.2916]
Sparse GLA8/A4 − Dense GLA8−0.0704[−0.4131, +0.2349]

We find that the capacity-quality relationship found in Bertsch et al (https://allenai.org/papers/olmpool) and our work extends from GQA to the GLA case, and larger KV cache memory is associated with better long context performance. Furthermore, GLA architectures drop in seamlessly to our sparsely routed implementation and we replicate the statistically significant long context gains from sparse models in this case as well. (We note that the GLA models tend to perform moderately worse than equivalent cache sized GQA models from our earlier experiments. For the GLA results we re used the hyperparameters that were optimized for dense GQA, and we suspect this explains the difference)

" If one of these approaches had not been trained under the same budget, we would not be able to determine whether tripling the storage for bandwidth is a good trade-off. There is also no dense KV12 at 3B (with the same cache as the sparse model)."

We strongly disagree with this assessment. One of the fundamental results of the paper, which we display prominently in figure 1, is that increasing the storage to reduce bandwidth is a good tradeoff. Total KV cache size of the sparse models is important because it reduces the maximum batch size that can be served during decoding. This reduces the extent to which model weight bandwidth may be amortized over multiple tokens, reducing total throughput. However, our critical finding is that for the models we studied, the KV memory bandwidth reduction is large enough to completely offset this effect, leading to a model with larger KV cache size but higher throughput. It is possible that the total KV batch size could cause issues in the edge case setting of one sequence with extremely long sequence length, in which case the KV cache would not be able to fit into memory. We will add a note of this potential edge case in the appendix.

"According to Tables 2 and 4, dense KV16 performs better than sparse KV12 on LongPPL (the confidence interval excludes zero)"

We agree, and display this result prominently. The KV16 model is intended to serve as a high performing upper bound on long context performance, and there is no expectation that the KV12A4 model should match or outperform it.

"and dense KV8 achieves the same result with only two-thirds of the cache, so on a per-stored-byte basis the sparse model is inferior to plain GQA. This point should be mentioned."

We discuss the fact that the sparse model has a larger total KV cache than dense, that fact is of central importance to one of our headline results, and is even referenced directly in the paper's title. Crucially, despite the larger cache, the sparse model has faster inference because the reduced KV bandwidth yields a large enough speedup to offset the smaller maximum batch size.

"3-Quality improvement is not significant, especially at 3B."

This is incorrect. At 220M scale the sparse configurations show multiple statistically significant long context quality improvements over dense baselines, with multiple seeds. At 3B we also observe statistically significant long context quality improvement over the dense baseline.

"4-One seed at 3B. All benchmark differences in Table 2 are within noise, and the sparse model is slightly worse than the cheapest baseline (dense KV4) on LAMBADA (accuracy and PPL), ARC-E, ARC-C and SIQA, while having around 5% more parameters (2.74B vs 2.6B)."

We disagree. Firstly, we note that using one seed for experiments of this scale is well within the bounds of standard practice in the community, and the expectation of multiple seeds for such expensive experiments in a pre training setting is non-standard. Second, not all results in table 2 are within noise, as we explain above and in the paper, the long context evaluations establish that the sparse KV12A4 model outperforms dense KV4 at a statistically significant level, nearly matches KV8. The other benchmarks are short context general language modeling evaluations that do not show a strong trend WRT KV capacity, which under our hypothesis (and the finding of Bertsch et al) that KV capacity is substantially important to long context performance without showing up on short context evals, is entirely expected.

"5- There is No RULER//HELMET, and nothing beyond 32K for quality results"

Thank you for raising this point, we agree. We found that at the compute budget available, the results of task based evals like RULER were very noisy, with accuracy range of 45.73% across 4 seeds for a dense MHA KV8 model config, preventing us from reliably replicating the RULER results of Bertsch et al (https://allenai.org/papers/olmpool) at the 220M parameter scale with a reasonable number of seeds. However, we found that longppl results were much less noisy, aligning with the finding of Madaan et al (https://arxiv.org/abs/2406.10229) that perplexity based evaluations can be more reliable than task/accuracy based evaluations, especially for small training budgets, and decided to keep longppl as the primary long context evaluation metric for the 220M class of models. For the 3B models, we expand our evaluation pipeline to include HELMET, which combines results from multiple long context benchmarks including RULER. We also perform context extension training on the 3B models by training on an additional 4.1 billion tokens at a 64K sequence length. We report the results in the table below:

HELMET average success rate, higher is better

ContextDense KV4Dense KV8Sparse KV12/A4Dense KV16
8K21.1523.7223.9922.57
16K19.0421.7421.1521.01
32K18.2618.2318.6020.36
64K15.2915.5616.4417.24

(Note, standard HELMET evaluations us a GPT 4o LLM judge on a portion of the tasks, due to the increased API cost of using an older model we use GLM 5.3 instead)

These results are consistent with the conclusions of our paper. We will update the manuscript to include this table.

", although throughput is shown up to 512K."

We do show throughput and decode efficiency results up to length 512K, but we do not claim that our evaluation results extend to that length. It is standard practice in the literature to report inference efficiency results at contexts beyond what is evaluated, we refer to Zadouri et al., which introduced GLA/GTA, in which evaluations are restricted to relatively short context general language modeling but serving efficiency results are reported up to 131K. Also, Gu & Dao, 2023, which introduced the Mamba architecture, showed systems efficiency results up to 512K, but did not have corresponding language evals at those lengths.

Reviewer_zZetOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2(+1)
Read review
Paper summary

This paper proposes Mixture of KV Groups, a method that increases the KV-cache size but groups queries to route to only a subset of the KV-cache. This method is different from Grouped-Query Attention (GQA) in that it gives attention a larger set of KV groups than in GQA and learns to select a small subset for each query token. This way the proposed approach (labeled "Sparse" in the paper) separates the KV-cache size from the number of groups in the cache each query needs to access. The authors implemented this method using FlexAttention and compared it with the dense GQA.

Critical strengths

S1. It is a good idea to separate the cache size from the amount of cache content to access for each query. The proposed approach aims to support fast decoding while maintaining long-context quality.

S2. The authors implemented the proposed approach and empirically compared it with an existing approach.

Critical weaknesses

W1. The implementation and evaluation are missing justifications and explanation. For example, the total number of KV-groups (G) and the number of active KV-groups for each token (A) are the most important parameters for the proposed scheme, but it is unclear how these two parameters are set, and the evaluation just picked values from 2 to 16 for G and 2 to 8 for A. Another example is that the cache size in the proposed approach would be a concern as every KV-group retains the entire history, yet there is no result about the memory consumption.

W2. The paper presentation is hard to follow. There are many places of confusion. To begin with, the abstract and the second paragraph of Introduction talk about "Mixture of KV groups", and then the 4th paragraph of Intro talks about "sparse routing" and "sparse models". It does not become clear that both "sparse" terms refer to the proposed method until one sees the legend key "Sparse (ours)" in Figure 1. Sentences are hard to understand or convey little specific meaning. Examples: Section 3.2. the 1st paragraph. "The central challenge is that groups receive different numbers of queries, while each group retains the complete KV history". I wonder why it is a challenge and it is central. Section 4.2. the last paragraph. "The interval contains zero, so this analysis does not resolve a difference between these checkpoints." What does it mean? Section 4.4. the second paragraph. "Changing the packing, tile sizes, or kernel can change this overhead". Seems common sense.

Questions for authors

Q1. What's the GPU memory capacity of the NVIDIA B200 used in the experiments? What were the memory consumptions of the modles in Table 3? Why did you pick the (G,A) configurations in Table 3, not other combinations?

4 comments

Thank you for your review, and for recognizing the value of seperating cache size from per-token KV access.

We will respond to each element of your review below:

"W1. The implementation and evaluation are missing justifications and explanation. For example, the number of query heads (G) per group, and the number of KV-groups for each token (A) are the most important parameters for the proposed scheme, but it is unclear how these two parameters are set, and the evaluation just picked values from 2 to 16 for G and 2 to 8 for A. "

First, a notation correction: G denotes the number of total KV groups, not the number of query heads per KV group. A denotes the number of active KV groups per token, in dense models (like standard GQA or MHA configurations) G = A. In sparse configurations G > A. In order to have fair comparison we equalize self attention compute for all models of a given scale, this corresponds to equalizing the number of active query heads for all models tested (we use 8 active query heads for the 220M class of models and 16 active query heads for the 3B class). Thus, a model with A active KV groups will have 8/A or 16/A query heads per group, respectively.

The GQA configurations for the dense models were chosen following standard practice, in which the number of query heads equals model_dim // head_dim, while the number of KV heads ranged from an MHA configuration to an aggressive GQA. For the selection of G and A in the sparse models we chose values of A that matched the number of KV groups in the dense baselines, so that we could get equal KV bandwidth comparisons. Our choice to use G \in {8,12,16} allowed us to test active ratios (given by A/G) over a wide range, from 12.5% to 67% active. We will add an explicit description of this reasoning to the manuscript.

"Another example is that the cache size in the proposed approach would be a concern as every KV-group retains the entire history, yet there is no result about the memory consumption."

The memory consumption is given by the number of KV groups. Since the head dimension is always 128, the KV cache size for a sequence is always given by 2128(precision, in this case 2 bytes) * sequence_length * G * num_attention_layers. We will add explicit accounting of each model's KV cache size at each context length to the appendix. Also, the higher memory consumption of sparse models is addressed in the paper and is to one of our most important results: Section 4.3 studies the impact of sparse model's larger KV cache on the system's potential decoding throughput, and we find that although the larger KV cache of sparse models leads to lower maximum batch size, the lower KV bandwidth cost makes up for it, and the sparse models achieve substantially better throughput than dense models of similar long context quality (this is also shown in figure 1B).

"W2. The paper presentation is hard to follow. There are many places of confusion. To begin with, the abstract and the second paragraph of Introduction talk about "Mixture of KV groups", and then the 4th paragraph of Intro talks about "sparse routing" and "sparse models". It does not become clear that both "sparse" terms refer to the proposed method until one sees the legend key "Sparse (ours)" in Figure 1. "

Thank you for calling attention to this point. We agree that this should be edited to reduce confusion, and will explicitly say in the introduction that unless otherwise stated, "sparse" refers to the implementation we study in the paper.

"Sentences are hard to understand or convey little specific meaning. Examples: Section 3.2. the 1st paragraph. "The central challenge is that groups receive different numbers of queries, while each group retains the complete KV history". I wonder why it is a challenge and it is central. "

Thank you for raising this. This sentence occurs in the context of section 3.2, in which we present our GPU efficient parallel training implementation. Different groups receiving different numbers of queries, combined with each group maintaining full KV history, is a central challenge because it requires attention masking logic and ragged shape handling that cannot be implemented in common attention kernels like Flash Attention efficienctly, or with a head-parallel flex attention implementation. We speculate that this challenge is why previous work that sparsely routes over KV heads/groups with full history did not actually use sparsity during training (in the released code, MoH computed the outputs of all heads and then applied a zero mask: https://github.com/SkyworkAI/MoH/blob/main/MoH-DiT/models.py#L75-L124). The challenge of executing sparse end to end training in a GPU friendly (no python loops) way is why we present our flex attention based implementation as an important secondary contribution. However, we agree that this should be made more clear in the paper, and we will edit the body of the paper to clarify this, as well as add a section to the appendix with a more in depth analysis of the issue, previous methods, and our solution.

"Section 4.2. the last paragraph. "The interval contains zero, so this analysis does not resolve a difference between these checkpoints." What does it mean? "

We agree that this is unclear. The sentence means that the confidence interval for eval performance difference between the two checkpoints contains zero, so the observed difference is not statistically significant at the 95% confidence level. We will rephrase this sentence to state that conclusion precisely.

" Section 4.4. the second paragraph. "Changing the packing, tile sizes, or kernel can change this overhead". Seems common sense."

We agree. This sentence provides no useful information, so we will delete it.

Response to reviewer Questions:

"Q1. What's the GPU memory capacity of the NVIDIA B200 used in the experiments? What were the memory consumptions of the modles in Table 3? Why did you pick the (G,A) configurations in Table 3, not other combinations?"

The GPU memory capacity of the B200 used in these experiments is 192GB. We will add this and other details on hardware to the appendix. The KV cache memory of the models in table 2 depend on the number of total KV groups in the model. The dense KV2 model has a KV cache of 13.92 GB, the KV4 has 27.09GB. In general, models with n KV groups have KV cache memory equal to (n/2) * 13.92GB. We will update the appendix with a KV cache memory table for each model/sequence length/batch size evaluated.

If there are specific soundness concerns behind the soundness score, we would be glad to address them.

thanks for your clarification. I have corrected my reference to the notations of G and V.

Thank you for your speedy reply. As we said, if there are specific soundness concerns behind the soundness score please let us know, we would be glad to address them or any other concerns.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We will address some common reviewer points here, we discuss each review in greater depth in the review comments.

Questions of Novelty

Two reviewers raised the question of novelty, primarily MoH, MoA, and Polar Sparsity. However, both MoA and MoH, perform routing only over query heads, neither has the dense KV write, I/O aware KV read cache semantics which are central to our work. Polar sparsity uses similar dense write sparse read cache semantics, and we discuss it in the paper as the most relevant prior work, but it is not trainable end to end, requiring a pre trained dense model checkpoint. We also observe a long context performance decline from highly sparse applications of polar sparsity to our dense checkpoints (detailed here: https://openreview.net/forum?id=z4MHMvdyJb&noteId=PTLWk1OECb).

Furthermore, the primary result of our work extends beyond the methodological novelty of the routing mechanism, we demonstrate that the long context quality improvements achieved by scaling KV capacity in dense models can be extended to our sparse models, allowing us to improve long context performance without increasing KV memory bandwidth. We also show that while the larger total KV cache memory lowers the maximum inference batch size, the KV bandwidth gains of our method are enough to offset this effect, maintaining high inference throughput. Together these results make a strong case that sparsely routed KV groups are an effective solution to maintain high throughput/low latency inference without sacrificing model quality.

Comparison to KV cache reduction techniques

Two reviewers raised a lack of comparison against prior methods like GLA (grouped latent attention, variant of MLA) and KV quantization, which are also used to improve long context inference performance. We stress that these techniques are orthogonal to our work and should not be seen as "competing" methods. In fact, both of these techniques are compatible with our method without alteration. GLA models maintain the same KV group structure of GQA, but use a latent combined vector to compress both K and V. We can still route to GLA groups as we do with GQA in our paper, and we demonstrate this with a set of GLA experiments at the 220M scale, with the statistically significant long context improvement from the sparse model, aligning with the GQA results from our submission (more discussion here: https://openreview.net/forum?id=z4MHMvdyJb&noteId=hSiqDb0azb). Similarly, KV cache quantization can be applied to the sparsely accessed KV caches used in our method without alteration, we perform and experiment demonstrating this with the 3B model checkpoints and again replicate the statistically significant long context quality and high decode performance discussed in the paper (more discussion here: https://openreview.net/forum?id=z4MHMvdyJb&noteId=dTTRGUDmpl)

LongPPL evaluation

Two reviewers pointed out that the bulk of our long context quality results came from LongPPL evaluations. While the results were statistically significant, we agree that the conclusions would be strengthened by results on task based benchmarks at a wider range of context lengths. We retain LongPPL as the metric used for evaluating the 220M due to the documented noise issues associated with task based evaluations on small models that haven't undergone post training. However, for the 3B class of models we perform context extension training on 4.1B tokens at a 64K sequence length, and then record HELMET results at 8K, 16K, 32K, and 64K sequence lengths. We use HELMET because it aggregates the results of multiple long context benchmarks into one score, giving a measure of long context performance across varying tasks. We report and discuss the HELMET results in this comment: https://openreview.net/forum?id=z4MHMvdyJb&noteId=6egw5vYes5, finding that they again are consistent with the LongPPL findings that our method improves long context quality without increasing KV bandwidth.

Clarity

Reviewers also raised concern about the clarity of some portions of the text, we largely agree with their assessments and have revised the relevant text.

AuthorsOpenReview ↗

Note on Routed Query Capacity

The sparse models studied in our paper generally have larger total parameter counts than the corresponding dense models. We trained a parameter-matched dense control model by increasing the size of the FFN layers, and found that the resulting model still did not match the long context performance of the sparse models. This is evidence that the sparse model improvement is not due to extra parameters. However, it does not isolate larger KV capacity as the cause of sparse model outperformance, since the sparse models also have a larger bank of query projections, giving them large routed query capacity that the dense parameter matched control didn't. To resolve this, we include a GQE (https://arxiv.org/abs/2606.20945) control at the 220M parameter scale. The GQE control has 32 total KV heads, of which two are selected from each KV group, for 8 active query heads per token. This control has equal sparse and active query capacity as our KV16A4 220M sparse model, but lacks the added KV capacity. Results on longppl and the associated CIs for paired differences are below:

ModelLongPPL ↓
Dense KV49.884
Dense KV4, wide FFN9.509
Sparse KV8/A49.328
Sparse KV12/A49.045
Sparse KV16/A49.034
GQE Q32/KV4, 8 active queries9.413
ComparisonΔLongPPL95% CI
GQE Q32/KV4 − Dense KV4−0.471[−0.788, −0.230]
GQE Q32/KV4 − Dense KV4, wide FFN−0.096[−0.342, +0.139]
GQE Q32/KV4 − Sparse KV8/A4+0.085[−0.148, +0.339]
GQE Q32/KV4 − Sparse KV12/A4+0.367[+0.211, +0.589]
GQE Q32/KV4 − Sparse KV16/A4+0.379[+0.212, +0.615]

We find that the GQE model performs better than standard dense, but is still significantly worse than the KV16A4 and KV12A4 models that have equal or less query capacity and more KV.

While reviewers did not request this control explicitly, we feel it it is relevant and will include it in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.