acceptodds
Under review as a conference paper at ICLR 2027

Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention

Abstract

Self-attention is central to Transformer performance and is often the most expensive part of the Transformer at long context lengths because its pairwise token interactions scale quadratically with sequence length. Standard dense attention also applies the same set of attention heads to every token regardless of token difficulty or information content. This uniform activation can waste compute, especially as sequences grow longer and attention cost increases rapidly. We propose Grouped Query Experts (GQE), a mixture-of-experts layer on top of grouped-query attention (GQA). Within each GQA group, a router selects k query-head experts per token while all key-value (KV) heads remain dense and unchanged. Thus, GQE keeps the KV cache benefits of GQA and reduces only the active query-head computation. On a fixed 30B token budget at the 250M parameter scale, GQE matches the all-active GQA baseline in downstream accuracy while activating half the query heads per token.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Grouped-Query Attention, Mixture-of-Experts, Sparse Attention, Attention Efficiency, Efficient Transformers

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kcpyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce Grouped Query Experts, a variant of Group Query Attention where each key-value group uses a router to select a set of query heads to use for attention computation, thus reducing the memory costs of the attention computation.

Critical strengths

The proposal is novel and somewhat interesting; there has been suggestions that not all heads are important for predictions thus by constructing an approach to potentially learn to use the correct heads could be a very sensible approach to reduce computation. The motivation is also clear, which is to reduce the amount of computation necessary for attention.

Critical weaknesses

Major

  • The presentation of standard attention wasting compute is somewhat contestible, as it's still rather established that this can still achieve accuracy relative to GQA or MQA; while the bottleneck of compute becomes an issue in this setting, framing this as wasting compute is a rather dubious claim and it should be framed rather as the trade-off between precision/quality as opposed to hardware efficiency.
  • The compared baselines is limited; the authors only compare against GQA as well as two concatenation variants of the introduced GEQ approach. Furthermore, there is information that is very unclear that compromises understanding of the results. The authors state the setup uses 8 heads and 8 groups for GQA while it is 16 and 8 for GEQ; the actual effect of the number of attention heads can be rather significant at the small model scale as well as because 8 heads is already rather small. Given that the method is an attention variant, it would be extremely relevant to compare with standard multi-headed attention as well as multi-query attention (or even multi-latent attention).
  • Beyond this, the use of 16/8 with k=1k=1 means that the model is essentially choosing one of two possible query heads at any choice; thus what would be an extremely critical baseline would be to actually just randomly select between the two to verify if any gains are actually coming from the router..
  • Evaluation is very limited; while MMLU, PiQA and Arc-E are standard evaluation tasks, they are all rather limited in what is being measured. Beyond just classification tasks, there can be evaluation on other settings such as more retrieval based tasks (ex. question answer or summarization), as well as possibly more difficult benchmarks which one can verify has not been leaked in the training dataset. Actual improvement over GQA at the 250M parameter scale is also limited (only 0.18 percentage points on average) and there is no standard deviation, thus there is the possibility that over multiple seeds, the actual performance gain is insignificant.
  • The throughput analysis doesn't show particularly signficant gains until the sequence lengths are extremely long, for example it remains less than 2 times faster for pre-filling throughput even at 1024K tokens. Furthermore, the authors should benchmark generation speed for a better understanding.

Minor

  • There is some redundancy in presentation; for example Figure 2 and 3 give the same detail from what I can observe and thus should be merged.
  • Section 3.5 appears unnecessary; the computational savings relative to GQA are already fairly evident (use kk query heads per group instead of the heads/group ratio). The section itself is fairly hand-wavy in its explanations so this should either be explained in further detail or removed; additionally, this doesn't contribute signficantly to the paper's approach so a more fitting place would be the appendix.
Questions for authors
  • Some details are confusing;
  1. What is the "shared attention head" (L193) and how is this selected relative to the number of heads or groups?
  2. L198 to L200 suggest that the attention softmax probabilites are computed prior to router selection; this implies that the actuall attention is computed over all query heads prior to the router computation, which actually would waste the memory savings of the approach.
  • I would also suggest ablations on the actual selection strategy; using Gumbel-Max or Gumbel-Softmax as this might resolve the issue of sum re-weighting in a more natural manner by using a straight-through estimator.
  • There should be an actual analysis of what happens with the selection of the query heads; if the task is improperly defined, there is the possiblity that the actual result is the same query head always being chosen within the same group and the setup devolves into multi-head attention except with the number of groups being the practical number of total heads. This would be a relevant comparison to verify if there is actually any gain or benefit from the GEQ setup rather than simply selecting fewer heads.
  • What happens when the number of query heads that are associated with each KV group varies; furthermore what happens then the number of activatied query heads varies? There should be signficant ablation studies since this is likely highly sensitive to the choice of hyperparameters for the heads as well as groups.
Reviewer_kjzeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Grouped Query Experts (GQE), which selects query-head experts independently within each GQA group while preserving dense key and value computation. Its output combines hard-selected expert outputs, a normalized weighted-sum slot that supplies router gradients, and an always-active shared attention head. At approximately 250M parameters and 30B training tokens, GQE obtains similar average downstream accuracy to the dense GQA baseline with nine active attention computations instead of sixteen.

Critical strengths
  1. The design addresses a useful problem: reducing attention computation while retaining GQA’s compact KV representation. Requiring every KV group to contribute selected queries is a clear architectural constraint.
  2. The distinction between eight routed attention computations, one shared attention computation, and ten output-projection slots is explicitly explained.
Critical weaknesses
  1. The novelty of this paper is somewhat limited. Token-dependent head selection already appears in MoA, and always-active shared heads appear in MoH. The potential contribution is the specific combination of within-GQA-group routing and output construction. Direct comparisons are needed to establish its value. MoA[1], MoH[2].

  2. The empirical evaluation is limited to a single approximately 250M-parameter model trained on 30B tokens and only three downstream benchmarks: HellaSwag, ARC-Easy, and PIQA. Broader evaluation across additional datasets and tasks is needed to establish whether GQE consistently preserves model quality. In particular, despite emphasizing long-context efficiency and reporting prefill speedups up to 1024k tokens, the paper provides no corresponding long-context quality evaluation, while training uses sequences of only 2048 tokens.

[1] https://aclanthology.org/2022.emnlp-main.278 [2] https://arxiv.org/html/2410.11842v1

Questions for authors

See weaknesses.

Reviewer_zWpXOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies compute reduction in Transformer self-attention at long context lengths. It argues that dense attention spends the same query-head compute on every token even though tokens differ in information content, and that the mixture-of-experts (MoE) idea has so far been applied mostly to MLP blocks rather than to attention. To address this, the authors propose Grouped Query Experts (GQE), which treats the query heads of each GQA group as experts: a per-group router selects the top-k query-head experts per token while all KV heads remain dense and the KV cache stays identical to the underlying GQA model; the layer output concatenates the hard-selected expert slots, one renormalized router-weighted slot that gives the router a differentiable learning signal, and one always-on shared head, with a Switch-style load-balancing loss. Models of about 250M parameters are trained from scratch on a fixed 30B-token FineWeb-Edu budget and compared against an all-active GQA baseline on HellaSwag, ARC-Easy and PIQA, together with prefill latency from 2k to 1024k tokens. The main results are that GQE matches the baseline average (56.04 vs 55.86) while activating 8 of 16 routed experts (9 of 16 query-attention computations including the shared head), and that measured prefill speedup grows from 1.15x at 2k tokens to roughly 1.7-1.8x at long context. The authors conclude that sparse query-head routing preserves quality only when the router receives a proper learning signal and is anchored by a shared head. The contribution is primarily conceptual and small-scale empirical.

However, the supplementary material shipped with Submission 10670 (supplementary_material zip, attachment on forum MBFD6iYt4Z) identifies the authors in at least three separate files. This is not a single stray token; it is the README's central claim, the package metadata, and the code docstring.

Critical strengths

Strength 1: The paper isolates a transferable, falsifiable empirical message about head-level MoE: hard top-k routing over query heads degrades below the dense GQA baseline, and parity is recovered only when the router gets a differentiable (renormalized weighted-sum) path and an always-on shared head anchors the layer. Evidence: Table 2 ablation ladder at 16 query heads / 8 KV heads, k=1, 30B tokens: 55.18 (weighted concat without renormalized slot), 55.43 (hard concat), 56.04 (GQE) vs 55.86 (dense GQA); stated as the central claim in Sec 1 ("Sparse query-head routing matches the dense GQA baseline only when...") and Sec 4.2. Why it is consequential: This is the part of the paper that would change how a practitioner builds an attention-MoE layer; it is a mechanism-level finding rather than a benchmark score, and it is the main reason the submission deserves review rather than dismissal.

Strength 2: The design is unusually self-consistent and honest about what it changes and what it does not: routing is confined to fixed GQA groups so per-token compute and the W_O shape are constant, the KV cache is provably identical to the underlying GQA model, and the paper corrects its own "50%" headline to 9/16 and concedes the comparison is not exactly parameter-matched. Evidence: Sec 3.2-3.5, in particular Eq. 7 and the sentence "it is not exactly parameter-matched in the attention output projection" (Sec 3.3); the supplementary implementation (gqe/gqe.py) matches the described layout, including the constant 10-slot output. Why it is consequential: Constant-shape, KV-preserving conditional computation is exactly what makes such a layer droppable into existing GQA stacks, and the explicit compute accounting makes the paper's claims auditable in principle - a rarity in this subfield, where speedups are often reported without a compute model.

Critical weaknesses

Weakness 1: The headline throughput claim (1.7-1.8x prefill speedup, abstract and Figure 1) is not auditable and is arithmetically inconsistent with the paper's own compute profile. Evidence: Figure 1 plots about 1.35x at 4k and roughly 1.67-1.80x from 8k/16k up to 1024k, but the paper and supplement state no hardware, no attention kernel/backend, no batch size, no absolute latencies, no repeats or variance, and never say whether the timing is end-to-end prefill or the attention operator alone. Under Sec 3.5's own accounting only the score/AV work of 7 of 16 query heads is removed (MLP and KV projections unchanged; the released gqe.py additionally computes the full 16-expert Q projection before gathering), so the asymptotic end-to-end upper bound is 16/9 = 1.78x, and reaching 1.67x requires query-side attention to consume more than 91% of prefill time - which for a 250M dense-MLP Transformer occurs only around 64k-100k tokens, not at the 4k where the paper says the long-context regime begins. An independent Amdahl recomputation over three plausible 250M configurations (intermediate/speedup_upper_bound.py) puts the end-to-end bound at about 1.17-1.24x at 4k and 1.77x at 1M tokens. Why it is consequential: Throughput is one of the two headline claims (abstract, contributions, and Sec 4.3's "throughput is a primary result"). As stated it overstates what the method can deliver, cannot be checked by any reader, and carries the paper's practical motivation. What would resolve it: State exactly what Figure 1 times; give absolute ms per point, GPU model, attention kernel/backend on both sides, batch size, warmup and repeat count with variance; and add a static baseline with the same 9-10 active query heads and the same 10-slot W_O but no router/top-k/gather, to show the gain comes from routing rather than from simply computing fewer heads.

Weakness 2: The empirical case rests on a single seed, three easy benchmarks, one configuration, and a confounded ablation, at a scale where the reported differences sit inside ordinary run-to-run noise. Evidence: Table 2 differences are +0.18 / -0.68 / -0.43 average points with no seeds or variance (Sec 5 concedes results "should be confirmed with multiple seeds"); Figs 4-7 show four fully overlapping training-loss and accuracy curves with no visible separation; the released code toggles the renormalized slot and the shared head together (variant="gqe" sets both flags), so the ladder cannot attribute the recovery to either component; there is no static-pruning or parameter/FLOPs-matched baseline, and Sec 3.3 concedes the comparison is not exactly parameter-matched. The paper also never analyzes what the router learns (expert usage vs token type, router entropy), so the motivating hypothesis that tokens differ in head need is never tested. Why it is consequential: Both substantive contributions - the parity claim and the "naive routing fails, our two fixes recover it" causal story - depend entirely on these numbers; without variance or a de-confounded ablation they remain suggestive rather than established. What would resolve it: Three or more seeds with standard deviations for Table 2 (or paired per-benchmark significance tests), an ablation that toggles the renormalized slot and the shared head independently, one static 9/10-head GQA baseline at matched parameters, and a short router-behavior analysis (e.g., expert usage by token category) to support the motivation.

Questions for authors

(Re: W1) [Clarification] Does Figure 1 time the full model prefill (end-to-end wall-clock) or only the attention module? Please provide absolute latencies in ms per sequence length, the GPU model, the attention kernel/backend used on both sides, the batch size, and the number of warmup runs and repeats. (Re: W1) [Potential correction] Sec 3.5 implies an asymptotic upper bound of 16/9 = 1.78x on the reduced term, and the released gqe.py computes the full 16-expert Q projection before gathering. Under that accounting, how are the ~1.80x point at 1024k and the ~1.35x point at 4k obtained? (Re: W1) [Clarification] Was a static GQA baseline with 9-10 always-active query heads and the same 10-slot W_O measured? If so, what are its latency and downstream accuracy relative to GQE? (Re: W2) [Clarification] How many seeds back Table 2 and Figures 4-7, and what is the run-to-run spread on HellaSwag, ARC-Easy and PIQA at this scale? (Re: W2) [Clarification] Can the renormalized weighted-sum slot and the always-on shared head be toggled independently (the released variant="gqe" enables both)? What does each contribute on its own? (Re: W2) [Clarification] In the released code the shared head attends to the mean of all group keys and values (key.mean(dim=2), value.mean(dim=2)); the paper does not describe this. Is that the shared head used in the reported results, and does it matter relative to a shared head with its own K/V projections?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.