acceptodds
Under review as a conference paper at ICLR 2027

Can we improve the performance of sparse attention in Grouped Query Experts without increasing experts?

Abstract

Sparse attention architectures, such as Grouped Query Experts (GQE), significantly reduce inference costs by routing input tokens to a small, statically fixed subset of query experts. However, rigidly discarding the unselected experts inevitably leads to information loss and degraded downstream performance. To address this, we introduce HyperGQE, a dynamic attention mechanism that recovers lost representational capacity by generating an auxiliary ”HyperExpert.” Rather than blindly adapting selected weights, HyperGQE computes a selection embedding based on the average representation of the unselected experts, which conditions a hypernetwork to generate dynamic bottleneck matrices (D and U). This HyperExpert processes the hidden states into a supplementary query projection (qhyper) that is integrated into a weighted attention slot alongside the standard routed experts. We evaluate HyperGQE on the SST-2 benchmark, observing that while standard GQE degrades accuracy to 75.92% compared to the dense GQA baseline (80.16%), HyperGQE not only fully recovers this gap but achieves 80.85% accuracy, exceeding the dense GQA baseline by 0.69 percentage points. Furthermore, hardware profiling demonstrates that HyperGQE achieves this superior accuracy using fewer active parameters (924.1M) than the dense baseline (946.0M), while maintaining competitive throughput and VRAM footprints across batch sizes. These results demonstrate that dynamically compensating for unselected experts allows sparse models to exceed the performance of dense counterparts without increasing the active parameter count beyond the dense baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Sparse Attention, Grouped Query Attention, Hypernetworks, Efficient Language Models, Grouped Query Experts

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6ik1OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper summary: The paper proposes to improve Grouped Query Experts (GQE) by adapting an approach inspired by HyperMoE: similar to what happens in HyperMoE where a dynamic expert is computed to “replace” the experts that were left out, here a dynamic query head is computed to compensate for the query heads that were not selected. The dynamic query head per token is created from a hypernetwork that takes as input the average embedding over all experts that were not chosen.

Critical strengths

Positive points:

  • The transfer of the HyperMoE idea to the GQE setting is reasonable and tries to address an actual drop in performance that occurs when transitioning from Grouped Query Attention (GQA) to GQE caused by selecting the top-K query experts.
Critical weaknesses

Negative points

  • The motivation of the approach was to avoid the drop in performance when switching from GQA to GQE. However, there is only a single comparison with GQA in table 1. Furthermore, no comparison with GQA exists in table 2 for the 16 zero-shot tasks. Moreover, it is not clear how statistically significant the 0.69 improvement reported in table 1 is. More generally, the experimental validation is quite limited (a single fine-tuning dataset SST-2 is used and a single model). Therefore, the claim “Exceeding Dense Baselines” (mentioned as one of the contributions) is not sufficiently validated.
  • HyperGQE introduces additional active parameters compared to GQE. Therefore, it is unclear if the improvement in performance compared to GQE comes from this or the dynamic expert mechanism. Comparisons with (i) a parameter-matched GQE and (ii) a GQE with an additional static adapter of the same type are needed.
  • The zero-shot scores reported in table 2 are very low (ie., close to chance level), so it is not clear how meaningful the gains are
  • The overview figure does not help at all in understanding what the proposed approach does (e.g. what do the static layers and dynamic layers correspond to ?)
Questions for authors

The authors should address the issues raised in the critical weaknesses section.

Reviewer_HCdgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a feasible approach to compensating for information loss caused by unselected experts in Grouped Query Experts (GQE). The key idea is to dynamically generate a lightweight HyperExpert conditioned on an aggregation of unselected experts for each input token. This approach is strongly inspired by prior work, HyperMoE, which similarly employs a hypernetwork to generate a HyperExpert to compensate for information loss caused by unselected experts, but in FFN layers rather than attention layers. The proposed method, HyperGQE, extends this idea to GQE by generating an additional query projection from the aggregated information of unselected query experts. The generated HyperExpert is then used to compute an additional attention output, which is combined with the outputs of the selected experts. The experimental results suggest that HyperGQE can improve predictive performance over the original GQE while retaining its sparse attention mechanism.

Critical strengths
  1. The paper explores the application of hypernetwork-based information compensation, previously studied in FFN-based MoE architectures, to the GQE setting.
  2. The proposed method is relatively simple and demonstrates empirical improvements over the original GQE.
Critical weaknesses
  1. This work suffers from a severe lack of technical novelty.

The key idea, that is, using a hypernetwork to dynamically generate an additional expert based on information aggregated from unselected experts, has already been proposed and studied in prior work, HyperMoE. The proposed method is essentially another application of the same idea to GQE, rather than FFN-based MoE. While the target architecture differs, the underlying motivation and technical approach remain largely unchanged.

  1. Although the experimental results show improvements in accuracy, these gains come at the cost of additional computational overhead.

For example, the reported latency increases by 670 ms alongside a 20% improvement in accuracy. This raises concerns about whether the performance gains justify the additional computational cost, particularly given that efficiency is a primary motivation for GQE. A more comprehensive analysis of the accuracy–efficiency trade-off is needed to establish the practical benefits of the proposed method.

  1. The presentation quality needs substantial improvement.

The paper contains considerable redundancy, with multiple sections repeatedly conveying essentially the same ideas, even when the wording differs. In addition, the figures are too small to read comfortably, and their captions are too brief to adequately explain the contents. These issues unnecessarily reduce the clarity and readability of the paper.

Questions for authors

How does HyperGQE compare with vanilla GQE under a matched computational or latency budget?

Reviewer_Ln8jOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

HyperGQE adapts HyperMoE to Grouped Query Experts (GQE). For each token, it averages learnable embeddings of the unselected query experts. A hypernetwork then uses this average to generate a low-rank "HyperExpert" query, whose attention output is added to the routed GQE output. On TinyLlama‑1.1B with LoRA fine-tuning, it reports SST‑2 accuracy of 80.85%, versus 80.16% for dense GQA and 75.92% for GQE. It uses fewer active parameters than GQA (924M vs. 946M) and has 13.5% lower latency than GQA at batch 16, sequence length 1024.

Critical strengths

The question (recovering capacity lost to sparse query-head routing) is reasonable, and the mechanism is simple and cheap. Since it is largely a direct port of HyperMoE, though, the paper's value depends on the evidence, which is weak.

Critical weaknesses
  • The evidence doesn't support the main claim. The headline result is a single SST‑2 number. The +0.69 pp gain is about six examples on the 872-example validation set, within one standard error (1.35 pp), and no seed variance is reported. The baselines (80%) are well below the >90% usually seen for LoRA-tuned 1B models. HyperGQE also adds ~49M parameters over GQE with no parameter-matched baseline. Finally, the mean embedding of the unselected experts is just an affine function of the selected set, so "conditioning on unselected experts" is not a real distinction. Nothing ablates it.
  • The efficiency claims are unreliable. Latencies are described as "analytically estimated." The paper reports both +1.3% and −13.5% latency relative to the baseline. A 34% speedup for GQE from removing only 7.5% of parameters is implausible. The "25% FLOP constraint" is never defined.
Questions for authors
  • What is the mean ± std over the three seeds on SST‑2, and why is the dense baseline only at 80%?
  • How do the results compare with parameter-matched baselines (GQE with a larger k, or a static bottleneck query) and with alternative conditioning?
  • Were latencies measured or estimated, and how are the conflicting latency numbers reconciled?
  • More tasks and larger models to make your claim solid.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.