acceptodds
Withdrawn from ICLR 2027

GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Decoding efficiency depends on KV-cache traffic, computation, and parallelism. Multi-head Latent Attention (MLA) offers absorbed and head-expanded forms, but its head-specific projections limit grouped KV sharing. We introduce Group-Query Latent Attention (GQLA), whose group-shared up-projections preserve a joint latent and enable a shardable expanded cache. The same weights support equivalent MQA-absorb and GQA execution using existing kernels. Roofline analysis relates path preference to GPU compute-to-bandwidth ratio and tensor parallelism (TP): absorption trades traffic for computation and replicates the latent across ranks; GQA shards KV groups. Pretraining with eight query heads per group yields terminal training losses close to MLA's across 6–24 layers at matched tokens. TransGQLA converts GQA and MLA checkpoints using calibration data. Without gradient updates, converted DeepSeek-V3.1-Base achieves eight-task accuracy versus the source's , while WikiText-2 perplexity rises from to . On eight H20-3e GPUs, its GQA path improves output throughput over MQA-absorb by – at matched resident batches across 8K–64K inputs. Its smaller per-rank cache permits larger batches, giving in a separate capacity-tuned comparison. These results support multiple execution paths as an architectural means of adapting decoding to hardware and parallelism.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Efficient LLM Inference, Attention Mechanisms, Grouped-Query Attention, Multi-head Latent Attention, KV Cache, Hardware-Aware Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w8SqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed GQLA, which is a latent attention mechanism. Unlike the original MLA, in which the latent KV expands into 256 KV-heads and matches the query heads (equivalent of MHA), GQLA expands the latent cache into a group of KV heads, which then attends to a greated number of query heads (similar to GQA). Moreover, the paper presents TransGQLA, which allows converting the existing models into GQLA without any need for gradient updates.

Critical strengths
  • The core idea is clear and intuitive. The proposed mechanism seems to be a natural and promising update over the existing MLA architectures.

  • The paper did not only present a new attention mechanism, but also introduces a framework to convert the existing MLA layers into the proposed GQLA, which increases the practicality of the proposed method.

Critical weaknesses
  • It's difficult to understand the claims, position the paper in the literature, interpret the results etc. Some figures are just put in the paper and never referenced in text (e.g., Figure 2). The writing quality of the manuscript is not up to ICLR's standards.

  • Method lacks experimental evaluation and comparison against important prior work:

  1. Grouped Latent Attention (GLA) is cited but never compared against.
  2. TPLA, Tensor-Parallel Latent Attention (Tang et al., 2025): not cited at all
  3. MLRA, Multi-Head Low-Rank Attention (Liu et al., ICLR 2026): not cited at all Without quantitative comparison against these methods, there is no motivation why any one would choose GQLA over any of these existing methods.
  • The kernel evaluations are between two different existing kernels; therefore, the differences are possibly originating from implementation differences. Therefore, these results do not provide convincing evidence about performance gains.
Questions for authors

See the weaknesses above.

1 comment

W1. Regarding the writing and presentation.

A1. In this paper, we first propose a novel model architecture, GQLA, which achieves a favorable balance on both compute-rich hardware, such as the H100, and compute-constrained hardware, such as the H20. To avoid the substantial cost of training a GQLA model from scratch, we further propose TransGQLA, which builds upon TransMLA by introducing two key improvements: grouping and NLL PCA. These improvements significantly reduce the performance degradation caused by training-free conversion.

The illustration in Figure 2 is inspired by the classical MHA and MQA representations of MLA, with the key differences between GQLA and MLA highlighted in orange and blue. Figure 2, together with the subsequent Section 3.1 of the Method section, explains the two equivalent forms of GQLA illustrated in the figure. To improve clarity and strengthen the connection between the figure and the main text, we will explicitly add references such as "As shown in Figure 2" in the revised manuscript.

W2. Comparison against GLA, TPLA, and MLRA.

A2. The key distinction between GQLA and GLA, TPLA, and MLRA is that GQLA is the first architecture capable of equivalently switching between GQA-style and MQA-style decoding, enabling it to adapt to hardware platforms with fundamentally different computational characteristics, such as the H100 and H20.

Specifically, GLA, TPLA, and MLRA all retain the MHA-style prefill structure of MLA and partition the latent representation into groups, supporting only a single decoding mode. In contrast, GQLA preserves the full latent dimensionality and replaces MHA heads with GQA heads to improve parallelism.

While GLA, TPLA, and MLRA enhance parallelism by partitioning the shared latent representation, potentially sacrificing the expressive capacity of the original high-dimensional latent space, GQLA improves parallelism without reducing the dimensionality of the shared latent representation. More importantly, its ability to switch equivalently between GQA-style and MQA-style decoding allows GQLA to achieve efficient inference across hardware platforms with contrasting computational bottlenecks.

W3. The differences are possibly originating from implementation differences.

A3. We would like to clarify that both execution paths use well-established, highly optimized kernels: FA3-MLA (FlashAttention-3) for MQA-absorb and Tencent HPC-Ops for GQA, rather than custom implementations.

Reviewer_6bBSOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Group-Query Latent Attention (GQLA), which uses group-shared key/value up-projections from a joint latent representation to support two mathematically equivalent execution paths for a fixed model: MQA-absorb and GQA. The authors use Roofline analysis to explain path preferences across hardware and tensor-parallel configurations, and introduce TransGQLA to convert existing GQA and MLA checkpoints. The evaluation includes pretraining from scratch, checkpoint conversion and recovery, attention-kernel benchmarks, and full-model serving. On eight H20-3e GPUs, the GQA path improves output throughput by 22.42–34.97% over MQA-absorb at matched batch sizes for the same converted DeepSeek-V3.1-Base checkpoint.

Critical strengths
  1. Two execution paths from the same model weights provide useful deployment flexibility. GQLA retains a joint latent while using group-shared up-projections to enable a shardable GQA cache. This allows a fixed model to select an execution path according to hardware and parallelism. The algebraic relationship between the two paths is clearly described, and the design offers a concrete deployment capability.

  1. The paper provides full-model serving measurements under matched workloads. Table 4 compares the two paths using the same converted DeepSeek-V3.1-Base checkpoint, with matched resident batches and client workloads. The reported 22.42–34.97% throughput improvements across 8K–64K inputs include model computation, communication, and scheduling costs beyond the attention kernel. This provides stronger evidence of practical value than kernel benchmarks alone. The authors also distinguish matched-workload gains from the separate capacity-tuned comparison.
Critical weaknesses
  1. The contribution is not sufficiently distinguished from prior work, and defensive presentation obscures the main insights. Grouped Latent Attention (GLA; Zadouri et al., 2025) already studies grouped latent representations and their cache-sharding advantages under tensor parallelism. GQLA's joint latent and group-shared up-projections are structurally different, but the Related Work section mainly cites this work in the context of Roofline analysis rather than systematically comparing their representations, available execution paths, and quality–resource tradeoffs. Consequently, the additional value of retaining a joint latent and supporting two paths is not sufficiently established. The manuscript also frequently adopts a defensive presentation style. Across the abstract, introduction, related work, and experimental discussion, claims are repeatedly followed by caveats about their scope, experimental coverage, or conclusions that cannot be drawn. These disclosures are necessary, but their placement fragments the argument. The paper devotes considerable attention to delimiting claims without making equally clear which new conclusions have been established and why they matter relative to prior work. Section 4 exemplifies this issue. It primarily applies an existing Roofline framework to selected GPU specifications, cache layouts, and tensor-parallel configurations, calculating cache sizes, arithmetic intensities, and ideal speedups. These calculations help explain performance, and the per-rank memory threshold is useful. However, the section does not clearly establish which conclusions provide new insight beyond existing hardware-aware attention analyses. General path-selection principles are dispersed among configuration details and numerical calculations, making the section read more like a deployment-performance report than a focused account of the paper's analytical contribution. The amount of main-text detail is not well justified by the incremental insight presented. I suggest reorganizing the exposition around the specific new capability or insight, consolidating the necessary scope statements, and moving detailed configuration calculations to the appendix. The main text should clarify how the analysis differs from prior work and how its predictions guide and are validated by the subsequent experiments.

  1. The quality cost of conversion is not sufficiently connected to deployment benefits relative to the original models. Table 2 shows substantial quality losses for direct conversion of GQA checkpoints. With the selected NLL-weighted configuration, LLaMA-3-8B's average accuracy falls from 69.02 to 56.94, and Qwen2.5-7B's from 70.39 to 67.17. Recovery training improves these averages, but requires additional training budgets, while perplexity and some individual task scores remain worse than those of the source models. Original GQA already supports KV grouping and sharding. The paper therefore needs to establish when the additional compact-latent execution path provides enough benefit to justify conversion losses and recovery costs. The main serving experiment does not directly answer this question for GQA-source models. For DeepSeek, the converted model retains a similar eight-task average, but perplexity increases from 3.21 to 6.05. The multiple-choice evaluations do not sufficiently characterize the impact on open-ended generation or long-context applications. Table 4 validly isolates the performance difference between two execution paths of the same converted model, but does not establish the complete quality–efficiency tradeoff of converting the original model.
Questions for authors
  1. What measurable benefit does the joint-latent, dual-path design provide over GLA? Please systematically compare GQLA and GLA in terms of representation, available execution paths, per-rank cache requirements, and parallelism constraints. Can a comparison at a manageable model scale, with comparable training and cache budgets, demonstrate the quality and inference-performance benefits of retaining the joint latent? Please also clarify which conclusions of Section 4 extend beyond existing hardware-aware attention analyses.

  1. Under which deployment conditions is converting an existing GQA model worthwhile? For at least one GQA-source model, please compare GQLA with the original GQA model on the same hardware and workload, reporting memory use, throughput, latency, the associated quality changes, and any recovery-training budget. Please distinguish benefits from the cache representation from those obtained by increasing the batch size, and make the required quality tradeoff explicit.

  1. What practical capabilities are affected by the increase in DeepSeek perplexity? Given the increase in perplexity from 3.21 to 6.05 despite a similar eight-task average, can representative generation or long-context quality evaluations clarify the converted model's applicability? Please distinguish comparisons between the original and converted models from comparisons between the two execution paths of the same converted model.
1 comment

W1. Regarding the defensive presentation

A1. As non-native English speakers, we used AI assistance for language editing, as disclosed in the AI Usage Disclosure section. This may have resulted in overly cautious and formal wording in some parts of the paper. We appreciate this feedback and will improve the clarity, readability, and overall flow of the manuscript.

W2. Regarding the benefits of converting GQA to GQLA

A2. GQA already supports efficient KV-cache grouping and sharding. The primary benefit of converting GQA to GQLA is the additional MQA-absorb execution path, which substantially reduces KV-cache memory traffic. Similar to TransMLA, this conversion trades additional computation for reduced memory access, making it particularly attractive on compute-rich hardware such as the H100.

Q1. What quantifiable benefits does GQLA offer over GLA?

A1. The key distinction is that GQLA supports two mathematically equivalent decoding paths, GQA and MQA-absorb, allowing the same model to adapt to hardware with contrasting computational characteristics, such as the H100 and H20.

GLA retains MLA's MHA-style prefill structure and partitions the latent representation into groups, supporting only one decoding mode. In contrast, GQLA preserves the full shared latent dimensionality while introducing grouped KV heads to improve parallelism. Thus, GQLA enables flexible execution without reducing the dimensionality of the shared latent representation.

While our experiments quantify the benefits of switching between GQLA's two execution paths, we acknowledge that a direct quantitative comparison against GLA remains necessary.

Q2. Under what deployment conditions is converting an existing GQA model worthwhile?

A2 Converting GQA to GQLA is particularly beneficial in memory-bandwidth-constrained scenarios, where the MQA-absorb path reduces KV-cache traffic. To validate this, we conducted additional serving experiments comparing the original Llama 3 8B (GQA) with its recovered GQLA counterpart (30B CPT + approximately 47M SFT tokens).

Experiments used NVIDIA L20Z 80GB GPUs, vLLM 0.22.1, BF16 weights and KV caches, and a maximum context length of 8,192 tokens, with prefix caching and speculative decoding disabled. Both models received identical input token sequences.

  • Fixed concurrency: At batch size 31 with 7,680 input and 512 output tokens, GQLA achieved 1,291.5 tokens/s versus 1,062.9 tokens/s for GQA, a 21.5% improvement. Results were averaged over six runs with GPU deployments swapped. However, at batch size 1 with 2,048 input tokens, GQLA was 10.1% slower.
  • Fixed KV-cache budget: Under the same 32 GiB KV budget, GQLA supported batch size 110 versus 31 for GQA, increasing throughput from 1,062.9 to 1,614.2 tokens/s (+51.9%). This improvement also reflects the higher concurrency enabled by KV-cache compression.
  • Offline batch inference: Across input/output lengths of 1,024/1,024, 2,048/2,048, and 4,096/4,096 tokens, GQLA achieved 1.265×, 1.534×, and 2.307× throughput, respectively, including the effects of increased scheduling capacity.

These results demonstrate that GQA-to-GQLA conversion is most beneficial for high-concurrency, long-context, and memory-constrained inference, whereas the original GQA may be preferable at low concurrency.

Q3. Which practical capabilities are affected by the increased perplexity of DeepSeek?

A3. Thank you for raising this important concern. Upon carefully reviewing our vLLM implementation of DeepSeek-V3.1-Base, we discovered a parameter and configuration loading error that caused the converted models to inadvertently use the original model weights during downstream evaluation. This explains why different conversion methods produced nearly identical results, with minor variations attributable to randomness.

The issue was limited to DeepSeek-V3.1-Base evaluations on PIQA, Winogrande, ARC, BoolQ, MMLU, HellaSwag, and OpenBookQA. Evaluations of other models and all perplexity measurements were unaffected. Consequently, the apparent discrepancy between the substantial perplexity increase and nearly unchanged downstream performance resulted from this implementation error.

We sincerely apologize for this oversight. Given its impact on the reliability of our reported results, we have decided to withdraw the paper, thoroughly re-evaluate the corrected implementation, and address the reviewers' concerns before resubmission.

Reviewer_TrYZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper “GQLA: Group-Query Latent Attention for Hardware-Adaptive Decoding” proposes a novel attention architecture that enables hardware-adaptive LLM inference by combining the advantages of Grouped-Query Attention (GQA) and Multi-head Latent Attention (MLA). Specifically, GQLA introduces group-shared key/value up-projections over a shared latent representation, allowing the same model weights to support two mathematically equivalent decoding paths: compact MQA-absorb and expanded GQA execution.

Critical strengths
  1. The paper addresses an important challenge in efficient LLM inference

  2. The proposed GQLA architecture combines the advantages of MLA and GQA through group-shared KV up-projections.

  3. Strong empirical evaluation.

Critical weaknesses
  1. Although the converted DeepSeek-V3.1-Base maintains similar average accuracy on eight downstream benchmarks (77.84% vs. 77.79%), its WikiText-2 perplexity increases substantially from 3.21 to 6.05. This suggests considerable information loss during conversion, which may affect generation quality and generalization to tasks not covered by the evaluation.

  2. The paper primarily evaluates model quality using eight conventional zero-shot benchmarks, which may not adequately capture the impact of architectural conversion on real-world LLM capabilities. More comprehensive evaluations, including instruction following, mathematical reasoning, coding, and long-context understanding, would strengthen the claim that GQLA preserves model performance.

Questions for authors

please see weakness

1 comment

W1. The WikiText-2 perplexity of DeepSeek-V3.1-Base increases from 3.21 to 6.05.

A1. In this paper, we first propose a novel model architecture, GQLA, which achieves a favorable balance on both compute-rich hardware, such as the H100, and compute-constrained hardware, such as the H20. To avoid the substantial cost of training a GQLA model from scratch, we further propose TransGQLA, which builds upon TransMLA by introducing two key improvements: grouping and NLL PCA. These improvements significantly reduce the performance degradation caused by training-free conversion.

Experiments on LLaMA-3-8B, Qwen2.5-7B, and GLM-4.7-Flash demonstrate that the performance of the converted models can be substantially recovered using only a small number of training tokens.

However, DeepSeek-V3.1-Base contains 671 billion parameters, making further training prohibitively expensive for us. Therefore, we did not conduct this experiment on DeepSeek-V3.1-Base. Nevertheless, we believe that TransGQLA can substantially reduce the training cost of obtaining a GQLA model that delivers strong inference performance on hardware platforms with contrasting computational bottlenecks.

W2. Please evaluate the model on a broader range of tasks.

A2. Following your suggestion, we have expanded our evaluation to include five additional benchmarks, the results are presented in the table below.

ModelVersionIFEvalGSM8KHumanEvalMBPPDROPAvg
Llama 3 8BOriginal9.0650.4237.8049.0056.2958.06
GQLA Direct Conversion + Hessian PCA8.870.451.830.0021.0637.52
GQLA After Recovery Training19.5949.5122.5634.8053.2856.30
Qwen2.5-7BOriginal28.4784.1557.3263.0057.1565.63
GQLA Direct Conversion + Hessian PCA18.8510.319.150.0044.4747.70
GQLA After Recovery Training24.2179.6843.2949.8060.7363.27
GLM-4.7-FlashOriginal49.5483.5535.9839.8068.5865.51
GQLA Direct Conversion + Similarity-Based Grouping + Hessian PCA29.3965.204.8815.2057.6355.24
GQLA After Recovery Training29.3974.9140.8547.6058.8663.44

These additional results further demonstrate that recovery training can substantially mitigate the performance degradation introduced by GQLA conversion across a diverse range of downstream tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.