- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposed GQLA, which is a latent attention mechanism. Unlike the original MLA, in which the latent KV expands into 256 KV-heads and matches the query heads (equivalent of MHA), GQLA expands the latent cache into a group of KV heads, which then attends to a greated number of query heads (similar to GQA). Moreover, the paper presents TransGQLA, which allows converting the existing models into GQLA without any need for gradient updates.
Critical strengths
-
The core idea is clear and intuitive. The proposed mechanism seems to be a natural and promising update over the existing MLA architectures.
-
The paper did not only present a new attention mechanism, but also introduces a framework to convert the existing MLA layers into the proposed GQLA, which increases the practicality of the proposed method.
Critical weaknesses
-
It's difficult to understand the claims, position the paper in the literature, interpret the results etc. Some figures are just put in the paper and never referenced in text (e.g., Figure 2). The writing quality of the manuscript is not up to ICLR's standards.
-
Method lacks experimental evaluation and comparison against important prior work:
- Grouped Latent Attention (GLA) is cited but never compared against.
- TPLA, Tensor-Parallel Latent Attention (Tang et al., 2025): not cited at all
- MLRA, Multi-Head Low-Rank Attention (Liu et al., ICLR 2026): not cited at all Without quantitative comparison against these methods, there is no motivation why any one would choose GQLA over any of these existing methods.
- The kernel evaluations are between two different existing kernels; therefore, the differences are possibly originating from implementation differences. Therefore, these results do not provide convincing evidence about performance gains.
Questions for authors
See the weaknesses above.
W1. Regarding the writing and presentation.
A1. In this paper, we first propose a novel model architecture, GQLA, which achieves a favorable balance on both compute-rich hardware, such as the H100, and compute-constrained hardware, such as the H20. To avoid the substantial cost of training a GQLA model from scratch, we further propose TransGQLA, which builds upon TransMLA by introducing two key improvements: grouping and NLL PCA. These improvements significantly reduce the performance degradation caused by training-free conversion.
The illustration in Figure 2 is inspired by the classical MHA and MQA representations of MLA, with the key differences between GQLA and MLA highlighted in orange and blue. Figure 2, together with the subsequent Section 3.1 of the Method section, explains the two equivalent forms of GQLA illustrated in the figure. To improve clarity and strengthen the connection between the figure and the main text, we will explicitly add references such as "As shown in Figure 2" in the revised manuscript.
W2. Comparison against GLA, TPLA, and MLRA.
A2. The key distinction between GQLA and GLA, TPLA, and MLRA is that GQLA is the first architecture capable of equivalently switching between GQA-style and MQA-style decoding, enabling it to adapt to hardware platforms with fundamentally different computational characteristics, such as the H100 and H20.
Specifically, GLA, TPLA, and MLRA all retain the MHA-style prefill structure of MLA and partition the latent representation into groups, supporting only a single decoding mode. In contrast, GQLA preserves the full latent dimensionality and replaces MHA heads with GQA heads to improve parallelism.
While GLA, TPLA, and MLRA enhance parallelism by partitioning the shared latent representation, potentially sacrificing the expressive capacity of the original high-dimensional latent space, GQLA improves parallelism without reducing the dimensionality of the shared latent representation. More importantly, its ability to switch equivalently between GQA-style and MQA-style decoding allows GQLA to achieve efficient inference across hardware platforms with contrasting computational bottlenecks.
W3. The differences are possibly originating from implementation differences.
A3. We would like to clarify that both execution paths use well-established, highly optimized kernels: FA3-MLA (FlashAttention-3) for MQA-absorb and Tencent HPC-Ops for GQA, rather than custom implementations.