- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Group-Transformed Multi-Head Attention (GT-MHA), which generates multiple attention heads from a smaller set of shared Q/K/V projections using head-specific matrix transformations constructed from a shared generator bank. The method aims to reduce attention parameters while retaining head-specific attention patterns, and it also enables KV caching at the shared-base level. The paper evaluates GT-MHA on BERT/GLUE and TinyStories, reporting comparable performance to MHA with fewer attention parameters and reduced persistent KV-cache storage.
Critical strengths
- Clear presentation and technically concrete idea. The paper provides a clear description of the paper, and the proposed parameterization is relatively easy to follow and implement.
Critical weaknesses
-
The efficiency improvements are not sufficiently established. Reducing attention parameters has limited practical significance for large-scale language models, where attention projections constitute a relatively small fraction of the total parameters, especially in MoE models. More importantly, the efficiency comparisons do not sufficiently reflect existing efficient attention architectures. The reported 12.6% FLOP reduction is measured relative to standard MHA, while the paper does not report a corresponding comparison with GQA or MLA at comparable KV-cache budgets. In fact, the paper itself shows substantially lower training throughput for GT-MHA than MHA, GQA, and MQA (61.1k vs. 94.5k, 93.1k, and 93.2k tokens/s), despite the lower profiler-reported FLOPs. The paper also notes that unsupported fused/custom operations are excluded from the profiler count. Thus, the reported FLOP reduction does not translate into a wall-clock speedup in the current implementation. Similarly, the 75% KV-cache reduction concerns persistent cache storage, but GT-MHA introduces additional head-specific transformations, and the paper does not establish an end-to-end inference speed or memory advantage over existing KV-sharing approaches such as GQA or latent-KV approaches such as MLA.
-
The experiment scale is too small, and the practical applicability of the method is unclear. The experiments are limited to BERT-base-scale GLUE experiments and a 12-layer TinyStories model. In particular, each TinyStories variant uses only one training run, so the reported bootstrap intervals do not capture training-run variability. These experiments therefore provide limited evidence that the proposed strong sharing of Q/K/V projections will remain competitive as model size and training data scale. The practical cost of the additional head-specific transformations is also not sufficiently evaluated within existing infrastructure frameworks. Finally, the role of the Lie-group formulation is somewhat unclear in the main experimental setting: the default transformation is (I+A), which does not in general guarantee group membership, while the exact exponential mapping performs worse in the reported ablation.
Questions for authors
-
Can the authors provide experiments on larger decoder-only language models and larger-scale training data, preferably with multiple training runs, to evaluate whether the performance of GT-MHA remains competitive as model size and training scale increase?
-
How does GT-MHA compare with GQA and MLA under matched KV-cache budgets? Can GT-MHA be combined with these approaches, and if so, what are the resulting parameter, KV-cache, FLOP, and throughput trade-offs?
-
The paper attributes the current throughput gap to the additional transformations and the lack of specialized fused kernels. Can the authors provide a more concrete analysis of how the head-specific transformations could be fused with FlashAttention/SDPA-style kernels, including which memory reads/writes and computation could be eliminated or shared, and what overhead would remain after fusion?
3 comments
4. Role of the group formulation.
The exact group construction provides the theoretical motivation for GT-MHA: head-specific group transformations generate distinct scoring geometries within shared projection subspaces. Matrix exponentiation guarantees invertibility, while the residual approximation retains the shared bases, generator banks, and head-specific coefficients with a less restrictive transformation mapping.
Our mapping ablation shows a practical advantage for the residual formulation in both predictive performance and computational cost. It achieves lower TinyStories validation NLL than exact exponentiation (0.2875 versus 0.2894), together with higher training throughput (329.5 versus 306.1 thousand tokens/s/GPU).
A plausible explanation for the predictive improvement is the flexibility introduced by relaxing the exact construction: residual transformations can depart from the invertible, positive-determinant family induced by real matrix exponentiation. The group formulation thus motivates the structured sharing mechanism, while the residual approximation provides the most favorable observed quality–runtime trade-off among the mappings evaluated. We have clarified this relationship in the method, ablation discussion, and limitations.
We appreciate the reviewer’s emphasis on practical applicability and hope that the added experiments and clarified claims address these concerns. The revised paper provides a broader empirical evaluation of GT-MHA, quantifies its parameter savings and quality–runtime trade-offs, and clarifies the relationship between the motivating group formulation and the practical residual implementation.
Thank you for the additional experiments and clarifications. I appreciate the authors' efforts to extend the evaluation to billion-parameter models and optimize the implementation with Triton kernels. However, two major concerns remain unsolved:
-
Practical efficiency gains. GT-MHA does not reduce KV-cache storage compared with GQA. While it reduces attention parameters by 27.5%, this translates to only a 5.9% reduction in total model parameters. As the authors also acknowledge, the overall savings depend on the proportion of parameters allocated to attention. In addition, the lack of a direct comparison with MLA makes it difficult to assess the practical advantages of GT-MHA over existing efficient attention architectures.
-
Inference throughput. While I appreciate the additional Triton optimization, in the current reported decoding setting GT-MHA still achieves only 429.32 tokens/s compared with 593.08 tokens/s for GQA. This represents a roughly 27.6% throughput reduction, which I consider a major drawback for practical deployment. The reported fusion speedups are relative to the reference GT-MHA implementation and do not resolve this gap against GQA.
Overall, I acknowledge that the additional experiments have improved the paper, but I do not think they sufficiently address my main concerns regarding the practical efficiency and competitiveness of GT-MHA. I therefore maintain my original score.
We thank reviewer y2wX for the careful assessment and concrete suggestions. We agree that parameter and persistent-cache reductions alone do not establish practical efficiency. We have expanded the evaluation to billion-parameter language models and added end-to-end inference measurements. Below, we distinguish the benefits supported by these experiments from the limitations that remain.
1. Larger decoder-only models and training-run variability.
Section 6.4 now evaluates billion-parameter decoder-only models trained from scratch on FineWeb-Edu. MHA, four-KV-head GQA, and GT-MHA share the non-attention architecture, tokenizer, data, and training protocol. We compare three training seeds per architecture at a common endpoint of 20B training tokens. The validation curves summarize all three seeds. For downstream evaluation, we select the checkpoint with the lowest validation NLL at that endpoint for each architecture. Across seven zero-shot tasks, these selected checkpoints achieve mean accuracies of 56.34% for MHA, 54.65% for GQA, and 55.63% for GT-MHA. GT-MHA uses 54.7% fewer attention parameters than MHA and 27.5% fewer than GQA. We also expand the TinyStories evaluation to include full-validation results for three MHA and three GT-MHA checkpoints, with checkpoint provenance and training configurations documented in Appendix D. Together, these additional evaluations and the matched three-seed FineWeb-Edu experiments broaden the empirical evidence from character-level modeling to billion-parameter language models.
2. Comparison with GQA and the scope of the efficiency claim.
We agree that GQA is a more informative practical reference than MHA alone. GT-MHA and four-KV-head GQA retain the same number of cached key/value streams; GT-MHA additionally shares query projections and generates head-specific transformations. Its persistent KV-cache storage therefore matches GQA and is 75% smaller than MHA.
Our contribution focuses on the attention-block architecture, preserving the feed-forward layers, embeddings, and other model components. In the FineWeb-Edu models, GT-MHA reduces attention parameters from 251.66M to 182.46M relative to GQA—a 27.5% reduction that translates into 5.9% fewer total model parameters. We report both quantities to distinguish savings within attention from their impact on the complete model. The resulting whole-model reduction depends on the proportion of parameters allocated to attention, particularly in architectures such as MoE models with substantial expert parameters.
Appendix D.2 measures the accompanying runtime and memory costs. For example, at batch size 8, decoding 128 continuation tokens after a 1,792-token prefix, fused GT-MHA achieves 429.32 tokens/s versus 593.08 for GQA. Peak allocated memory is 2.780 versus 2.923 GiB, with identical 720 MiB persistent caches. Cached decoding remains slower than GQA across all tested configurations. We therefore present a parameter–quality–memory–runtime trade-off, rather than a general speed advantage. We have not completed a matched-cache MLA benchmark and do not claim superiority over MLA. Combining GT-MHA with latent KV compression would require a separately specified and validated architecture and cache implementation; we do not report untested hybrid efficiency estimates.
3. Concrete fusion implementation and remaining overhead.
Following the reviewer’s suggestion, we have gone beyond attributing overhead to an unfused implementation. We implemented and benchmarked Triton transformation kernels around PyTorch SDPA.
On the query pathway, the kernel combines the generator coefficients, constructs the residual transformation, and applies it to the shared-base queries within one kernel. This avoids separately writing and rereading the intermediate head-transformation matrices. The value pathway similarly combines transformation construction with application to the aggregated head outputs, writing the result in the layout consumed by the output projection. Base-level keys and values remain shared in the persistent cache. This is transformation fusion around SDPA, not fusion into the attention kernel itself. Head-specific transformation arithmetic, intermediate query/output tensors, kernel launches, attention computation, and the output projection remain. We therefore do not assume that fusion eliminates the additional cost. On the same GH200, across batch sizes 1, 4, and 8 and four context lengths per phase, fusion provides geometric-mean speedups of 1.30× for prefill and 1.46× for decoding relative to reference GT-MHA. Each configuration uses 30 timing repeats. We also retain the qualification that profiler-reported FLOPs omit unsupported operations. The reported FLOP reduction is therefore not presented as evidence of an end-to-end speedup.