acceptodds
Under review as a conference paper at ICLR 2027

Transforming Attention: Generating Multi-Head Attention from Learned Group Orbits

Abstract

Multi-head attention allows transformers to capture different relationships between tokens, but typically learns separate projection parameters for every head. Sharing parameters across heads offers a route to smaller attention blocks, with the central challenge of preserving each head’s ability to specialize. We introduce group-transformed multi-head attention (GT-MHA), which generates attention heads by applying learned transformations to a compact set of shared query, key, and value projections. Each head combines shared Lie-algebra generators to form its transformation through the matrix exponential or low-order Taylor approximations. Our analysis shows how heads within a shared orbit represent different interactions within common query and key subspaces. We then extend GT-MHA to multiple learned bases (a multi-orbit formulation), allowing different groups of heads to specialize in different subspaces while controlling the degree of parameter sharing. The resulting architecture preserves the standard multi-head interface and supports key–value caching at the level of shared bases. Under matched BERT pretraining and fine-tuning protocols, residual GT-MHA achieves an official GLUE score of 70.5 versus 70.3 for standard multi-head attention, using 47.2% fewer attention-block parameters. On TinyStoriesV2 character-level language modeling, GT-MHA achieves comparable validation perplexity to standard multi-head attention (1.3331 vs. 1.3355) under the same training budget, with 54.7% fewer attention-block parameters and 75% less persistent KV-cache storage. Ablations and head-diversity analyses provide evidence of learned specialization within shared bases, supporting structured parameter sharing as an effective design for parameter-efficient attention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-head attention, Lie groups, KV-cache compression, Structured attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y2wXOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Group-Transformed Multi-Head Attention (GT-MHA), which generates multiple attention heads from a smaller set of shared Q/K/V projections using head-specific matrix transformations constructed from a shared generator bank. The method aims to reduce attention parameters while retaining head-specific attention patterns, and it also enables KV caching at the shared-base level. The paper evaluates GT-MHA on BERT/GLUE and TinyStories, reporting comparable performance to MHA with fewer attention parameters and reduced persistent KV-cache storage.

Critical strengths
  1. Clear presentation and technically concrete idea. The paper provides a clear description of the paper, and the proposed parameterization is relatively easy to follow and implement.
Critical weaknesses
  1. The efficiency improvements are not sufficiently established. Reducing attention parameters has limited practical significance for large-scale language models, where attention projections constitute a relatively small fraction of the total parameters, especially in MoE models. More importantly, the efficiency comparisons do not sufficiently reflect existing efficient attention architectures. The reported 12.6% FLOP reduction is measured relative to standard MHA, while the paper does not report a corresponding comparison with GQA or MLA at comparable KV-cache budgets. In fact, the paper itself shows substantially lower training throughput for GT-MHA than MHA, GQA, and MQA (61.1k vs. 94.5k, 93.1k, and 93.2k tokens/s), despite the lower profiler-reported FLOPs. The paper also notes that unsupported fused/custom operations are excluded from the profiler count. Thus, the reported FLOP reduction does not translate into a wall-clock speedup in the current implementation. Similarly, the 75% KV-cache reduction concerns persistent cache storage, but GT-MHA introduces additional head-specific transformations, and the paper does not establish an end-to-end inference speed or memory advantage over existing KV-sharing approaches such as GQA or latent-KV approaches such as MLA.

  2. The experiment scale is too small, and the practical applicability of the method is unclear. The experiments are limited to BERT-base-scale GLUE experiments and a 12-layer TinyStories model. In particular, each TinyStories variant uses only one training run, so the reported bootstrap intervals do not capture training-run variability. These experiments therefore provide limited evidence that the proposed strong sharing of Q/K/V projections will remain competitive as model size and training data scale. The practical cost of the additional head-specific transformations is also not sufficiently evaluated within existing infrastructure frameworks. Finally, the role of the Lie-group formulation is somewhat unclear in the main experimental setting: the default transformation is (I+A), which does not in general guarantee group membership, while the exact exponential mapping performs worse in the reported ablation.

Questions for authors
  1. Can the authors provide experiments on larger decoder-only language models and larger-scale training data, preferably with multiple training runs, to evaluate whether the performance of GT-MHA remains competitive as model size and training scale increase?

  2. How does GT-MHA compare with GQA and MLA under matched KV-cache budgets? Can GT-MHA be combined with these approaches, and if so, what are the resulting parameter, KV-cache, FLOP, and throughput trade-offs?

  3. The paper attributes the current throughput gap to the additional transformations and the lack of specialized fused kernels. Can the authors provide a more concrete analysis of how the head-specific transformations could be fused with FlashAttention/SDPA-style kernels, including which memory reads/writes and computation could be eliminated or shared, and what overhead would remain after fusion?

3 comments

We thank reviewer y2wX for the careful assessment and concrete suggestions. We agree that parameter and persistent-cache reductions alone do not establish practical efficiency. We have expanded the evaluation to billion-parameter language models and added end-to-end inference measurements. Below, we distinguish the benefits supported by these experiments from the limitations that remain.

1. Larger decoder-only models and training-run variability.

Section 6.4 now evaluates billion-parameter decoder-only models trained from scratch on FineWeb-Edu. MHA, four-KV-head GQA, and GT-MHA share the non-attention architecture, tokenizer, data, and training protocol. We compare three training seeds per architecture at a common endpoint of 20B training tokens. The validation curves summarize all three seeds. For downstream evaluation, we select the checkpoint with the lowest validation NLL at that endpoint for each architecture. Across seven zero-shot tasks, these selected checkpoints achieve mean accuracies of 56.34% for MHA, 54.65% for GQA, and 55.63% for GT-MHA. GT-MHA uses 54.7% fewer attention parameters than MHA and 27.5% fewer than GQA. We also expand the TinyStories evaluation to include full-validation results for three MHA and three GT-MHA checkpoints, with checkpoint provenance and training configurations documented in Appendix D. Together, these additional evaluations and the matched three-seed FineWeb-Edu experiments broaden the empirical evidence from character-level modeling to billion-parameter language models.

2. Comparison with GQA and the scope of the efficiency claim.

We agree that GQA is a more informative practical reference than MHA alone. GT-MHA and four-KV-head GQA retain the same number of cached key/value streams; GT-MHA additionally shares query projections and generates head-specific transformations. Its persistent KV-cache storage therefore matches GQA and is 75% smaller than MHA.

Our contribution focuses on the attention-block architecture, preserving the feed-forward layers, embeddings, and other model components. In the FineWeb-Edu models, GT-MHA reduces attention parameters from 251.66M to 182.46M relative to GQA—a 27.5% reduction that translates into 5.9% fewer total model parameters. We report both quantities to distinguish savings within attention from their impact on the complete model. The resulting whole-model reduction depends on the proportion of parameters allocated to attention, particularly in architectures such as MoE models with substantial expert parameters.

Appendix D.2 measures the accompanying runtime and memory costs. For example, at batch size 8, decoding 128 continuation tokens after a 1,792-token prefix, fused GT-MHA achieves 429.32 tokens/s versus 593.08 for GQA. Peak allocated memory is 2.780 versus 2.923 GiB, with identical 720 MiB persistent caches. Cached decoding remains slower than GQA across all tested configurations. We therefore present a parameter–quality–memory–runtime trade-off, rather than a general speed advantage. We have not completed a matched-cache MLA benchmark and do not claim superiority over MLA. Combining GT-MHA with latent KV compression would require a separately specified and validated architecture and cache implementation; we do not report untested hybrid efficiency estimates.

3. Concrete fusion implementation and remaining overhead.

Following the reviewer’s suggestion, we have gone beyond attributing overhead to an unfused implementation. We implemented and benchmarked Triton transformation kernels around PyTorch SDPA.

On the query pathway, the kernel combines the generator coefficients, constructs the residual transformation, and applies it to the shared-base queries within one kernel. This avoids separately writing and rereading the intermediate head-transformation matrices. The value pathway similarly combines transformation construction with application to the aggregated head outputs, writing the result in the layout consumed by the output projection. Base-level keys and values remain shared in the persistent cache. This is transformation fusion around SDPA, not fusion into the attention kernel itself. Head-specific transformation arithmetic, intermediate query/output tensors, kernel launches, attention computation, and the output projection remain. We therefore do not assume that fusion eliminates the additional cost. On the same GH200, across batch sizes 1, 4, and 8 and four context lengths per phase, fusion provides geometric-mean speedups of 1.30× for prefill and 1.46× for decoding relative to reference GT-MHA. Each configuration uses 30 timing repeats. We also retain the qualification that profiler-reported FLOPs omit unsupported operations. The reported FLOP reduction is therefore not presented as evidence of an end-to-end speedup.

4. Role of the group formulation.

The exact group construction provides the theoretical motivation for GT-MHA: head-specific group transformations generate distinct scoring geometries within shared projection subspaces. Matrix exponentiation guarantees invertibility, while the residual approximation retains the shared bases, generator banks, and head-specific coefficients with a less restrictive transformation mapping.

Our mapping ablation shows a practical advantage for the residual formulation in both predictive performance and computational cost. It achieves lower TinyStories validation NLL than exact exponentiation (0.2875 versus 0.2894), together with higher training throughput (329.5 versus 306.1 thousand tokens/s/GPU).

A plausible explanation for the predictive improvement is the flexibility introduced by relaxing the exact construction: residual transformations can depart from the invertible, positive-determinant family induced by real matrix exponentiation. The group formulation thus motivates the structured sharing mechanism, while the residual approximation provides the most favorable observed quality–runtime trade-off among the mappings evaluated. We have clarified this relationship in the method, ablation discussion, and limitations.

We appreciate the reviewer’s emphasis on practical applicability and hope that the added experiments and clarified claims address these concerns. The revised paper provides a broader empirical evaluation of GT-MHA, quantifies its parameter savings and quality–runtime trade-offs, and clarifies the relationship between the motivating group formulation and the practical residual implementation.

Thank you for the additional experiments and clarifications. I appreciate the authors' efforts to extend the evaluation to billion-parameter models and optimize the implementation with Triton kernels. However, two major concerns remain unsolved:

  1. Practical efficiency gains. GT-MHA does not reduce KV-cache storage compared with GQA. While it reduces attention parameters by 27.5%, this translates to only a 5.9% reduction in total model parameters. As the authors also acknowledge, the overall savings depend on the proportion of parameters allocated to attention. In addition, the lack of a direct comparison with MLA makes it difficult to assess the practical advantages of GT-MHA over existing efficient attention architectures.

  2. Inference throughput. While I appreciate the additional Triton optimization, in the current reported decoding setting GT-MHA still achieves only 429.32 tokens/s compared with 593.08 tokens/s for GQA. This represents a roughly 27.6% throughput reduction, which I consider a major drawback for practical deployment. The reported fusion speedups are relative to the reference GT-MHA implementation and do not resolve this gap against GQA.

Overall, I acknowledge that the additional experiments have improved the paper, but I do not think they sufficiently address my main concerns regarding the practical efficiency and competitiveness of GT-MHA. I therefore maintain my original score.

Reviewer_sXpQOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper introduces group-transformed multi-head attention, which incorporates learned linear transformations into the attention mechanism. The proposed design aims to reduce redundancy across attention heads and improve overall performance. The paper evaluates the method against conventional attention variants, although the novelty, strength of the empirical gains, and applicability beyond the evaluated scales remain insufficiently established.

Critical strengths

The paper is generally well organized, with a clear separation between the proposed method and the experimental evaluation. This structure makes the overall contribution reasonably easy to follow, even though the motivation for some specific architectural choices requires further clarification.

Critical weaknesses
  1. The architectural motivation and novelty relative to prior work are insufficiently established.

The reasoning leading from the identified limitation of multi-head attention to the proposed group transformations is difficult to follow. The paper should explain why this particular design is a principled response to the stated problem, rather than merely another architectural variation.

In particular, the motivation should distinguish between a low-rank bottleneck within individual attention heads and redundancy across different heads. These are not interchangeable claims. The former was already investigated in Low-Rank Bottleneck in Multi-head Attention Models (2020), which analyzed the relationship between head count, head dimension, and expressivity. The paper needs to clarify what new limitation or insight it identifies beyond this earlier work, and provide direct evidence that its transformations address the claimed redundancy.

  1. The baseline coverage is limited, and the reported gains are not compelling.

The improvements appear modest even against the relatively limited set of included baselines. More importantly, the evaluation does not sufficiently establish the benefit of the proposed design over closely related approaches.

A particularly relevant comparison is Talking-Heads Attention (2020), which introduces learned linear projections across the head dimension immediately before and after the softmax. Although this does not establish that the two methods are equivalent, it makes a direct conceptual and empirical comparison important. The authors should explain how their transformations differ and whether those differences yield a meaningful advantage under comparable training and parameter budgets.

Comparisons with such methods would be more informative than demonstrating improvements only over standard attention variants. They would also help distinguish the value of the proposed group structure from the more general benefit of introducing additional learned transformations.

  1. The omission of MQA from Figure 1(a) leaves an important comparison unresolved.

Figure 1(a) appears to show that moving from multi-head attention (MHA) to grouped-query attention (GQA) reduces the number of attention parameters without an obvious reduction in the reported score. This makes the missing multi-query attention (MQA) result particularly relevant: how far can the parameter reduction be pushed before performance degrades, and where does the proposed method improve on that trade-off?

The trend from MHA to GQA does not imply that MQA will also preserve performance, but this is precisely why its result would be informative. Including MQA under the same evaluation protocol would help establish whether the proposed transformations offer a meaningful benefit over simpler alternatives. The paper should distinguish gains in task performance from gains in parameter efficiency.

  1. The experimental scale is too limited to support broader claims.

The experiments are conducted at a small scale and do not establish whether the observed benefits persist as model size or training budget increases. The current evidence also provides limited support for generalization beyond the evaluated settings.

The paper does not necessarily need to demonstrate improvements at every scale, but it should either provide a meaningful scaling comparison or explicitly restrict its conclusions to the configurations studied. As presented, the relevance of the proposed modification to larger-scale models remains an open empirical question.

  1. Important explanatory figures should be included in the main paper.

Some figures in the appendix, particularly Figure 4, appear useful for understanding the architectural design. These should be moved into the main paper and discussed alongside the relevant method description. Readers should not need to consult the appendix to understand the central mechanism or the rationale for its components. The appendix is better suited to supplementary results and implementation details.

Questions for authors
  1. What evidence shows that the proposed transformations specifically reduce harmful inter-head redundancy, rather than simply changing the parameterization or adding capacity? How does this motivation differ from the previously studied low-rank bottleneck?

  2. How does the method compare with Talking-Heads Attention under comparable parameter and training budgets? Which specific aspect of the proposed group structure accounts for any advantage?

  3. Why is MQA omitted from Figure 1(a), and how would including it change the interpretation of the performance–parameter trade-off?

  4. Is there evidence that the gains persist as model size or training budget increases? Which conclusions should be understood as limited to the current experimental scale?

4 comments

We thank reviewer sXpQ for the careful review and specific suggestions. We agree that the original manuscript needed a clearer architectural motivation and stronger empirical comparisons. We have revised the explanation, expanded the evaluation to billion-parameter language models, included MQA in the parameter–performance comparison, and added functional diagnostics and pruning experiments. We address the reviewer’s concerns below, including the requested Talking-Heads comparison, which remains incomplete.

1. Motivation, the low-rank bottleneck, and evidence of useful specialization.

We agree that limited expressivity within an individual head and redundancy across heads are distinct issues. Low-Rank Bottleneck in Multi-head Attention (2020) studies the former. GT-MHA retains the usual per-head rank bound and we do not claim to remove that bottleneck. Our question is whether heads can retain useful differences in their scoring functions while sharing projection parameters. GT-MHA separates two choices: the projection subspaces available to a head and the bilinear interactions learned within those subspaces. Heads assigned to a base share query and key projections, while a head-specific group transformation—or its residual approximation—determines how the projected coordinates interact. Unlike coordinatewise reweighting, which only rescales matching query–key coordinates, a dense transformation allows each query coordinate to interact with multiple key coordinates. Shared generator banks compactly parameterize the transformations through head-specific coefficient vectors, while multiple learned bases allow groups of heads to operate in different projection subspaces.

We have added complementary evidence for this design. Appendix F measures attention and output diversity separately within and across bases. For example, the FineWeb-Edu checkpoint has mean normalized attention JSD of 0.341 within a shared base, demonstrating differentiated attention despite shared projections. Separately, the transformation-pathway ablations in Section 6.6 test predictive utility: restoring the value and scoring pathways raises official GLUE scores from 70.0 to 70.2 to 70.5 and lowers TinyStories NLL from 0.2925 to 0.2915 to 0.2875. These observations support useful head-specific behavior under sharing. They do not establish that GT-MHA causally removes harmful redundancy, and we have revised the motivation accordingly. We also clarify that the residual mapping relaxes exact group membership. Our mapping ablation favors the residual variant, but does not establish a predictive benefit from exact group constraints.

2. Comparison with Talking-Heads Attention.

We agree that Talking-Heads is a relevant comparison and now discuss the distinction explicitly. Talking-Heads learns linear mixing across head logits and attention weights, respectively before and after softmax. GT-MHA instead shares projection bases and constructs head-specific bilinear interactions within their query–key subspaces. These are different mechanisms, and the conceptual distinction alone does not establish an empirical advantage. We have implemented a TinyStories Talking-Heads baseline with the same tokenizer, data splits, non-attention architecture, effective batch size, and target training budget. Training is still in progress, and we will update this response if verified results become available before the discussion closes. We do not currently claim an empirical advantage over Talking-Heads.

3. MQA and the parameter–performance trade-off.

We have included MQA in the updated parameter–performance plot, Figure 4(a), alongside the official results in Table 1. MQA achieves a GLUE aggregate of 69.0 with 15.37M attention parameters, compared with 70.5 and 14.98M for GT-MHA. Thus, GT-MHA has an observed 1.5-point higher aggregate with slightly fewer attention parameters in this setting. We distinguish this result from a general claim of superiority across tasks or training runs.

4. Evidence beyond the original experimental scale.

Section 6.4 adds billion-parameter decoder-only models trained from scratch on FineWeb-Edu, with the non-attention architecture, tokenizer, data, and training protocol held fixed. We compare MHA, four-KV-head GQA, and GT-MHA at a common 20B-token endpoint using three training seeds per architecture. Validation curves summarize all three seeds. Downstream evaluation uses the checkpoint with the lowest validation NLL at that endpoint for each architecture. Across seven zero-shot tasks, these selected checkpoints achieve mean accuracies of 56.34% for MHA, 54.65% for GQA, and 55.63% for GT-MHA. GT-MHA uses 54.7% fewer attention parameters than MHA and 27.5% fewer than GQA.

Section 6.5 additionally compares against post-training pruning. At approximately matched attention-parameter counts, physically head-pruned MHA achieves 47.04% mean downstream accuracy versus 55.63% for GT-MHA. This comparison uses head removal without recovery training or distillation; it supports an advantage over that tested procedure, rather than over pruning with recovery. These experiments extend the evaluation substantially beyond the original settings. Please note that they do not establish a scaling law or competitiveness at arbitrary model sizes, and we retain those limitations explicitly.

5. Architectural illustration in the main paper.

Following the reviewer’s suggestion, we moved the architectural schematic into the main paper as Figure 1 and explain its components alongside the method. The appendix retains the detailed pseudocode and supplementary analysis.

We appreciate the reviewer’s suggestions and hope that the expanded experiments and clearer positioning address the principal concerns. We have also made the remaining limitations explicit, particularly the incomplete Talking-Heads comparison and the distinction between a structured parameterization and a demonstrated benefit from exact group constraints.

Thank you for the detailed response and the additional experiments. I appreciate the expanded evaluation, the inclusion of MQA, and the clearer distinction between the proposed parameterization and the original claims about redundancy. The new experiments strengthen the paper and lead me to improve my assessment. However, two central gaps remain, and increasing model size alone does not resolve them.

1. An architectural-efficiency gap: reducing attention parameters does not establish that the method addresses the relevant KV-cache bottleneck.

A central motivation for moving from standard MHA to GQA/MQA is that maintaining separate keys and values for every query head incurs substantial KV-cache storage and memory-bandwidth costs during autoregressive decoding. Sharing these representations can reduce those costs while retaining much of the predictive quality. This efficiency objective extends beyond reducing the number of projection parameters.

Accordingly, demonstrating differentiated head behavior or a more compact parameterization of MHA is insufficient to establish practical value. The paper needs to show whether preserving those differences produces a worthwhile accuracy–memory–latency trade-off against strong GQA/MQA configurations. The response does not sufficiently establish the resulting benefits in KV-cache memory, prefill and decoding throughput, or end-to-end latency.

A larger experiment cannot substitute for evaluating the bottleneck that motivates efficient attention design. Attention-parameter savings and improvements in head diversity do not, by themselves, establish that the resulting architecture makes better use of GPU memory during generation.

2. An application gap: the evaluation still provides limited evidence for the intended use in practical LLMs.

The original GLUE and TinyStories experiments were not well aligned with the generation workloads in which KV-cache efficiency matters. I acknowledge that the added billion-parameter decoder-only experiments on FineWeb-Edu extend the evaluation to a more relevant setting. Nevertheless, the results at a single 20B-token endpoint leave open whether the benefits persist with longer training and translate into useful trade-offs in long-context processing and autoregressive generation.

The new result against four-KV-head GQA is positive: GT-MHA achieves 55.63% mean accuracy versus 54.65%, with fewer attention parameters. I consider this a meaningful addition to the evidence and an important reason to revise my assessment upward. Further validation with larger training-token budgets and generation workloads that directly expose the relevant memory and latency costs would make the practical contribution more convincing.

Given the concern about prefill overhead, I also think Multi-head Latent Attention (MLA) would be a useful comparison for understanding the trade-off between cache savings and computation. However, this is an additional suggestion raised late in the discussion, and I do not count the absence of an MLA comparison as an additional weakness in my current rating.

I recognize that completing a more comprehensive evaluation within the rebuttal period may be difficult. The added experiments represent constructive progress, but the remaining uncertainty still prevents me from supporting acceptance.

The specific value of the proposed group structure also remains unclear.

The revised explanation makes the architecture easier to understand, but substantially narrows its original motivation. The authors acknowledge that the diagnostics do not establish the removal of harmful redundancy, and that the mapping ablation does not demonstrate a predictive benefit from exact group constraints. The residual approximation performs better.

The pathway ablations show that additional transformations can improve the tested model, but they do not establish why the proposed group construction is a particularly effective choice. The central question remains what this design contributes beyond simpler parameter sharing and learned linear transformations.

The previously requested empirical comparisons remain incomplete.

The Talking-Heads comparison was a central request in my original review and remains unavailable. I accept that the mechanisms differ, but that distinction alone does not establish a meaningful advantage. Likewise, comparing against head pruning without recovery training provides limited evidence of competitiveness against practical compression approaches.

Overall, the revision provides stronger empirical support and addresses part of my original concerns. I therefore raise my score from 1 (clear rejection) to 2 (weak rejection). The evidence is still insufficient for acceptance in my assessment, but I encourage the authors to build on these results, substantiate the practical efficiency benefits, and develop a more thoroughly validated contribution for a future submission.

Thank you for taking the time to read our revision and reconsider your assessment. We appreciate your thoughtful and constructive feedback, especially your suggestion to compare directly with Talking-Heads Attention.

The TinyStories Talking-Heads run is still in progress. We can now share preliminary results for seed 200 at 90,000 updates, the latest common point in the validation logs we analyzed. Each model had seen approximately 94.37 billion sampled training characters by that point.

MethodSampled validation NLL ↓Perplexity ↓Total parameters
Talking-Heads0.297401.34636152.11M
MHA0.298441.34775152.10M
GT-MHA0.304511.35596124.58M

At this training exposure, Talking-Heads has the lowest logged validation NLL. GT-MHA uses 27.53M fewer total parameters (18.1%) than Talking-Heads, with a sampled validation NLL 0.0071 nats per character higher. The runs use the same validation split.

Unfortunately, the MHA and GT-MHA checkpoints at 90,000 updates are unavailable for reevaluation, so we could not perform a comparison of full-validation loss and generation quality at this training budget.

Reviewer_HbSrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GT-MHA, which shares base Q/K/V projections across attention heads and differentiates heads through transformations generated from shared matrices. It provides a group-orbit interpretation and shared-subspace analysis. Experiments on BERT/GLUE and character-level TinyStories show comparable performance to standard MHA with fewer parameters. Ablations examine the sharing structure and transformation choices, with the first-order residual variant performing better than exact exponentiation in the reported experiments.

Critical strengths

The paper presents a clear way to share attention projections while retaining head-specific transformations. It achieves comparable performance with 12.2%–18.1% fewer total parameters than MHA in the evaluated models. The ablations provide useful evidence that learned transformations contribute beyond projection sharing, and the paper is transparent about implementation and evaluation limitations.

Critical weaknesses
  1. I am not yet convinced that the practical benefit over GQA is substantial enough. The total parameter savings are about 3.9% for BERT and 6.5% for TinyStories, with the same persistent KV-cache size as GQA with four KV heads. GT-MHA has the lowest BERT training throughput among the evaluated baselines: 61.1k versus 93.1k tokens/s for GQA. The authors attribute this to the unfused implementation, but decoding throughput and latency are not reported either. Retaining the compact parameterization requires applying head-specific transformations during decoding, whose runtime impact is not measured. Fully folding these transformations into standard dense GQA projections would eliminate the inference weight-storage advantage over GQA. Without evaluating this trade-off, the practical value of the additional compression remains unclear.

  2. The conceptual and empirical evidence feels insufficient for a broader architectural contribution. The best-performing variant drops the exact group constraints, so it remains unclear what the group formulation adds beyond a structured matrix parameterization. The identity ablations do not resolve this question. Evaluation is limited to BERT-base-scale models and a small character-level TinyStories model. Given the small GLUE gain and the lack of a clear generation-quality advantage over GQA, I would need stronger evidence of robustness and relevance beyond these settings to support acceptance.

Questions for authors

Check above.

4 comments

We thank reviewer HbSr for their time and valuable feedback. Their comments helped us identify two areas that needed stronger evidence: (1) the practical trade-off against GQA and (2) the applicability of GT-MHA beyond the original experimental scale. We address these concerns below with new inference measurements and experiments on billion-parameter language models trained from scratch on FineWeb-Edu.

1. Practical benefit over GQA and the cost of the transformations.

We benchmark MHA, four-KV-head GQA, reference GT-MHA, and fused kernel GT-MHA on the same NVIDIA GH200 GPU, using identical inputs and numerical precision. The measurements cover batch sizes 1, 4, and 8, with four context lengths per phase and 30 timing repeats per configuration. Appendix D.2 reports prefill latency, cached-decoding throughput, persistent KV-cache storage, and peak allocated GPU memory (highlighted in purple).

Representative inference measurements on one GH200 in BF16. Matched 5B-token checkpoints; batch size 8. Prefill uses 2,048 tokens; decoding processes 128 continuation tokens after a 1,792-token prefix. Timing results are medians; peak memory is maximum allocated GPU memory.

MethodPrefill (ms) ↓Decode (tokens/s) ↑KV cache (MiB) ↓Peak memory (GiB) ↓
MHA131.41616.792,8805.295
GQA118.14593.087202.923
GT-MHA, reference123.33293.717202.781
GT-MHA, fused117.23429.327202.780

Across the complete workload grid, fusion provides geometric-mean speedups of 1.30× for prefill and 1.46× for decoding over reference GT-MHA, although cached-decoding throughput remains below GQA across all tested configurations. GT-MHA therefore offers a trade-off among parameter count, predictive performance, and decoding speed. Our method targets the attention blocks, reducing their parameter count from 251.66M for GQA to 182.46M for GT-MHA—a 27.5% reduction—while retaining the same persistent KV-cache size. Because the feed-forward layers and embeddings remain unchanged, this translates to a 5.9% reduction in total model parameters. The FineWeb-Edu results discussed below show that this additional compression retains competitive predictive performance, with slower cached decoding in the current implementation.

2. Evidence beyond the original experimental scale.

We add a larger-scale evaluation on FineWeb-Edu in Section 6.4 to test whether GT-MHA remains competitive beyond BERT and character-level TinyStories. We train billion-parameter decoder-only models from scratch using MHA, four-KV-head GQA, and GT-MHA, keeping the non-attention architecture, tokenizer, data, and training protocol fixed. We compare the models after 20 billion training tokens, averaging validation curves over three training seeds. For each architecture, we select the seed with the lowest validation NLL at this endpoint for downstream evaluation. Across seven zero-shot tasks, GT-MHA achieves a mean accuracy of 55.63%, compared with 54.65% for GQA and 56.34% for MHA. GT-MHA therefore exceeds GQA by 0.98 percentage points and remains within 0.71 points of MHA, while using 27.5% fewer attention parameters than GQA and 54.7% fewer than MHA. Details can be found in Section 6.4 (highlighted in purple).

We also examine whether comparable compression can be obtained by pruning an existing MHA model. We apply gradient-based attention-head pruning to the 20B-token MHA checkpoint, physically removing selected heads without recovery training or distillation. At approximately matched attention-parameter counts—182.45M for pruned MHA and 182.46M for GT-MHA—pruned MHA achieves 47.04% mean downstream accuracy, compared with 55.63% for GT-MHA. At the GQA parameter budget of 251.66M attention parameters, pruned MHA achieves 52.12%, compared with 54.65% for GQA. In this experiment, learning the compact GT-MHA parameterization during training preserves substantially more downstream accuracy than post-training head removal at a comparable parameter budget. Details can be found in Section 6.5 (highlighted in purple).

3. What the group formulation contributes.

The exact group construction provides the theoretical motivation for GT-MHA: head-specific group transformations generate scoring geometries within shared projection subspaces. Matrix exponentiation guarantees invertibility, while the residual approximation retains the shared bases, generator banks, and head-specific coefficients with a less restrictive transformation mapping.

Our mapping ablation shows a practical advantage for the residual formulation in both predictive performance and computational cost. It achieves lower TinyStories validation NLL than exact exponentiation (0.2875 versus 0.2894), together with higher training throughput (329.5 versus 306.1 thousand tokens/s/GPU).

A plausible explanation for the predictive improvement is the flexibility introduced by relaxing the exact construction: residual transformations can depart from the invertible, positive-determinant family induced by real matrix exponentiation. The identity ablations further support a contribution from learned transformations beyond base sharing. The group formulation thus motivates the structured sharing mechanism, while the residual approximation provides the most favorable observed quality–runtime trade-off among the mappings evaluated.

We hope that the added experiments and clarified claims address the reviewer concerns. The revised paper presents GT-MHA as a measured trade-off among parameter count, predictive quality, memory, and runtime, with its current implementation limitations stated explicitly.

Thank you for the additional experiments. The larger-scale training results and inference benchmarks strengthen the empirical evidence and clarify the trade-off. However, my main concern remains: the total parameter reduction over GQA is still modest at 5.9%, with no additional KV-cache savings, while cached decoding remains slower than GQA even with the fused implementation. I am therefore still not convinced that the benefits justify the added complexity and runtime cost, and I maintain my weak reject recommendation.

Thank you for reviewing the additional experiments and for clarifying your remaining concern. We agree that GQA retains a cached-decoding throughput advantage in our benchmarks, and that the 5.9% reduction in total parameters should be considered alongside this cost.

However, we would also like to highlight the predictive quality side of this trade-off. At the matched 20B-token endpoint on FineWeb-Edu, GT-MHA achieved 55.63% mean zero-shot accuracy across seven downstream tasks, compared with 54.65% for four-KV-head GQA. This 0.98 percentage-point improvement came with 27.5% fewer attention parameters and the same persistent KV-cache size. We see this as the empirical benefit of GT-MHA in this setting: higher measured downstream accuracy with a smaller model at the same cache budget, at the cost of slower cached decoding in the current implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.