acceptodds
Under review as a conference paper at ICLR 2027

Multi-Head Attention Residuals

19 views

Abstract

Transformers propagate information across depth through a single additive residual stream. Attention residuals relax this by letting each sublayer attend, through a learned softmax, over all earlier sublayer outputs - but that read shares one query across the entire width, forcing every feature subspace through the same depth distribution, at a cost that grows with model width. We introduce Multi-Head Attention Residuals (MHAR): reshape the routing query into H per-subspace heads, each with its own softmax over depth. The reshape adds zero parameters and negligible compute, and H=1 recovers attention residuals exactly. Trained from scratch on a STEM- and code-heavy corpus we construct, MHAR improves validation loss over a standard Transformer by 0.061/0.149/0.140 at 100M/350M/1B - the best of four methods in every setting - and the gains transfer to held-out perplexity and LAMBADA accuracy, survive a document-disjoint re-evaluation, and persist at a compute-optimal token budget rather than vanishing as an under-training artifact. Validation loss is U-shaped in H with a flat optimum at H=4-8; we adopt H=8. Temperature-matched controls attribute most of the gain to the flatter per-head softmax the reshape provides automatically - a single query's depth softmax over-sharpens as width grows - with a small residual consistent with per-subspace routing. Fused Triton kernels lift attention-residual training from 0.2-0.5x to 0.55-0.88x baseline throughput at near-baseline memory, and an identity-preserving conversion brings MHAR to 8B mid-training (+3.2 GSM8K, +3.1 GPQA).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Head Attention Residuals

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6aC6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies Multi-Head Attention Residuals, applying the multi-head aspect of regular token attention to depth-wise selection. This variation uses multiple queries, each seeing only a subset of the total key and each producing its own softmax distribution over previous sublayer outputs. This allows different subsets of the input to a sublayer to “see” different depth mixtures of previous sublayer outputs. This method adds no extra parameters. The single-head formulation is recovered at H=1H=1. The paper evaluates 100M, 350M, and 1B models trained from scratch on two corpora, and an 8B model adapted through a gated delta-routing retrofit. It reports improvements over a standard pre-norm Transformer, explores routing-head ablations and temperature, and provides fused forward and backward Triton kernels. The temperature controls indicate that much of the observed improvement over ordinary single-head routing can be recovered by changing the single-head logit scale.

Critical strengths
  1. Fused kernel

I appreciate the paper providing fused forward and backward routing kernels with a shared source buffer, reducing repeated source materialization and combining several memory-bound operations. The reported improvements over eager and compiled reference implementations make this a potentially useful contribution. It would be even more informative if the authors could compare their kernel at H=1H=1 with popular public implementations, e.g. the AttnRes kernel in Flash Linear Attention.

  1. Empirical reconsideration of routing-head count and temperature

The paper revisits multi-head depth routing across several head counts, scales, and data settings, and includes a particularly informative temperature-matched single-head control. The finding that logit scaling can recover much of the single-head deficit is useful for understanding the mechanism and configuring attention residuals.

Critical weaknesses

The paper does not convincingly establish a consistent benefit from independent routing heads beyond the temperature effect. At 350M, the temperature-matched single-head model achieves 2.847 compared with 2.848 for MHAR, while the remaining 1B web-data advantage is single-seeded and near the stated noise floor. This makes it difficult to determine how much of the improvement is actually due to independent subspace routing.

Despite these qualifications, Section 4 attributes the head-count curve to mismatches between subspace preferences and KV-group granularity. Dense Q/K/V projections do not inherently assign particular residual-coordinate slices to particular KV groups, and the reported experiments do not establish this mechanism.

The same attribution problem applies to the 8B result: the comparison is between plain continued pretraining and a combined gated delta-routing/MHAR retrofit, without an otherwise identical delta H=1H=1 control. The current 8B result therefore supports the combined retrofit but does not isolate the contribution of multiple heads. Likewise, the longer-budget comparison in Table 10 establishes the persistence of a benefit over the plain Transformer, but does not isolate the contribution of multiple heads.

This matters because the paper’s additional contribution should be demonstrated against appropriately configured single-head routing. The current evidence supports the importance of routing temperature more strongly than it supports an independent benefit from multiple depth distributions.

Questions for authors

1. Comparators, mechanism, and contribution

(a) Temperature-matched comparison.

At 350M, scaling the single-head logits already matches MHAR: 2.847 versus 2.848. The remaining 1B web-data advantage is also from a single seed and near the noise floor. What do independent heads add once initialization, normalization, logit scale, optimizer settings, and implementation are matched?

(b) Subspace disagreement and the head-count optimum.

Figure 4 shows that the heads learn different distributions, but what shows that these differences improve loss? Does the U-shaped head-count curve remain after controlling logit scale? I also don’t see what establishes the KV-group explanation, since dense projections use all residual coordinates and H=4H=4–8 is preferred across several KV counts. If this is still a hypothesis, please say so consistently.

(c) Isolating the 8B multi-head contribution.

Why not include the same delta retrofit with H=1H=1, alongside plain CPT, keeping the sources, gates, initialization, optimizer, data order, and schedule fixed? Ideally, this would also include a temperature-matched control. Without it, how much of the gain can be attributed to multiple heads rather than the combined retrofit?

(d) Relationship to prior work.

Please also discuss Low-Rank Attention Residuals and, if possible, include a comparison. This seems directly relevant to whether the benefit comes from smaller routing keys or from allowing different subspaces to choose different depth distributions.

2. Validation, budgets, downstream evaluation, and scaling

(a) Main and longer training budgets.

Why train the nominal 100M model on approximately 2.6B tokens but give each larger model only 0.66B? I see that you added longer runs in §3.4/Table 10, but why not include ordinary and temperature-matched single-head routing there? Those comparisons would show whether the additional multi-head benefit survives the longer budget.

(b) Primary validation and tail means.

Why average eleven evaluations of changing checkpoints on reshuffled training data instead of evaluating a fixed checkpoint on a large held-out set? Table 11 helps, but can it also include the temperature-matched single-head control with paired uncertainty estimates? Please clarify whether the main number measures final-checkpoint performance or average performance near the end of training.

(c) Downstream sample sizes and comparisons.

Why only 200 HellaSwag examples and 500 examples per Table 12 task? The WikiText-2/LAMBADA comparisons with single-head routing are already mixed: MHAR is better in three, worse in two, and tied in one at the reported precision. Can you evaluate larger fixed splits, explain how examples were selected, and report paired uncertainty?

(d) The 7B result and scaling.

The 7B result is an important limitation: the reported improvement is only 0.010 nats, within noise. Please give it a complete table row with the configuration, training budget, comparators, and uncertainty. What does this result, together with the non-monotonic gains, imply for the scaling claim? The 8B retrofit uses a different training regime, so it does not directly resolve this question.

3. Memory

(a) Peak-memory measurements.

The peak memory increase from 41.5 to 42.0 GB at 100M is surprisingly small for Attention Residuals on the full setting. One would expect around a 2x increase. Can you provide the Table 8 benchmark, runtime settings, and an allocation trace if possible?

4. Presentation and conclusions

(a) Figure 2.

The zeros in Figure 2 can give the impression that standard residuals cannot access earlier outputs, even though those contributions are already in the latest cumulative state. Could you show all methods over the same raw-output history, with weights of one for standard residuals, or make the change of basis explicit?

(b) Scope of the conclusions.

The conclusion reads more broadly than the results support. Please make clear which “lowest loss” claims refer to the selected H=8H=8 configuration, and separate gains over the plain Transformer from gains over ordinary or temperature-matched single-head routing. Table 11 also shows a tie at 100M at the reported precision. The observed head-count optimum and the proposed KV/subspace explanation should be presented with different levels of certainty.

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer 6aC6

Thank you for the review. Our new controls clarify gains over plain Transformers and the role of temperature.

Q1(a): Temperature-matched controls

AR uses one depth distribution; scaled AR scales logits by 1/81/\sqrt{8}; MHAR uses eight, with no extra parameters over AR. New 533M FineWeb-Edu runs match initialization, normalization, optimizer, data and training settings within each seed, using 655M tokens. Final held-out losses:

SeedPlainARScaled ARMHAR
423.3522153.2897603.2639143.265344
13.3703823.2981853.2773203.286515
23.3678843.3047903.2753443.279407

MHAR minus scaled AR is +0.004896 ± 0.003949 nats (paired mean ± sample SD across seeds): scaled AR is better in all three pairs. New single-seed block-routing losses are 3.282414/3.268976/3.280565 for AR/scaled AR/MHAR, with the same ordering.

Appendix E's 1B web losses are 3.270/3.152/3.129 for AR/scaled AR/MHAR: single-seed tail means, unlike our new final-checkpoint results. The 0.023 MHAR advantage is near the noise floor and does not establish a consistent head benefit or scaling trend.

An enlarged-plain control adds 147,456 FFN parameters (4096→4098), exceeding MHAR's parameter count. Its losses at seeds 42/1 are 3.351784/3.371461. MHAR improves by 0.085693 ± 0.001056 nats (paired mean ± sample SD). This supports gains beyond that added capacity, without isolating independent heads.

Q1(b): Head-count curve and mechanism

The H sweep changes logit scale; distinct distributions alone do not establish a loss benefit. A scale-controlled sweep remains missing. Dense projections do not tie residual slices to KV groups. We will label the KV-group explanation a hypothesis and remove the probe's unsupported “upper-bound” interpretation.

Q1(c–d): 8B attribution and related work

The 8B result evaluates the combined gated delta/MHAR retrofit. Ordinary/scaled H1 controls with matched checkpoint, sources, gates, initialization and training remain incomplete; from-scratch block results cannot replace them.

We will discuss Low-Rank Attention Residuals and distinguish reduced routing-key width from separate subspace distributions. A matched comparison remains outstanding.

Q2(a): Training budgets and compute

At 20K updates, batch×length 64×2048 versus 32×1024 explains 2.62B versus 0.655B tokens; budgets match within each scale. Table 10's MHAR−plain gaps are −0.119/−0.081 at 7.0B/21.6B tokens. The 60K-step web runs give −0.056/−0.062 and include H1. Both omit scaled H1, leaving longer-budget head attribution unresolved.

Approximately time-matched comparison (single seed):

ModelTokensActive B200 GPU-hoursLoss ↓
533M Block MHAR655M2.2953.281
1.439B plain496M2.2953.348

Time excludes initialization, evaluation and checkpoint I/O. This supports the tradeoff under these recipes; convergence and separately tuned comparisons remain untested.

Q2(b): Validation and uncertainty

Table 1 averages eleven changing checkpoints on reshuffled samples without guaranteed held-out separation. Table 11 uses 400 fixed document-disjoint windows but omits scaled H1. New runs exclude held-out document IDs and evaluate final checkpoints on 19.98M identical scored tokens. Window/block bootstrap intervals favor scaled H1 in all three pairs, conditional on checkpoints, separately from seed variation. Blocks are not document clusters; content deduplication is unverified.

Q2(c–d): Downstream and scaling

The 200/500-example evaluations have limited resolution; WikiText-2/LAMBADA gives three wins, two losses and one tie against H1. We will document selection and narrow claims; larger splits with paired uncertainty remain missing. The 7B gap (−0.010 nats) is within noise. A verified row requires the original records; the different 8B regime cannot resolve this. We will remove the broad scaling claim.

Q3(a): Memory and kernels

Fusion stores sources once and recomputes normalization in backward, avoiding repeated stacks. Total peak includes weights, optimizer and other activations. New peak allocated memory (GB):

ScalePlainEager H8Fused H1Fused H8
115M36.2163.5839.1739.32
533M17.0868.5718.7418.89
1079M16.2387.2218.9319.11

Single B200; batch×length 8×2048/4×1024/2×1024; bf16; TF32 off; no compilation/activation checkpointing; 30 warmup + 90 measured training steps. These do not verify H100's 41.5→42.0 GB; its allocation trace and FLA H1 comparison are missing. H1/H8 runtimes differ.

Q4(a–b): Figures and conclusions

We will use the same raw-output history: unit weights for standard residuals, one shared distribution for H1, one per slice for MHAR. We will include scaled H1 and the 100M tie, scope “lowest loss” to tested configurations/comparators, and separate the observed H preference from the KV hypothesis.

Thank you for adding the matched controls and final-checkpoint held-out evaluation. These meaningfully address my evaluation concerns and provide useful evidence about routing temperature. The fused kernels also remain a strength.

However, my central concern remains unresolved: the additional benefit of independent routing heads over appropriately configured single-head routing has not been established. In the new 533M comparison, scaled single-head routing performs better in all three matched seed pairs, with MHAR’s loss higher by approximately 0.0049 nats on average. I appreciate the proposed qualifications to the mechanism and scaling claims. Nevertheless, the current evidence supports the importance of routing temperature more clearly than an additional benefit from independent depth distributions. I therefore maintain my score of 2 (weak rejection).

AuthorsOpenReview ↗

Rebuttal to Reviewer 6aC6

Thank you for the follow-up. We agree that routing temperature explains a large part of the improvement. We have now run additional softmax-scale experiments for both methods, and they reinforce your point: at 533M, a well-configured single head remains a strong baseline.

We would like to add a positive result at 1B that we would like to highlight. Our earlier 533M comparison and the existing 1B control are summarized below:

Model scaleSingle-head AR ↓Scaled single-head AR ↓MHAR ↓MHAR − scaled AR ↓
533M3.2983.2723.277+0.005
1B3.2703.1523.129−0.023

At 533M, scaled single-head routing does better. At 1B, MHAR has a 0.023-nat advantage. This is encouraging evidence that multiple heads may offer more benefit in larger models,.

One possible explanation is that scaling a single head makes its routing softer, but all features still share the same weights over earlier layers. MHAR lets different groups of features use different layer weights. That freedom may become more useful as models get wider.

We appreciate the reviewer’s insight into the connection with softmax scaling. We will make this connection more prominent in the ablation study, add the new scale-sweep results, and clearly separate what the experiments show from what still needs to be tested.

Reviewer_WPDKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a straightforward method of splitting Attention Residuals into multiple heads, which adds no parameters and in the H=1 case reduces exactly to single-headed Attention Residuals. It shows a variety of experiments on the impacts of this change on pretraining results as well as mid-training. It also provides a method for applying Multi-Head Attention Residuals at mid-training while avoiding training spikes, with downstream results. Additionally, fused kernels are provided that significantly increase speed compared with a simple autograd based approach.

Critical strengths

The presented methods are of interest, the paper is generally clear, and it presents a variety of experimental evidence.

Critical weaknesses

The main claim is overstated or untrue, the main models are dramatically undertrained with poor hyperparameters, the mid-training experiment is unclear and does not compare to attention residuals, and the appendix provides evidence that contradicts the main thesis and topic of the paper and calls into question all other results.

The main claim of "achieving the lowest loss among four methods in every setting" is overstated and at best misleading, contradicted by the paper's results buried in Appendix E: "the temperature-matched single head reaches 2.847 (tail-mean)—indistinguishable from MHAR’s 2.848". The abstract goes on to note that "Temperature-matched controls attribute most of the gain to flatter per-head softmax distributions, with a small residual consistent with per-subspace routing." Yet the actual paper text states "the residual increment on our corpus is a softmax-temperature effect rather than subspace independence". The abstract claim about the small residual is unsubstantiated; there are few to no experiments in support of it. This is a severe problem for a paper that is trying to demonstrate the benefit of using multiple heads. If anything, the paper implies the opposite: that softmax temperature matters but multiple heads do not.

The extremely low amount of pre-training tokens calls all related experiments into question. Appendix I attempts to address this, but is confounded by the softmax temperature issue. The unified peak learning rate of 5 × 10−4 is improper to use across such a range of models, and the seemingly excessive loss improvement demonstrated for all forms of attention residuals may be related to poor hyperparameters.

The mid-training experiment is written in an unclear manner, but it seems to imply that it compares a baseline non-attention-residual mid-training against a multi-head attention residual mid-training. If this is the case, then it does not demonstrate anything about multiple heads. For the mid-training experiment to demonstrate relevant result it would need, at a minimum, to include mid-training with single headed attention residuals, with appropriate testing for softmax temperature.

Questions for authors

Given the paper's demonstration of the importance of softmax temperature, why aren't there more experiments around that and why aren't all of the main experiments revised to account for that?

Why should the reader believe that headcount matters at all?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer WPDK

Thank you for the insightful review. Your comments helped us clarify the relationship between MHAR and softmax-logit scaling. Their empirical benefits substantially overlap, consistent with a shared effect of softening the depth-wise softmax. We will highlight this connection in the main-text ablation study.

W1 / Q1–Q2: Head count, depth softmax, and single-head controls

Single-head AR shares one depth distribution across all coordinates. MHAR uses one distribution per subspace. Scaled single-head AR retains the shared distribution and multiplies its logits by 1/81/\sqrt{8}.

Each MHAR head computes logits over d/Hd/H coordinates rather than dd. Under comparable coordinate-wise statistics and weak correlations, this reduces the typical logit scale by approximately 1/H1/\sqrt{H}. This provides a plausible explanation for why splitting into eight heads and explicitly scaling single-head logits can produce similar benefits.

Our new 533M results and the existing 1B temperature-matched control on FineWeb-Edu provide complementary evidence:

Model scaleSingle-head ARScaled single-head ARMHARMHAR − scaled AR ↓
533M3.2983.2723.277+0.005
1B3.2703.1523.129−0.023

The 533M and 1B results are three-seed final-checkpoint means. At 533M, both methods improve over ordinary AR and achieve close mean losses, with scaled AR lower by 0.005. At 1B, MHAR achieves a numerical advantage of 0.023 over scaled AR. The larger 1B web setting thus shows a possible additional benefit, but the gap is close to the reported noise floor. Different training and evaluation protocols prevent this comparison from establishing a scaling trend; confirmation requires matched conditions and additional seeds.

MHAR additionally allows different feature subspaces to select different mixtures of prior-layer outputs, without adding parameters relative to single-head AR. Explicit logit scaling retains one shared mixture across all coordinates. This flexibility offers a possible explanation for the 1B gap, although its causal contribution remains unresolved.

We thank the reviewer for highlighting this connection. We will bring these controls into the main-text ablation study and distinguish the shared temperature-related benefits from MHAR's additional routing flexibility.

W2: Training budgets and hyperparameters

We agree that training budget and learning-rate selection matter. The main 20K-update study uses 2.62B tokens at nominal 100M and 0.655B at larger settings. These are finite-budget comparisons under a shared peak learning rate of 5×10−45\times10^{-4}, which need not be optimal for every method.

Appendix I/Table 13 sweeps {1e-4, 5e-4, 1e-3} per method. The plain Transformer and MHAR prefer 1e-3, while larger single-head AR models prefer 5e-4. These web-data runs use randomly initialized routing queries and omit scaled AR, leaving the comparison with separately tuned scaled AR unresolved. Table 10 shows that MHAR's improvements over the plain Transformer persist at 7.0B/21.6B tokens, but missing scaled single-head controls prevent isolating the contribution of independent heads.

Our new comparison at approximately matched active training time (single seed) is:

ModelTraining tokensActive B200 GPU-hoursLoss ↓
533M Block MHAR655M2.2953.281
1.439B plain Transformer496M2.2953.348

Active training time excludes initialization, evaluation, and checkpoint I/O. This single-seed result supports a favorable performance–training-time tradeoff against one larger plain model under the tested recipes; it does not establish superiority across fully converged runs or independently optimized configurations.

W3: The 8B comparison

We have completed two runs in a preliminary single-seed 8B comparison (seed 42). H8 and scaled H1 (logits multiplied by 1/81/\sqrt{8}) use the same Marin-8B@phoenix start, blocks, gates, data order and training settings: 100.7M tokens from a new high-quality Nemotron-CC web subset, evaluated on 21.0M fixed held-out tokens.

RetrofitFinal loss ↓EMA loss ↓
Ordinary H1RunningRunning
Scaled H12.3002.327
H8 (MHAR)2.3002.327

Both perform nearly identically under this short budget. Ordinary H1 is still running. This pilot differs from the original 10B-token data and budget; it does not isolate the head contribution to the original downstream gains, which remain evidence for the combined retrofit.

In the revised abstract and conclusions, we will distinguish the shared temperature effect and subspace-specific routing capability from demonstrated loss gains, scope numerical advantages to the tested settings, and remove unsupported claims of independent-head benefits and universally lowest loss.

Thank you for giving this significant effort and providing illuminating new experimental evidence. The new experiments help demonstrate that the original temperature result was not merely a random aberration. From the current data in the first new experiment it is hard to tell how much variance/noise there is. Giving a range across seeds would be helpful. I also see that there is a new experiment coming in your response to another reviewer, and will be interested to see that result.

It is quite interesting to see the new regimes in which temperature scaling gave similar or sometimes even better loss results. This leaves open the resulting question: what temperature is optimal? There is some indirect preliminary evidence in the paper already regarding this based on tests of different headcounts, but it is confounded by the heads being separated.

The experiment showing improvement in loss using less compute time to train more tokens is a nice direction. But is your idea to reframe the paper's main conclusion to say that multi-head provides a significant compute advantage while matching the quality of temperature adjusted results? It could be quite challenging to undertake such a reworking even if it ends up being provable, and might be outside the scope of a single conference submission. Evidence that the new temperature is optimal would be important. Once temperature has been revealed as a main focus, it creates the problem that there may be an optimal temperature for which multiple heads is no longer beneficial relative to single-headed attention residuals. This would need testing. As this is a lot of effort, it may be more useful as proposed future directions than for the current submission. I would also need to see significant revision to the papers claims and direction for my score to be raised much.

We thank the reviewer for the follow-up. Two direct responses below, on the experiment we owed you and on the optimal-temperature question.

(1) The experiment we owed you: zero-initialized, scaled H1 at 20 TPP

it has now completed. Both arms use the zero-initialized routing query prescribed by AttnRes, both are trained to a 20-TPP budget from the same starting point under identical data order, optimizer and schedule, and the only difference is the head reshape. H1 uses softmax scale 0.176777; H8 uses scale 1.

Model scaleScaled single-head AR ↓MHAR (H=8) ↓MHAR − scaled AR
533M2.67862.6784−0.0002
1B2.47902.4687−0.0113

two seed avg at each scale. We want to be explicit about what this does and does not show:

  • At 533M the two are indistinguishable. −0.0002 is well inside the ±0.004 seed-level SD we measured above, so this is a null
  • At 1B, MHAR is ahead by 0.0113 nats — larger than the 533M seed-level noise.

So at the initialization, temperature and budget you specified, we do gound a single-seed 0.011-nat gap at 1B,which is consistent with a benefit that grows with width.

(2) What temperature is optimal?

The paper does contain indirect eve connected it to this questionexplicitly. Our head-count sweep is U-shaped with a flat optimum at H=4–8. Since each head computes logits over d/H coordinate typical logit scale by roughly 1/√H, so under that mapping the observed optimum corresponds to an effective logit scale of about 1/√4 = 0.50 to 1/√8 = 0.354 — the sweepdy a coarse temperature sweep, and it says the optimum is interior: both scale 1 (H=1, unscaled) and much softer settings are worse.

This also speaks to whether our single-head control was adequately tuned. The scaled H1 baseline uses 0.176777 ≈ 1/√32, which underonds to H≈32 — i.e. softer than theoptimum the H sweep indicates. Our temperature-matched single head is therefore not an under-softened straw baseline; if inferred optimum, and a single headtuned to ~1/√8 might be stronger still.

(3) On reframing and scope. We agree that the revision makes routing-rimary empirical finding, presents the head reshape as a zero-parameter route to a softened per-subspace softmax that matches a tuned single head at 533M and gain at 1B, and states the optimal temperature should be match with optimal head number.

Thank you, those results are useful to see. I agree that we cannot know from this whether the H1 baseline was scaled properly, and that your head-count sweep implies that 1/√4 to 1/√8 might be a stronger range. Also, even with this discrepancy, at 533M there is no benefit seen for headcount versus temperature change, so that implies that for your future experiments it would be informative to see if temperature scale changes based on model scale.

Unfortunately, this results in the 1B test giving us very little additional insight. The paper still does not have enough experimental evidence directed at temperature control to know whether separating the attention residuals into heads is more beneficial, or even possibly less beneficial, than temperature change alone.

Reviewer_8fs7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a multi-headed variant of Attention Residuals (AR) (Kimi Team, 2026), termed Multi Head Attention Residuals (MHAR). In a Transformer LM, a standard residual connection combines just sums the current layer with the previous layer; regular AR differs by instead creating a gate (a softmax) over all previous layers and adding in a weighted average of those previous layers. The proposed MHAR separates out these residuals into separate heads, (the number of heads in the residual does not need to match the number of attention heads; they are distinct). Then each residual head can have its own softmax over previous layers, so that its residual is customized for that head. Experiments include pre-training from scratch with models 100M-1B, and mid-training only experiments on an 8B model. The model gets lower validation loss, but at the cost of more compute time (only ~50% throughput of a standard Transformer at the 1B scale).

Critical strengths
  • Attention residuals are one of the key new architectural innovations in Kimi K3. This paper offers an alternative that seems to be essentially a drop in replacement for attention residuals with only a modest increase in compute requirements (i.e. it requires one softmax per, say, 8 heads instead of one softmax for a single head), yet that achieves improved validation perplexity.
Critical weaknesses
  • The MHAR has 0.55x throughput of a standard baseline transformer at the 1B scale. As well, it seems that the deeper the model gets, the slower it gets. E.g. the throughput goes from 0.88x at 100M to 0.71x at 350M to 0.55x at 1B. Although the paper spends a lot of time noting that it is identical in parameters and flops to standard AR, this seems like an important point worth considering: if you're willing to take a 0.55x hit in throughput relative to a baseline Transformer, what other changes could you have made instead of MHAR? For example, why not just use a larger model.
  • In general, this could be viewed as a paper about increasing test time compute, but if so, it doesn't meaningfully compare to any other papers in that vein besides AR.
  • The paper obtains validation loss reductions of 0.06-0.14 relative to baseline transformer, but it is not parameter matched to that transformer. Although it's only adding 0.02% of new parameters, the paper never actually quantifies whether that 0.02% of parameters is meaningful -- perhaps it is when the models are undertrained? This comes up later when performing mid training on an 8B model with and without MHAR: the baseline of "Plain CPT" and the "+MHAR" are not parameter matched, so it's difficult to judge whether the improvement is from MHAR or the the additional parameters. A better experiment would simply add in that 0.02% of parameters to the baseline transformer, even if you can't get the parameter count to match exactly giving the baseline transformer the advantage would then cement in that MHAR is the winner and not just more params.
  • I realize these experiments are quite expensive, but the main results in Table one are only for a single fixed seed. It would be nice to have (at least for a small model perhaps at a lower tokens per parameter level) the results averaged over multiple seeds.
Questions for authors
  • Why are the 3x larger budget results relegated to Appendix I?
  • One concern is that the Kimi models do not use Attention Residuals as you have done so here: they exclusively use Block Attention Residuals because it substantially reduces the communication overhead. Have you compared AR and MHAR in the blocked setting? Presumably this would actually make your experiments cheaper to run and more representative of how they are used at scale.

Suggestions:

  • Figure 1 is really unhelpful and does not differentiate the most importat part of this paer. The RHS could be interpreted as simply zooming in on the LHS (since the attention is multi-headed etither way). The figure shows /nothing/ about the multi-headed nature of the residual computation. Figure 2 is actually useful, maybe drop Figure 1.
1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer 8fs7

Thank you. We agree that quality gains need to justify their training cost.

W1–W2 Training cost

We agree that the loss improvement needs to justify the extra training time. At 1B, 0.55x baseline throughput means about 1.82x the time for the same tokens. Section 3.4 therefore also compares MHAR with a plain Transformer trained for longer at equal GPU time.

Nominal scaleMHAR minus plain loss, equal stepsEqual GPU time
100M-0.061-0.006
350M-0.149-0.080
1B-0.140-0.018

MHAR trains for 20K steps, versus 22,700/28,200/36,400 for plain with a stretched cosine schedule. These single-seed results favor MHAR at 350M, but the 100M/1B gaps are within the stated noise floor. A larger plain model at equal training cost remains untested.

Table 4 and Appendix G already include kernel optimization. Fusion raises throughput from 0.54/0.32/0.23x to 0.88/0.71/0.55x baseline at 100M/350M/1B. Appendix F fuses normalization, softmax and mixing with shared buffers and in-place gradient accumulation. Full routing still reads a growing history of layer outputs.

Our 8B block results appear below. Routing remains active during inference. Decode-time overhead is unmeasured.

Separate B200 measurements found different H1 and H8 runtimes, so we will correct the claim that they are identical.

W3 Parameter matching

We completed two enlarged-plain runs. FFN width 4096 to 4098 adds 147,456 active parameters (0.0277%), giving 533,270,528 versus MHAR's 533,223,424. Shared initialization, data, optimizer and token budget match the runs below. Final held-out losses are shown below.

SeedPlainEnlarged plainMHAR
423.3522153.3517843.265344
13.3703823.3714613.286515

MHAR remains better by 0.085693 +/- 0.001056 nats (paired mean +/- sample SD across two seeds). This supports a gain over plain beyond the added FFN capacity tested here. It does not isolate independent heads or resolve the gated 8B retrofit.

W4 Multiple seeds

The main-table corpus results use one seed. Appendix I/Table 14 reports three seeds on FineWeb-Edu at a common learning rate of 5e-4.

Nominal scaleMHAR minus plain loss, mean +/- sample SD
100M-0.045 +/- 0.007
350M-0.090 +/- 0.004
1B-0.071 +/- 0.002

These are paired differences within each seed. The runs use randomly initialized routing queries and omit scaled H1, so they do not provide error bars for Table 1's zero-initialized recipe.

All four models have completed three seeds at 533M on FineWeb-Edu, using 655M input tokens per run and final-checkpoint evaluation on 19.98M fixed held-out scored tokens.

SeedPlainH1Scaled H1H8
423.3522153.2897603.2639143.265344
13.3703823.2981853.2773203.286515
23.3678843.3047903.2753443.279407

The paired gap is +0.004896 +/- 0.003949 nats (mean +/- sample SD across training seeds). H8 is worse in all three runs. These results do not support an additional gain from independent heads. Shared initialization, frozen data, and routing/training settings are matched within each seed.

Q1 Longer-budget results

We will give the longer-budget results more prominence, distinguishing the 60K-step web runs from Table 10's 7.0B/21.6B-token comparisons. Both show that the gap over the plain baseline can persist with more training, but neither includes temperature-matched H1.

Q2 Block Attention Residuals

Our 8B experiment already implements block-aggregated delta MHAR, with four layers per block and at most nine routing sources (Section 3.3). Table 3 reports the following downstream accuracies for the schedule-matched runs.

TaskPlain CPTBlocked MHAR
MMLU0.6430.645
GPQA0.3150.346
GSM8K0.4700.502
MATH0.1910.191
HumanEval0.4090.415
MBPP0.3860.392

Table 5 reports routing time per microbatch for this 8B variant (forward+backward, bf16, one H100).

ImplementationRouting time (ms)
Compiled PyTorch651
Fused Triton323

Fusion raises end-to-end throughput by about 30% on 8 H100s within this blocked setup (Appendix F).

These results evaluate the combined gated delta/H8 retrofit. A matched Block AR versus Block MHAR comparison, including H1 and scaled H1 with the same blocks, sources, gates, data and schedule, remains outstanding.

We will also redraw Figure 1 using the same raw-output history for all methods, showing cumulative addition, one depth distribution for H1, and separate distributions for MHAR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.