- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies Multi-Head Attention Residuals, applying the multi-head aspect of regular token attention to depth-wise selection. This variation uses multiple queries, each seeing only a subset of the total key and each producing its own softmax distribution over previous sublayer outputs. This allows different subsets of the input to a sublayer to “see” different depth mixtures of previous sublayer outputs. This method adds no extra parameters. The single-head formulation is recovered at . The paper evaluates 100M, 350M, and 1B models trained from scratch on two corpora, and an 8B model adapted through a gated delta-routing retrofit. It reports improvements over a standard pre-norm Transformer, explores routing-head ablations and temperature, and provides fused forward and backward Triton kernels. The temperature controls indicate that much of the observed improvement over ordinary single-head routing can be recovered by changing the single-head logit scale.
Critical strengths
- Fused kernel
I appreciate the paper providing fused forward and backward routing kernels with a shared source buffer, reducing repeated source materialization and combining several memory-bound operations. The reported improvements over eager and compiled reference implementations make this a potentially useful contribution. It would be even more informative if the authors could compare their kernel at with popular public implementations, e.g. the AttnRes kernel in Flash Linear Attention.
- Empirical reconsideration of routing-head count and temperature
The paper revisits multi-head depth routing across several head counts, scales, and data settings, and includes a particularly informative temperature-matched single-head control. The finding that logit scaling can recover much of the single-head deficit is useful for understanding the mechanism and configuring attention residuals.
Critical weaknesses
The paper does not convincingly establish a consistent benefit from independent routing heads beyond the temperature effect. At 350M, the temperature-matched single-head model achieves 2.847 compared with 2.848 for MHAR, while the remaining 1B web-data advantage is single-seeded and near the stated noise floor. This makes it difficult to determine how much of the improvement is actually due to independent subspace routing.
Despite these qualifications, Section 4 attributes the head-count curve to mismatches between subspace preferences and KV-group granularity. Dense Q/K/V projections do not inherently assign particular residual-coordinate slices to particular KV groups, and the reported experiments do not establish this mechanism.
The same attribution problem applies to the 8B result: the comparison is between plain continued pretraining and a combined gated delta-routing/MHAR retrofit, without an otherwise identical delta control. The current 8B result therefore supports the combined retrofit but does not isolate the contribution of multiple heads. Likewise, the longer-budget comparison in Table 10 establishes the persistence of a benefit over the plain Transformer, but does not isolate the contribution of multiple heads.
This matters because the paper’s additional contribution should be demonstrated against appropriately configured single-head routing. The current evidence supports the importance of routing temperature more strongly than it supports an independent benefit from multiple depth distributions.
Questions for authors
1. Comparators, mechanism, and contribution
(a) Temperature-matched comparison.
At 350M, scaling the single-head logits already matches MHAR: 2.847 versus 2.848. The remaining 1B web-data advantage is also from a single seed and near the noise floor. What do independent heads add once initialization, normalization, logit scale, optimizer settings, and implementation are matched?
(b) Subspace disagreement and the head-count optimum.
Figure 4 shows that the heads learn different distributions, but what shows that these differences improve loss? Does the U-shaped head-count curve remain after controlling logit scale? I also don’t see what establishes the KV-group explanation, since dense projections use all residual coordinates and –8 is preferred across several KV counts. If this is still a hypothesis, please say so consistently.
(c) Isolating the 8B multi-head contribution.
Why not include the same delta retrofit with , alongside plain CPT, keeping the sources, gates, initialization, optimizer, data order, and schedule fixed? Ideally, this would also include a temperature-matched control. Without it, how much of the gain can be attributed to multiple heads rather than the combined retrofit?
(d) Relationship to prior work.
Please also discuss Low-Rank Attention Residuals and, if possible, include a comparison. This seems directly relevant to whether the benefit comes from smaller routing keys or from allowing different subspaces to choose different depth distributions.
2. Validation, budgets, downstream evaluation, and scaling
(a) Main and longer training budgets.
Why train the nominal 100M model on approximately 2.6B tokens but give each larger model only 0.66B? I see that you added longer runs in §3.4/Table 10, but why not include ordinary and temperature-matched single-head routing there? Those comparisons would show whether the additional multi-head benefit survives the longer budget.
(b) Primary validation and tail means.
Why average eleven evaluations of changing checkpoints on reshuffled training data instead of evaluating a fixed checkpoint on a large held-out set? Table 11 helps, but can it also include the temperature-matched single-head control with paired uncertainty estimates? Please clarify whether the main number measures final-checkpoint performance or average performance near the end of training.
(c) Downstream sample sizes and comparisons.
Why only 200 HellaSwag examples and 500 examples per Table 12 task? The WikiText-2/LAMBADA comparisons with single-head routing are already mixed: MHAR is better in three, worse in two, and tied in one at the reported precision. Can you evaluate larger fixed splits, explain how examples were selected, and report paired uncertainty?
(d) The 7B result and scaling.
The 7B result is an important limitation: the reported improvement is only 0.010 nats, within noise. Please give it a complete table row with the configuration, training budget, comparators, and uncertainty. What does this result, together with the non-monotonic gains, imply for the scaling claim? The 8B retrofit uses a different training regime, so it does not directly resolve this question.
3. Memory
(a) Peak-memory measurements.
The peak memory increase from 41.5 to 42.0 GB at 100M is surprisingly small for Attention Residuals on the full setting. One would expect around a 2x increase. Can you provide the Table 8 benchmark, runtime settings, and an allocation trace if possible?
4. Presentation and conclusions
(a) Figure 2.
The zeros in Figure 2 can give the impression that standard residuals cannot access earlier outputs, even though those contributions are already in the latest cumulative state. Could you show all methods over the same raw-output history, with weights of one for standard residuals, or make the change of basis explicit?
(b) Scope of the conclusions.
The conclusion reads more broadly than the results support. Please make clear which “lowest loss” claims refer to the selected configuration, and separate gains over the plain Transformer from gains over ordinary or temperature-matched single-head routing. Table 11 also shows a tie at 100M at the reported precision. The observed head-count optimum and the proposed KV/subspace explanation should be presented with different levels of certainty.
6 comments
Thank you for adding the matched controls and final-checkpoint held-out evaluation. These meaningfully address my evaluation concerns and provide useful evidence about routing temperature. The fused kernels also remain a strength.
However, my central concern remains unresolved: the additional benefit of independent routing heads over appropriately configured single-head routing has not been established. In the new 533M comparison, scaled single-head routing performs better in all three matched seed pairs, with MHAR’s loss higher by approximately 0.0049 nats on average. I appreciate the proposed qualifications to the mechanism and scaling claims. Nevertheless, the current evidence supports the importance of routing temperature more clearly than an additional benefit from independent depth distributions. I therefore maintain my score of 2 (weak rejection).
Rebuttal to Reviewer 6aC6
Thank you for the follow-up. We agree that routing temperature explains a large part of the improvement. We have now run additional softmax-scale experiments for both methods, and they reinforce your point: at 533M, a well-configured single head remains a strong baseline.
We would like to add a positive result at 1B that we would like to highlight. Our earlier 533M comparison and the existing 1B control are summarized below:
| Model scale | Single-head AR ↓ | Scaled single-head AR ↓ | MHAR ↓ | MHAR − scaled AR ↓ |
|---|---|---|---|---|
| 533M | 3.298 | 3.272 | 3.277 | +0.005 |
| 1B | 3.270 | 3.152 | 3.129 | −0.023 |
At 533M, scaled single-head routing does better. At 1B, MHAR has a 0.023-nat advantage. This is encouraging evidence that multiple heads may offer more benefit in larger models,.
One possible explanation is that scaling a single head makes its routing softer, but all features still share the same weights over earlier layers. MHAR lets different groups of features use different layer weights. That freedom may become more useful as models get wider.
We appreciate the reviewer’s insight into the connection with softmax scaling. We will make this connection more prominent in the ablation study, add the new scale-sweep results, and clearly separate what the experiments show from what still needs to be tested.
Rebuttal to Reviewer 6aC6
Thank you for the review. Our new controls clarify gains over plain Transformers and the role of temperature.
Q1(a): Temperature-matched controls
AR uses one depth distribution; scaled AR scales logits by 1/8; MHAR uses eight, with no extra parameters over AR. New 533M FineWeb-Edu runs match initialization, normalization, optimizer, data and training settings within each seed, using 655M tokens. Final held-out losses:
MHAR minus scaled AR is +0.004896 ± 0.003949 nats (paired mean ± sample SD across seeds): scaled AR is better in all three pairs. New single-seed block-routing losses are 3.282414/3.268976/3.280565 for AR/scaled AR/MHAR, with the same ordering.
Appendix E's 1B web losses are 3.270/3.152/3.129 for AR/scaled AR/MHAR: single-seed tail means, unlike our new final-checkpoint results. The 0.023 MHAR advantage is near the noise floor and does not establish a consistent head benefit or scaling trend.
An enlarged-plain control adds 147,456 FFN parameters (4096→4098), exceeding MHAR's parameter count. Its losses at seeds 42/1 are 3.351784/3.371461. MHAR improves by 0.085693 ± 0.001056 nats (paired mean ± sample SD). This supports gains beyond that added capacity, without isolating independent heads.
Q1(b): Head-count curve and mechanism
The H sweep changes logit scale; distinct distributions alone do not establish a loss benefit. A scale-controlled sweep remains missing. Dense projections do not tie residual slices to KV groups. We will label the KV-group explanation a hypothesis and remove the probe's unsupported “upper-bound” interpretation.
Q1(c–d): 8B attribution and related work
The 8B result evaluates the combined gated delta/MHAR retrofit. Ordinary/scaled H1 controls with matched checkpoint, sources, gates, initialization and training remain incomplete; from-scratch block results cannot replace them.
We will discuss Low-Rank Attention Residuals and distinguish reduced routing-key width from separate subspace distributions. A matched comparison remains outstanding.
Q2(a): Training budgets and compute
At 20K updates, batch×length 64×2048 versus 32×1024 explains 2.62B versus 0.655B tokens; budgets match within each scale. Table 10's MHAR−plain gaps are −0.119/−0.081 at 7.0B/21.6B tokens. The 60K-step web runs give −0.056/−0.062 and include H1. Both omit scaled H1, leaving longer-budget head attribution unresolved.
Approximately time-matched comparison (single seed):
Time excludes initialization, evaluation and checkpoint I/O. This supports the tradeoff under these recipes; convergence and separately tuned comparisons remain untested.
Q2(b): Validation and uncertainty
Table 1 averages eleven changing checkpoints on reshuffled samples without guaranteed held-out separation. Table 11 uses 400 fixed document-disjoint windows but omits scaled H1. New runs exclude held-out document IDs and evaluate final checkpoints on 19.98M identical scored tokens. Window/block bootstrap intervals favor scaled H1 in all three pairs, conditional on checkpoints, separately from seed variation. Blocks are not document clusters; content deduplication is unverified.
Q2(c–d): Downstream and scaling
The 200/500-example evaluations have limited resolution; WikiText-2/LAMBADA gives three wins, two losses and one tie against H1. We will document selection and narrow claims; larger splits with paired uncertainty remain missing. The 7B gap (−0.010 nats) is within noise. A verified row requires the original records; the different 8B regime cannot resolve this. We will remove the broad scaling claim.
Q3(a): Memory and kernels
Fusion stores sources once and recomputes normalization in backward, avoiding repeated stacks. Total peak includes weights, optimizer and other activations. New peak allocated memory (GB):
Single B200; batch×length 8×2048/4×1024/2×1024; bf16; TF32 off; no compilation/activation checkpointing; 30 warmup + 90 measured training steps. These do not verify H100's 41.5→42.0 GB; its allocation trace and FLA H1 comparison are missing. H1/H8 runtimes differ.
Q4(a–b): Figures and conclusions
We will use the same raw-output history: unit weights for standard residuals, one shared distribution for H1, one per slice for MHAR. We will include scaled H1 and the 100M tie, scope “lowest loss” to tested configurations/comparators, and separate the observed H preference from the KV hypothesis.