acceptodds
Under review as a conference paper at ICLR 2027

On the Effectiveness and Inductive Bias of Block Grouping in Attention Residuals

Abstract

Attention Residuals (AttnRes) make residual history addressable, raising a natural follow-up question: what should constitute a unit of retrieval? Full AttnRes exposes every sublayer output independently, whereas Block AttnRes groups outputs and forces them to share a readout coefficient. We find that this restriction can improve both learning and efficiency. On FineWeb-Edu, Block outperforms Full for all three seeds at 124M parameters and both seeds at 350M under our standard recipe, reversing the original AttnRes ordering. To separate candidate count from candidate construction, we compare Block with a local non-aligned control matching its final group count and sizes, with similar per-reader candidate counts. Block remains better across the reported comparisons, showing that candidate-count reduction alone is insufficient to explain the gain. Learning-rate sweeps at both scales and a fourfold training-budget range at 124M show that the Block-Full ordering survives retuning, even when retuning restores Full's advantage over standard residuals. With a fused Triton implementation, Block uses approximately 14% and 26% less GPU time per token than Full at 124M and 350M. These results show that residual-source grouping is not merely an efficiency choice but a meaningful inductive bias: how residual history is partitioned can matter as much as how finely it is accessed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention Residuals, Transformers, inductive bias, residual connections, attention, pretraining, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_XabeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies what should serve as the retrieval unit in Attention Residuals (AttnRes). Full AttnRes attends separately to earlier attention and MLP outputs, whereas Block AttnRes combines the two outputs within each Transformer block before retrieval. Across the reported settings, Block AttnRes consistently achieves lower validation loss while reducing measured GPU time per token. Shifted and random grouping controls further suggest that the gain depends on the structure of the retrieval basis rather than candidate count alone.

Critical strengths

S1. The comparison between retrieval from individual sublayers and retrieval from complete blocks isolates a design choice that is easy to overlook in learned residual routing. Treating a Transformer block as a coherent intermediate representation is intuitive and may be useful beyond this particular method.

S2. Block AttnRes consistently outperforms Full AttnRes across the reported model sizes and seeds while also reducing measured training cost. The grouping controls and optimization sweeps provide evidence that the result is not simply an artifact of fewer retrieval candidates or one training recipe.

Critical weaknesses

W1. The experiments convincingly show that final candidate-count reduction alone is insufficient. However, the more specific inference about block-boundary preservation rests on a single fixed local partition at each scale, without control-specific retuning, and only one control seed at 350M. Thus, the Full–Block ordering is well supported, whereas the identification of the relevant structural prior is somewhat less conclusive.

W2. The quality evidence is restricted to FineWeb-Edu validation loss for 124M- and 350M-parameter decoder-only models. The results support Block AttnRes in the tested regimes, but they do not yet establish that the ranking transfers to substantially larger models, other data distributions, or downstream task performance.

W3. This work appears to be a comprehensive experimental design, rather than proposing new algorithm. This point does not satisfy my academic standard, personally.

W4. The results in Table 2 show that the (Full-Block) gain is rather limited, ranging from 0.01 to 0.03 in nats of validation loss.

Questions for authors

Q1. How sensitive is the comparison between Block and the local control to the particular randomly sampled non-aligned partition? Evidence from additional partition draws, if available, would materially strengthen the claim that preserving complete block boundaries is the relevant structural prior.

2 comments

Thank you for the review. We have added partition replications, exposure controls, and downstream evaluations that address the main empirical concerns.

W1 / Q1. Sensitivity to the sampled partition

We expanded the original local non-aligned control from one to nine distinct partitions at 124M. Block outperforms all nine at training seed 9. Four of these partitions were also evaluated at seed 1337, and all four again favor Block.

Training seedPartitions evaluatedComparisons favoring BlockMean local − Block CERange
999/9+0.02284+0.01032 to +0.03677
133744/4+0.02108+0.00723 to +0.03060

Differences are in nats. These are nine distinct partitions and thirteen training runs, with each local control compared against Block at the same training seed.

At 350M, we also completed the second training seed for the original local partition. The local-minus-Block gaps are +0.01454 and +0.00510 nats. Thus, the advantage is not confined to the original 124M partition or a single 350M control seed.

We additionally introduced window controls that match candidate counts, partial-bucket source memberships, and release times at every reader. Aligned grouping performs best on both seeds:

Training seedCrossed − aligned CESame-type − aligned CE
9+0.00993+0.02004
1337+0.00836+0.02723

We interpret these results as evidence that candidate composition contributes at fixed exposure.

W2 / W4. Downstream performance and the size of the gains

We evaluated nine existing checkpoints zero-shot on four downstream tasks, without task-specific tuning.

SizeSeedResidualHellaSwagPIQAARC-easyLAMBADA
124M9Standard29.1162.1348.3620.73
124M9Full29.5261.9247.9420.26
124M9Block30.2361.3748.5721.73
350M1337Standard34.8864.0952.4029.32
350M1337Full35.5164.4755.0928.86
350M1337Block35.7465.5155.7230.93
350M8Standard35.2464.7455.9828.49
350M8Full35.6464.3655.6030.06
350M8Block36.3465.1856.7831.85

Values are percentages. HellaSwag uses normalized accuracy; the other tasks use accuracy. Bold marks the highest point estimate within each size/seed group.

At 350M, Block improves all four tasks over both Full and standard residuals in both seeds. At 124M, it improves three of four tasks over standard residuals; PIQA decreases by 0.76 points.

We agree that the validation-loss gains are modest. The 0.01–0.03-nat range highlighted by the reviewer corresponds to approximately 1–3% lower perplexity. In the tested AdamW settings, these gains accompany approximately 14% and 26% lower training GPU time per token than Full, at 124M and 350M respectively. The practical benefit is a small quality improvement together with lower cost, with downstream evidence beyond pretraining loss.

W3. Nature of the contribution

This is an empirical study. The contribution is controlled evidence about retrieval-unit design: which grouping choices matter, whether the result survives alternative partitions, and how it depends on training conditions.

The additional exploratory MoE/Muon runs illustrate the last point. With one training seed, Full−Block is +0.02179 nats for MoE+AdamW, −0.00217 for Dense+Muon, and +0.00296 for MoE+Muon. These results suggest that the value of grouping depends on optimization conditions.

We will foreground these findings and distinguish the supported design conclusions from the remaining mechanistic questions.

Thank the authors for the response. After reading the response, I believe my original scores remain appropriate. Hence I am inclined to keep them unchanged.

Reviewer_Sbd4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how a language model with residual attention model uses information from its earlier layers: does the model learn better when every earlier layer can be read independently, or does it work better to first group intermediate outputs of earlier layers?

The evaluation uses two small language models and a collection of web text data.

The paper compare three designs:

  • "standard attention" maintains a running sum of the output of each previous block.
  • "full" lets later layers choose how much weight to give each earlier output.
  • "block” first combines outputs from the same section of the model.

The main insights are:

  • "block" makes better predictions than "full" and is computationally less expensive (also reduces memory footprint).
  • Grouping outputs from the same section works better than mixing outputs from neighboring sections.
  • These findings are robust to various training settings.
Critical strengths
  • The paper tests why grouping helps. The authors compare several ways of grouping earlier outputs, including groups with the same number and size. This gives evidence that choosing which outputs belong together matters.

  • The paper checks that the result holds under different training choices. The authors repeat experiments, use two model sizes, change training settings, and train for longer. Block’s lead over Full remains after these checks.

  • The paper measures training cost as well as prediction quality. The paper shows that "block" makes better predictions than "full" while taking less training time.

Critical weaknesses

The new contribution is fairly thin. Kimi Team et al. (2026) already showed that "block" keeps most of "full"'s prediction quality while reducing cost. Su (2026) also found a "block" setting that beats "full". This paper mainly adds tests of which outputs should be grouped and whether the results hold under different training choices.

The tests cover a limited range. The paper uses one collection of web text and two small models. It does not test larger models, other kinds of text, or tasks such as answering questions.

The reason grouping helps remains unclear. Changing the groups also changes what each layer can read while those groups are being built.

There is no comparison using equal computing budgets: the standard Transformer baseline uses less compute/memory. The paper does not test whether giving it the same computing budget would close the gap.

The paper measures training time, but not memory use or speed when generating text.

Questions for authors

How well does the result carry over to other settings? Could you check the main finding on a larger model or a different collection of text? A few tests of tasks such as answering questions would also help show whether the gain holds.

Can you separate the effect of grouping from the other differences between your tests? The groups differ in what each layer can read before a group is complete. Could these differences explain part of "block"’s advantage, and could an extra test help rule this out?

Does "block" still beat the standard Transformer with the same computing budget?

What are the memory and text-generation costs? Could you report memory use during training and inference? (for the standard transformer, "full" and "block").

4 comments

Thank you for recognizing the controlled comparisons. We address the overlapping comments in four parts.

1. Contribution and the grouping–exposure confound

We agree that Block itself, and the observation that it can outperform Full, are not new. Our contribution is controlled evidence on which outputs should be grouped and under what training conditions this helps.

We added window controls that match candidate counts, partial-bucket source memberships, and release times at every reader. Aligned grouping performs best on both seeds:

Training seedCrossed − aligned CESame-type − aligned CE
9+0.00993+0.02004
1337+0.00836+0.02723

Differences are in nats; positive values favor aligned grouping. The crossed gaps fall below our prespecified 0.015-nat criterion, so we interpret these results as evidence that composition matters at fixed exposure, without claiming a complete explanation of the original Block–Full gap.

The result is also robust to partition choice: Block beats all nine sampled local partitions at the primary training seed and all four partitions repeated at a second seed. The mean local-minus-Block gaps are 0.02284 and 0.02108 nats, respectively.

A single-seed MoE/Muon extension further suggests optimizer dependence: Full−Block is +0.02179 with MoE+AdamW, −0.00217 with Dense+Muon, and +0.00296 with MoE+Muon. Thus, the Block advantage persists with MoE under AdamW but becomes small under Muon in these exploratory runs.

2. Downstream performance and generality

We evaluated nine existing checkpoints zero-shot on HellaSwag, PIQA, ARC-easy, and LAMBADA-openai, without task-specific tuning.

SizeSeedResidualHellaSwagPIQAARC-easyLAMBADA
124M9Standard29.1162.1348.3620.73
124M9Full29.5261.9247.9420.26
124M9Block30.2361.3748.5721.73
350M1337Standard34.8864.0952.4029.32
350M1337Full35.5164.4755.0928.86
350M1337Block35.7465.5155.7230.93
350M8Standard35.2464.7455.9828.49
350M8Full35.6464.3655.6030.06
350M8Block36.3465.1856.7831.85

All values are percentages. HellaSwag uses normalized accuracy; the other tasks use accuracy. Bold indicates the highest point estimate within each size/seed group. At 350M, Block improves all four tasks over both standard residuals and Full in both seeds. At 124M, it improves three of four tasks over standard residuals; PIQA decreases by 0.76 percentage points.

These results provide downstream support beyond pretraining loss.

3. Comparison at matched training GPU-hours

We completed two new 124M standard-residual runs with full cosine schedules of 6,140 and 6,184 updates, matching the historical 4,800-update Block runs. Actual training-cost mismatch is below 0.05% per pair, excluding validation and checkpointing.

SeedStandard CEBlock CEStandard − Block
93.093603.06081+0.03279
13373.076673.07808−0.00141

Block is better on one seed and slightly worse on the other, with a mean advantage of 0.01569 nats. Both methods use AdamW with peak LR 0.0024.

At 350M, existing trajectory comparisons do not show an advantage for Block, but truncate its learning-rate schedule. We do not have dedicated matched-budget endpoints at that scale and do not claim general equal-compute superiority.

4. Memory and generation costs

Our measured cost advantage is lower training GPU time for Block than Full. We have not collected a complete standard/Full/Block memory comparison or benchmarked autoregressive decoding. We will make this scope explicit and avoid extending the training-time result to peak-memory savings or inference-speed improvements.

Hello, thank you for the rebuttal.

"1. Contribution and the grouping–exposure confound": the differences in nats seem quite small, are you able to give confidence intervals and tell if they are statistically significant?

"2. Downstream performance and generality" addresses my question on question answering, and it seems "block" is generally more accurate indeed.

"3. Comparison at matched training GPU-hours": the result appears to be non conclusive?

Thank you for the follow-up and for acknowledging the downstream results.

1. Confidence intervals for the exposure-matched comparisons

Using each paired training seed as one observation, we obtain the following conventional two-sided 95% paired-t intervals:

ComparisonSeed 9Seed 1337Mean difference95% paired-t interval
Crossed − aligned+0.00993+0.00836+0.00915[−0.00083, +0.01912]
Same-type − aligned+0.02004+0.02723+0.02364[−0.02204, +0.06931]

All differences are in validation CE, measured in nats; positive values favor aligned grouping.

These intervals are calculated across training seeds on the fixed validation set, with two paired observations and one degree of freedom. They assume normally distributed seed-level differences. They should therefore be treated as exploratory uncertainty estimates. The observed ordering is consistent across the two seeds, and the experiment controls the exposure differences, but the evidence for the structural interpretation remains suggestive. We will state this distinction explicitly.

3. Comparison at matched training GPU-hours

Their mean is +0.01569, but rigorously these results do not establish a consistent winner. We will explicitly state that equal-GPU-hour superiority over standard residuals remains relatively unestablished. Our training-cost claim is supported by the separate comparison with Full: in the tested AdamW settings, Block achieves better quality with lower GPU time per token. We will keep these two comparisons clearly distinguished.

Hello, the fact that the confidence interval cross zero makes them non statistically significant.

Reviewer_7wKUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares Full and Block Attention Residuals in 124M and 350M language models on FineWeb-Edu. Block consistently improves validation loss while reducing training cost relative to Full. Grouping controls, learning-rate sweeps, and longer-training experiments investigate the roles of candidate construction and optimization.

Critical strengths
  1. The comparison is carefully controlled, with paired initialization and token streams, multiple seeds, shared kernels, and a normalization-epsilon check. Block's advantage survives learning-rate retuning, making the empirical ranking credible within the tested settings.

  2. The local and shifted grouping controls provide useful evidence beyond a simple Full-versus-Block comparison. The finding that retuning restores Full's advantage over standard residuals also illustrates why fixed-recipe budget trends can be misleading.

Critical weaknesses
  1. The alignment interpretation remains underdetermined. Intermediate candidate exposure differs, only one local random partition is tested per scale, and the controls are not individually retuned (Section 4.3). These acknowledged limitations leave partition choice and optimization as alternative explanations for part of the gap, weakening the central structural interpretation.

  2. Practical significance is less established than the Block-versus-Full ranking. Evaluation covers one corpus and two small model scales, without downstream validation. Block also costs about 28 percent and 58 percent more GPU time per token than standard residuals at the two scales (Table S2), so its advantage over that baseline at equal training compute remains unknown.

Overall, I lean toward rejection: the empirical ranking is credible, but the structural interpretation and practical benefit remain insufficiently established.

Questions for authors
  1. Does Block's advantage over local non-aligned grouping persist across independently sampled partitions and separately tuned learning rates? A small replication would materially strengthen the structural conclusion.

  2. How does Block compare with standard residuals at matched GPU-hours, accounting for Table S2's overhead? This would clarify the practical value of the reported loss improvements.

1 comment

We thank the reviewer for recognizing the careful controls and the robustness of the Block–Full comparison. We have added experiments addressing intermediate candidate exposure, partition replication, downstream performance, and training cost. We report these results below.

W1. Structural interpretation and intermediate candidate exposure

We added a control specifically targeting the exposure confound. Each two-block window exposes a single partial sum until completion, then releases two completed candidates. The aligned, crossed, and same-type variants share candidate counts, partial-bucket source memberships, and release schedules at every reader, while varying how completed outputs are grouped.

Aligned grouping achieves the lowest validation loss on both training seeds:

Training seedCrossed − aligned CESame-type − aligned CE
9+0.00993+0.02004
1337+0.00836+0.02723

The window-aligned variant also remains close to ordinary Block: their losses differ by −0.00445 and +0.00036 nats on the two seeds.

These comparisons provide evidence that candidate composition contributes beyond differences in intermediate exposure. The effect is smaller for crossed than for same-type grouping. We interpret the experiment as evidence for a composition effect under the tested recipe, without attributing the entire original Block–Full gap to alignment. We will revise the structural claim accordingly. The additional partition replications below further test whether the advantage depends on the particular local partition originally chosen.

W2. Downstream relevance and the scope of the study

We have added zero-shot evaluation on HellaSwag, PIQA, ARC-easy, and LAMBADA-openai, using nine existing checkpoints without task-specific tuning: standard, Full, and Block at 124M for one seed, and at 350M for two seeds.

At 350M, Block improves all four tasks over standard residuals in both seeds:

Task and metricBlock − standard, seed 1337Block − standard, seed 8
HellaSwag, normalized accuracy+0.86+1.10
PIQA, accuracy+1.42+0.44
ARC-easy, accuracy+3.32+0.80
LAMBADA-openai, accuracy+1.61+3.36

Values are percentage-point differences. Block also outperforms Full on all four tasks in both 350M seeds. At 124M, Block improves three of the four tasks over standard residuals; PIQA decreases by 0.76 points.

These results show that the validation-loss improvements extend to downstream tasks within the studied settings.

Q1. Replication across partitions and separately tuned learning rates

We expanded the original local non-aligned control to nine distinct, prespecified partitions at 124M. At training seed 9, Block outperforms all nine, with a mean local-minus-Block gap of 0.02284 nats and a range of 0.01032–0.03677. Four of these partitions were also evaluated at seed 1337; all four again favor Block, with a mean gap of 0.02108 and a minimum gap of 0.00723.

Thus, the advantage is not specific to the original partition realization. At 350M, replication of the original local partition at a second training seed also preserves the ordering: local-minus-Block gaps are 0.01454 and 0.00510 nats.

There is additional evidence across recipes: the original local control remains worse than Block under the sample-B recipe with peak LR 0.0024, by 0.01786 and 0.02185 nats on two seeds.

Q2. Comparison with standard residuals at matched GPU-hours

We completed a direct comparison at 124M by training two fresh sample-B standard-residual baselines to match the training GPU-hours of the corresponding Block runs.

SeedStandard CEBlock CEStandard − Block
93.093603.06081+0.03279
13373.076673.07808−0.00141

The standard baselines completed full cosine schedules of 6,140 and 6,184 updates, with these horizons fixed before training to match the historical Block costs. Block used 4,800 updates. The actual training-cost mismatch is below 0.05% in each pair, excluding validation and checkpointing.

Block improves by 0.03279 nats on seed 9 and is 0.00141 nats worse on seed 1337, giving a mean advantage of 0.01569 nats. The result is therefore favorable on average. Both methods use AdamW with peak LR 0.0024; the baseline LR was not retuned for the longer schedules. Timing comparisons also use historical Block measurements.

At 350M, retrospective comparisons of existing trajectories do not show a compute-matched advantage for Block. These readouts occur before Block completes its learning-rate schedule and are not new matched-budget training endpoints. We will report them separately from the completed 124M comparison and avoid a general claim of equal-compute superiority over standard residuals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.