- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies what should serve as the retrieval unit in Attention Residuals (AttnRes). Full AttnRes attends separately to earlier attention and MLP outputs, whereas Block AttnRes combines the two outputs within each Transformer block before retrieval. Across the reported settings, Block AttnRes consistently achieves lower validation loss while reducing measured GPU time per token. Shifted and random grouping controls further suggest that the gain depends on the structure of the retrieval basis rather than candidate count alone.
Critical strengths
S1. The comparison between retrieval from individual sublayers and retrieval from complete blocks isolates a design choice that is easy to overlook in learned residual routing. Treating a Transformer block as a coherent intermediate representation is intuitive and may be useful beyond this particular method.
S2. Block AttnRes consistently outperforms Full AttnRes across the reported model sizes and seeds while also reducing measured training cost. The grouping controls and optimization sweeps provide evidence that the result is not simply an artifact of fewer retrieval candidates or one training recipe.
Critical weaknesses
W1. The experiments convincingly show that final candidate-count reduction alone is insufficient. However, the more specific inference about block-boundary preservation rests on a single fixed local partition at each scale, without control-specific retuning, and only one control seed at 350M. Thus, the Full–Block ordering is well supported, whereas the identification of the relevant structural prior is somewhat less conclusive.
W2. The quality evidence is restricted to FineWeb-Edu validation loss for 124M- and 350M-parameter decoder-only models. The results support Block AttnRes in the tested regimes, but they do not yet establish that the ranking transfers to substantially larger models, other data distributions, or downstream task performance.
W3. This work appears to be a comprehensive experimental design, rather than proposing new algorithm. This point does not satisfy my academic standard, personally.
W4. The results in Table 2 show that the (Full-Block) gain is rather limited, ranging from 0.01 to 0.03 in nats of validation loss.
Questions for authors
Q1. How sensitive is the comparison between Block and the local control to the particular randomly sampled non-aligned partition? Evidence from additional partition draws, if available, would materially strengthen the claim that preserving complete block boundaries is the relevant structural prior.
2 comments
Thank the authors for the response. After reading the response, I believe my original scores remain appropriate. Hence I am inclined to keep them unchanged.
Thank you for the review. We have added partition replications, exposure controls, and downstream evaluations that address the main empirical concerns.
W1 / Q1. Sensitivity to the sampled partition
We expanded the original local non-aligned control from one to nine distinct partitions at 124M. Block outperforms all nine at training seed 9. Four of these partitions were also evaluated at seed 1337, and all four again favor Block.
Differences are in nats. These are nine distinct partitions and thirteen training runs, with each local control compared against Block at the same training seed.
At 350M, we also completed the second training seed for the original local partition. The local-minus-Block gaps are +0.01454 and +0.00510 nats. Thus, the advantage is not confined to the original 124M partition or a single 350M control seed.
We additionally introduced window controls that match candidate counts, partial-bucket source memberships, and release times at every reader. Aligned grouping performs best on both seeds:
We interpret these results as evidence that candidate composition contributes at fixed exposure.
W2 / W4. Downstream performance and the size of the gains
We evaluated nine existing checkpoints zero-shot on four downstream tasks, without task-specific tuning.
Values are percentages. HellaSwag uses normalized accuracy; the other tasks use accuracy. Bold marks the highest point estimate within each size/seed group.
At 350M, Block improves all four tasks over both Full and standard residuals in both seeds. At 124M, it improves three of four tasks over standard residuals; PIQA decreases by 0.76 points.
We agree that the validation-loss gains are modest. The 0.01–0.03-nat range highlighted by the reviewer corresponds to approximately 1–3% lower perplexity. In the tested AdamW settings, these gains accompany approximately 14% and 26% lower training GPU time per token than Full, at 124M and 350M respectively. The practical benefit is a small quality improvement together with lower cost, with downstream evidence beyond pretraining loss.
W3. Nature of the contribution
This is an empirical study. The contribution is controlled evidence about retrieval-unit design: which grouping choices matter, whether the result survives alternative partitions, and how it depends on training conditions.
The additional exploratory MoE/Muon runs illustrate the last point. With one training seed, Full−Block is +0.02179 nats for MoE+AdamW, −0.00217 for Dense+Muon, and +0.00296 for MoE+Muon. These results suggest that the value of grouping depends on optimization conditions.
We will foreground these findings and distinguish the supported design conclusions from the remaining mechanistic questions.