- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds on Attention Residuals (AttnRes), where each layer's input is a softmax-weighted mixture of earlier sources that replaces the residual stream (Section 2.1, Eq. 1). I really like the idea to bring back the original residual stream on top of AttnRes which is quite a new method and gives the flexibility for the model. The authors keep the standard residual stream and add a routed mixture of earlier sublayer deltas on top of it (Delta Attention Residuals; Eq. 2), plus a coarser block-level version similar with the original paper, Delta Block (Section 2.3, Appendix C). They report 1.7-8.2% lower validation perplexity than a standard residual across models from 220M to 7.6B parameters (Abstract, Tables 1 and 3), and say AttnRes degrades at the 1044M and 7.6B scales (Sections 3.2 and 3.3). An ablation over the source type (cumulative vs. delta) and the update rule (replacement vs. additive) and some routing statistics are used to explain why (Section 4, Table 6, Appendices D and E).
Critical strengths
- The design is simple and clear. It is one change to AttnRes: keep the stream and add the routed mixture, reusing the same router and sources (Sections 2.3 and 3.2, Table 7).
- The design question is well posed. The ablation over source type and update rule is parameter-matched (Appendix D) and repeated on a LLaMA-style model (Appendix F), and it reports cells that go against the paper's own story (for example cumulative + replacement beating delta + replacement at 220M, 37.05 vs 37.90; Table 6).
- Using deltas under an additive update is a principled choice, since cumulative states already contain the stream. The R2 and cosine statistics give some backing for this (Sections 2.2 and 4.1, Table 8).
Critical weaknesses
Weakness
-
The explanation for why replacement hurts is not tested at the scales where it hurts. AttnRes beats the baseline in validation perplexity at 220M (37.39 vs 38.71) and 533M (31.75 vs 32.00), but loses at 1044M (31.76 vs 29.70) and 7.6B (18.58 vs 17.43) (Tables 1 and 3). The paper attributes this to losing the identity path (Sections 3.2 and 4.2) and says the controlled ablation in Section 4.2 separates the effects of the source type and the update rule (Section 3.2). But that ablation (Table 6, Appendix D) only runs at 220M and 533M with a single seed, and none of its cells is AttnRes or Full AttnRes (Appendix D). The claim that Full AttnRes degrades most rests on one scale (33.36 vs 31.76; Table 1, Section 4.2). Also, the AttnRes baseline is a reimplementation that uses the same hyperparameters as the other methods, with 4 blocks and no tuning (Sections 3.1 and 3.2).
-
The headline gains are mostly against the standard residual, and the 7.6B result is Delta Block only. The 1.7-8.2% range is Delta Block's (1044M in Table 1 and 7.6B in Table 3), and Delta AttnRes was not run at 7.6B (Abstract, Section 3.3). Against AttnRes the gain for Delta AttnRes is about 1.5-2.2% at 220M/533M (Table 1) with a single seed (Appendices D and H). The large gaps only appear where AttnRes underperforms the baseline (Tables 1 and 3).
-
Hyperparameters and training budget are not justified. One learning rate is used (6e-4, and 3e-4 at 8B) with no tuning (Section 3.1). Batch size and sequence length change with scale (32x1024 vs 64x2048), so the models see about 0.33B vs about 1.3B tokens (10K steps each, computed from Section 3.1; Appendix H gives about 0.3B for the 220M-1B runs), and the scale trends are confounded with tokens. All runs use 10K steps, and the authors say they have not checked that the orderings hold at convergence-scale token budgets (Appendix H).
-
The redundancy evidence is weaker than it sounds. The 0.93-0.97 cosine and R2 of 0.99 are depth-0.9 values (at depth 0.1 they are 0.64-0.67 and 0.45-0.46; Table 8, Appendix E), and are partly true by construction (Section 2.2 notes that adjacent cumulative states share a growing common prefix). The delta |cos| of at most 0.06 is close to a random baseline (about 0.036 at d=768, from 1/sqrt(d), and the delta R2 rises to 0.22-0.28 (Section 4.1, Table 8). Table 6 is mixed (at 220M, cumulative + replacement at 37.05 beats cumulative + additive at 38.10; Appendix D). The contributions say the paper shows cumulative sources are a poor choice (Section 1), while the limitations section calls the explanation a hypothesis (Appendix H).
-
The fine-tuning experiment (Section 3.5) is under-controlled. Delta Block gets a 100x learning rate on the routing parameters (5e-3 vs 5e-5), a gate, and an extra zero-init source, while AttnRes gets none of these (Section 3.5, Setup). There is no ablation and no AttnRes baseline with the same higher routing learning rate (Section 3.5 compares only baseline, AttnRes and Delta Block). The gain (55.6 vs 55.0) comes entirely from ARC-Easy (+5.0; Table 5). It is below the baseline on 4 of the other 7 tasks, ties on BoolQ and is above on 2 (Table 5). It is one seed, one 0.6B model and one recipe (Appendix H). "Improves on the pretrained model" is trivially true for baseline fine-tuning (Table 5: baseline fine-tuning 55.0 vs pretrained 50.6; contribution 3 in Section 1), and "starts at the pretrained loss" follows directly from the zero gate (Section 3.5, Initialization and training loss). No gate values are reported, and there is no Delta AttnRes in fine-tuning (Section 3.5, Figure 5, Table 5).
-
The fine-tuning comparison gives AttnRes no adaptation stage. AttnRes starts from a disrupted state (the text reports its loss rising from 2.8 to 3.96) and is compared after the same 20K steps, with no separate adaptation stage that brings it back to the pretrained loss before the comparison. In Figure 5(c), its validation loss only meets Delta Block's at about 12,500 steps and both end at roughly the same value, so the downstream gap (54.1 vs 55.6, Table 5) may reflect adaptation cost rather than the design. This suggests AttnRes catches up in loss. The text (recovery in about 2000 steps) does not match the plot, and the baseline curve is missing from panel (c) (Section 3.5, Figure 5). Prior work on converting pretrained checkpoints includes an adaptation stage before the converted model is judged, for example the dense warm-up stage in DeepSeek-V3.2 (DeepSeek-AI, 2025, arXiv 2512.02556, Section 2.1.1).
-
Some clarity issues affect the claims. The Table 6 cells (page 9) are only defined in Appendix D (page 15). There, delta + replacement keeps the running stream as a source, so it is not the same as Full AttnRes (37.90 vs 37.30 at 220M; Tables 6 and 1), although Section 4.2 says Full AttnRes is replacement over the same sources. "Residual preservation" is listed as an advantage (Section 2.3), but it is just the design.
Formatting and presentation issues
-
"L=" is repeated about 23 times (L=28 in Fig 1, Fig 4, Table 2 captions, Section 3.3, Section 4.3 and "N=L=28"; L=36 repeated for both 1044M and 7.57B; L=12/L=24 in Section 3.1, Table 1, Section 3.4 and Table 4).
-
Eq. 1 (page 2; a sum with a nested softmax fraction) takes a lot of space, and the softmax is defined twice (Eq. 1 and Section 2.3). Fig. 2 also takes about 1/3 of page 3.
-
The Figure 3 pseudocode (page 4) takes half a page, and the wrapped lines leave fragments like "attn_norm)" and "mlp_norm)".
-
It is unclear which models are Qwen3. Is the 220M, 533M, and 1044M Qwen3? Or only the 0.6B and 7B (section 3.1). The 8B/7.6B/7.57B naming is inconsistent (used 13/6/3 times; Sections 3.1 and 3.3, Table 3).
-
The 0.6B learning rate is missing from the training bullet and only appears in the Table 2 caption (Section 3.1 on page 4, Table 2 on page 6).
-
The Table 3 "Scale" column is odd, and the caption says "8B" while the row says 7.57B (Table 3, page 6).
-
Figures 1 and 4 are the exact same image (Figure 1 on page 2 and Figure 4 on page 6; both are Qwen3-0.6B with L=28 and report the same 0.62 vs 0.35 average max weight in the captions).
-
"AttnRes" means Block AttnRes with 4 blocks in Table 1 (Section 3.2), "AttnRes (N=28)" (which is Full AttnRes by the paper's own definition) in Table 2, and the block count is not stated in Table 3 (page 6).
Questions for authors
-
Does starting from a pretrained model still make sense here? Wouldn't the latent representation change a lot when moving from the standard residual to AttnRes? The paper has no proper adaptation method for this transition. Instead it introduces the delta mechanism, which suggests that the gate values may mostly just keep the standard residual (the model was pretrained with it). The only adaptation step is a bigger learning rate, but still...
-
The paper uses its own learning rates (6e-4 and 3e-4 for from-scratch training; Section 3.1), not the ones from the original AttnRes report (Kimi Team, 2026, Section 5.1 and Table 2). Were the learning rates tuned for the baseline, AttnRes and the Delta variants, both from scratch and in fine-tuning? If not, is there any argument for the values used?
-
What do the gate values look like over training? Do some stay near zero? The gate is introduced in Section 2.3 and no values are reported in Section 3.5.
Response to Reviewer Kxtf
Thank you for the careful review and suggestions.
W1/W7 — Definitions
The 2×2 variants use different sources:
deltaincludes the embedding and earlier outputs;delta_replaceuses earlier outputs and the current cumulative state, without a separate embedding. It differs from Full AttnRes. We will clarify sources, resets, gates, readout, and the trained alternating recurrence. Residual preservation is a design property; our Full AttnRes evidence concerns one scale.W2/W3 and Q2 — Learning rates and longer training
Following your suggestion, we compared three learning rates at 220M and extended training to ~20 tokens per parameter (TPP), a Chinchilla-inspired reference budget (Hoffmann et al., 2022). Original LRs were shared, without per-method tuning. New pairs share initialization and data.
Table R1 — Validation PPL
Alternating improves in all three seed pairs (mean ± sample SD shown). At 20 TPP (4.404B tokens), PPL falls 1.45% versus standard residuals. Budgets differ in batch size and cosine duration. This is a limited development grid: at 2K, AttnRes has lower PPL (84.45 vs 90.74); the 0.10 gap at 3e-4 above remains small.
The headline 1.7–8.2% is Delta Block versus standard residuals. The 7.57B and fine-tuning results concern Delta Block; the 7.57B budget is 1.31B tokens (0.17 TPP). We will clarify the variants and finite-budget scope.
W4 — Redundancy across depth
As you noted, cumulative states share a prefix. We therefore measured actual block sources across depth, comparing alternating with true adjacent differences:
Table R2 — Projection R²
R² is the mean per-token uncentered squared-norm fraction explained by earlier sources; shuffling uses another sequence. Alternating is more redundant at late depth. We will discuss low redundancy as a possible explanation for utility, consistent with the mixed cumulative-source ablation.
W5/W6 and Q1/Q3 — Gating and adaptation
To address your adaptation concerns, we tested multiplicative gates on Qwen3-0.6B. DAR uses
p + g*r: residual streamp, routed mixturer, and zero-initialized learned scalarg. For Block AttnRes, we added a cumulative bypass and gates, including the final readout: zero gates recover pretrained logits and unit gates recover native Block AttnRes. This new control preserves native candidates and resets.Both methods share the backbone, data, training budget and schedule, with no null source. Router LR is 1× or 100× the backbone LR; these are single-seed comparisons.
Table R3 — Gated continuation (PPL)
Higher router LR produces larger gates, with PPL improvements below 0.03%. Some gates remain near zero, as DAR 1× shows:
Table R4 — Gate evolution
We also tested routing interventions with backbone weights fixed:
Table R5 — Routing interventions (PPL)
Uniform routing zeros the queries. The scratch checkpoint depends on routing, whereas the 1× continuation checkpoint changes little when routing is disabled. All PPLs use the same held-out validation set.
Matched curves now include the baseline and both gated controls. We will revise the ~2K recovery claim: recovering the initial loss differs from matching an adapted baseline, and late catch-up supports an adaptation-cost explanation. The original +0.6 downstream average is ARC-Easy-driven, with mixed results elsewhere. Our loader audit reproduced gate resets; the new checkpoints pass reload checks, while the original Table 5 provenance remains unverified.
Presentation
We will remove repeated L and duplicate Figs. 1/4, define softmax once, compact Eq. 1/Fig. 2, fix pseudocode wrapping, specify Qwen3 models/0.6B LR, unify 7.57B naming, and clarify Table 3 scale/AttnRes block counts.