acceptodds
Under review as a conference paper at ICLR 2027

Delta Attention Residuals

Abstract

Attention Residuals (AttnRes) replace the standard residual connection with learned softmax attention over previous sublayer outputs, and the routed mixture replaces the residual stream as the next sublayer's input. We find that this replacement limits AttnRes at scale: it improves on standard residuals at 220M but is 6.9% worse in perplexity at 1044M and 6.6% worse at 7.6B, and the variant without any residual stream degrades most. We propose Delta Attention Residuals, whose core change is the update rule: they keep the residual stream and add a softmax-routed mixture of deltas (v_i = h_i+1 - h_i) to it. Under an additive update, the natural alternative source, cumulative hidden states, is highly redundant (adjacent cosine 0.93-0.97, R^2 up to 0.99 in deep layers) and gives higher perplexity than deltas on two architecture families. Across 220M-7.6B, Delta Attention Residuals reduce validation perplexity by 1.7-8.2% relative to standard residuals, and a zero-initialized gate on the added term supports fine-tuning from pretrained checkpoints. Code is in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Delta Attention Residuals

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KxtfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on Attention Residuals (AttnRes), where each layer's input is a softmax-weighted mixture of earlier sources that replaces the residual stream (Section 2.1, Eq. 1). I really like the idea to bring back the original residual stream on top of AttnRes which is quite a new method and gives the flexibility for the model. The authors keep the standard residual stream and add a routed mixture of earlier sublayer deltas on top of it (Delta Attention Residuals; Eq. 2), plus a coarser block-level version similar with the original paper, Delta Block (Section 2.3, Appendix C). They report 1.7-8.2% lower validation perplexity than a standard residual across models from 220M to 7.6B parameters (Abstract, Tables 1 and 3), and say AttnRes degrades at the 1044M and 7.6B scales (Sections 3.2 and 3.3). An ablation over the source type (cumulative vs. delta) and the update rule (replacement vs. additive) and some routing statistics are used to explain why (Section 4, Table 6, Appendices D and E).

Critical strengths
  1. The design is simple and clear. It is one change to AttnRes: keep the stream and add the routed mixture, reusing the same router and sources (Sections 2.3 and 3.2, Table 7).
  2. The design question is well posed. The ablation over source type and update rule is parameter-matched (Appendix D) and repeated on a LLaMA-style model (Appendix F), and it reports cells that go against the paper's own story (for example cumulative + replacement beating delta + replacement at 220M, 37.05 vs 37.90; Table 6).
  3. Using deltas under an additive update is a principled choice, since cumulative states already contain the stream. The R2 and cosine statistics give some backing for this (Sections 2.2 and 4.1, Table 8).
Critical weaknesses

Weakness

  1. The explanation for why replacement hurts is not tested at the scales where it hurts. AttnRes beats the baseline in validation perplexity at 220M (37.39 vs 38.71) and 533M (31.75 vs 32.00), but loses at 1044M (31.76 vs 29.70) and 7.6B (18.58 vs 17.43) (Tables 1 and 3). The paper attributes this to losing the identity path (Sections 3.2 and 4.2) and says the controlled ablation in Section 4.2 separates the effects of the source type and the update rule (Section 3.2). But that ablation (Table 6, Appendix D) only runs at 220M and 533M with a single seed, and none of its cells is AttnRes or Full AttnRes (Appendix D). The claim that Full AttnRes degrades most rests on one scale (33.36 vs 31.76; Table 1, Section 4.2). Also, the AttnRes baseline is a reimplementation that uses the same hyperparameters as the other methods, with 4 blocks and no tuning (Sections 3.1 and 3.2).

  2. The headline gains are mostly against the standard residual, and the 7.6B result is Delta Block only. The 1.7-8.2% range is Delta Block's (1044M in Table 1 and 7.6B in Table 3), and Delta AttnRes was not run at 7.6B (Abstract, Section 3.3). Against AttnRes the gain for Delta AttnRes is about 1.5-2.2% at 220M/533M (Table 1) with a single seed (Appendices D and H). The large gaps only appear where AttnRes underperforms the baseline (Tables 1 and 3).

  3. Hyperparameters and training budget are not justified. One learning rate is used (6e-4, and 3e-4 at 8B) with no tuning (Section 3.1). Batch size and sequence length change with scale (32x1024 vs 64x2048), so the models see about 0.33B vs about 1.3B tokens (10K steps each, computed from Section 3.1; Appendix H gives about 0.3B for the 220M-1B runs), and the scale trends are confounded with tokens. All runs use 10K steps, and the authors say they have not checked that the orderings hold at convergence-scale token budgets (Appendix H).

  4. The redundancy evidence is weaker than it sounds. The 0.93-0.97 cosine and R2 of 0.99 are depth-0.9 values (at depth 0.1 they are 0.64-0.67 and 0.45-0.46; Table 8, Appendix E), and are partly true by construction (Section 2.2 notes that adjacent cumulative states share a growing common prefix). The delta |cos| of at most 0.06 is close to a random baseline (about 0.036 at d=768, from 1/sqrt(d), and the delta R2 rises to 0.22-0.28 (Section 4.1, Table 8). Table 6 is mixed (at 220M, cumulative + replacement at 37.05 beats cumulative + additive at 38.10; Appendix D). The contributions say the paper shows cumulative sources are a poor choice (Section 1), while the limitations section calls the explanation a hypothesis (Appendix H).

  5. The fine-tuning experiment (Section 3.5) is under-controlled. Delta Block gets a 100x learning rate on the routing parameters (5e-3 vs 5e-5), a gate, and an extra zero-init source, while AttnRes gets none of these (Section 3.5, Setup). There is no ablation and no AttnRes baseline with the same higher routing learning rate (Section 3.5 compares only baseline, AttnRes and Delta Block). The gain (55.6 vs 55.0) comes entirely from ARC-Easy (+5.0; Table 5). It is below the baseline on 4 of the other 7 tasks, ties on BoolQ and is above on 2 (Table 5). It is one seed, one 0.6B model and one recipe (Appendix H). "Improves on the pretrained model" is trivially true for baseline fine-tuning (Table 5: baseline fine-tuning 55.0 vs pretrained 50.6; contribution 3 in Section 1), and "starts at the pretrained loss" follows directly from the zero gate (Section 3.5, Initialization and training loss). No gate values are reported, and there is no Delta AttnRes in fine-tuning (Section 3.5, Figure 5, Table 5).

  6. The fine-tuning comparison gives AttnRes no adaptation stage. AttnRes starts from a disrupted state (the text reports its loss rising from 2.8 to 3.96) and is compared after the same 20K steps, with no separate adaptation stage that brings it back to the pretrained loss before the comparison. In Figure 5(c), its validation loss only meets Delta Block's at about 12,500 steps and both end at roughly the same value, so the downstream gap (54.1 vs 55.6, Table 5) may reflect adaptation cost rather than the design. This suggests AttnRes catches up in loss. The text (recovery in about 2000 steps) does not match the plot, and the baseline curve is missing from panel (c) (Section 3.5, Figure 5). Prior work on converting pretrained checkpoints includes an adaptation stage before the converted model is judged, for example the dense warm-up stage in DeepSeek-V3.2 (DeepSeek-AI, 2025, arXiv 2512.02556, Section 2.1.1).

  7. Some clarity issues affect the claims. The Table 6 cells (page 9) are only defined in Appendix D (page 15). There, delta + replacement keeps the running stream as a source, so it is not the same as Full AttnRes (37.90 vs 37.30 at 220M; Tables 6 and 1), although Section 4.2 says Full AttnRes is replacement over the same sources. "Residual preservation" is listed as an advantage (Section 2.3), but it is just the design.

Formatting and presentation issues

  1. "L=" is repeated about 23 times (L=28 in Fig 1, Fig 4, Table 2 captions, Section 3.3, Section 4.3 and "N=L=28"; L=36 repeated for both 1044M and 7.57B; L=12/L=24 in Section 3.1, Table 1, Section 3.4 and Table 4).

  2. Eq. 1 (page 2; a sum with a nested softmax fraction) takes a lot of space, and the softmax is defined twice (Eq. 1 and Section 2.3). Fig. 2 also takes about 1/3 of page 3.

  3. The Figure 3 pseudocode (page 4) takes half a page, and the wrapped lines leave fragments like "attn_norm)" and "mlp_norm)".

  4. It is unclear which models are Qwen3. Is the 220M, 533M, and 1044M Qwen3? Or only the 0.6B and 7B (section 3.1). The 8B/7.6B/7.57B naming is inconsistent (used 13/6/3 times; Sections 3.1 and 3.3, Table 3).

  5. The 0.6B learning rate is missing from the training bullet and only appears in the Table 2 caption (Section 3.1 on page 4, Table 2 on page 6).

  6. The Table 3 "Scale" column is odd, and the caption says "8B" while the row says 7.57B (Table 3, page 6).

  7. Figures 1 and 4 are the exact same image (Figure 1 on page 2 and Figure 4 on page 6; both are Qwen3-0.6B with L=28 and report the same 0.62 vs 0.35 average max weight in the captions).

  8. "AttnRes" means Block AttnRes with 4 blocks in Table 1 (Section 3.2), "AttnRes (N=28)" (which is Full AttnRes by the paper's own definition) in Table 2, and the block count is not stated in Table 3 (page 6).

Questions for authors
  1. Does starting from a pretrained model still make sense here? Wouldn't the latent representation change a lot when moving from the standard residual to AttnRes? The paper has no proper adaptation method for this transition. Instead it introduces the delta mechanism, which suggests that the gate values may mostly just keep the standard residual (the model was pretrained with it). The only adaptation step is a bigger learning rate, but still...

  2. The paper uses its own learning rates (6e-4 and 3e-4 for from-scratch training; Section 3.1), not the ones from the original AttnRes report (Kimi Team, 2026, Section 5.1 and Table 2). Were the learning rates tuned for the baseline, AttnRes and the Delta variants, both from scratch and in fine-tuning? If not, is there any argument for the values used?

  3. What do the gate values look like over training? Do some stay near zero? The gate is introduced in Section 2.3 and no values are reported in Section 3.5.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Kxtf

Thank you for the careful review and suggestions.

W1/W7 — Definitions

The 2×2 variants use different sources: delta includes the embedding and earlier outputs; delta_replace uses earlier outputs and the current cumulative state, without a separate embedding. It differs from Full AttnRes. We will clarify sources, resets, gates, readout, and the trained alternating recurrence. Residual preservation is a design property; our Full AttnRes evidence concerns one scale.

W2/W3 and Q2 — Learning rates and longer training

Following your suggestion, we compared three learning rates at 220M and extended training to ~20 tokens per parameter (TPP), a Chinchilla-inspired reference budget (Hoffmann et al., 2022). Original LRs were shared, without per-method tuning. New pairs share initialization and data.

Table R1 — Validation PPL

Updates / LR / seedsStandard residualAttnResAlternating Delta Block
10K / 3e-4 / 4255.0951.9651.86
10K / 6e-4 / 4242.9442.1840.30
10K / 1.2e-3 / 4239.9138.7737.54
10K / 6e-4 / 42,1,242.90 ± 0.28—40.44 ± 0.26
33.6K (~20 TPP) / 6e-4 / 4223.33748—22.99913

Alternating improves in all three seed pairs (mean ± sample SD shown). At 20 TPP (4.404B tokens), PPL falls 1.45% versus standard residuals. Budgets differ in batch size and cosine duration. This is a limited development grid: at 2K, AttnRes has lower PPL (84.45 vs 90.74); the 0.10 gap at 3e-4 above remains small.

The headline 1.7–8.2% is Delta Block versus standard residuals. The 7.57B and fine-tuning results concern Delta Block; the 7.57B budget is 1.31B tokens (0.17 TPP). We will clarify the variants and finite-budget scope.

W4 — Redundancy across depth

As you noted, cumulative states share a prefix. We therefore measured actual block sources across depth, comparing alternating with true adjacent differences:

Table R2 — Projection R²

DepthAlternatingAdjacentShuffled: alt / adj
0.250.0040.0040.002 / 0.002
0.500.0200.0200.006 / 0.006
0.750.4410.0950.011 / 0.012

R² is the mean per-token uncentered squared-norm fraction explained by earlier sources; shuffling uses another sequence. Alternating is more redundant at late depth. We will discuss low redundancy as a possible explanation for utility, consistent with the mixed cumulative-source ablation.

W5/W6 and Q1/Q3 — Gating and adaptation

To address your adaptation concerns, we tested multiplicative gates on Qwen3-0.6B. DAR uses p + g*r: residual stream p, routed mixture r, and zero-initialized learned scalar g. For Block AttnRes, we added a cumulative bypass and gates, including the final readout: zero gates recover pretrained logits and unit gates recover native Block AttnRes. This new control preserves native candidates and resets.

Both methods share the backbone, data, training budget and schedule, with no null source. Router LR is 1× or 100× the backbone LR; these are single-seed comparisons.

Table R3 — Gated continuation (PPL)

MethodRouter LR 1×Router LR 100×Final median abs(g), 1× / 100×
Standard residual15.64745——
Alternating Delta Block15.6476615.643590.0025 / 0.0281
Identity-gated Block AttnRes15.6476815.643780.0044 / 0.0258

Higher router LR produces larger gates, with PPL improvements below 0.03%. Some gates remain near zero, as DAR 1× shows:

Table R4 — Gate evolution

UpdateMedian abs(g)Gates below 1e-3 / 56
1000.00039250
5000.00199115
10000.00249513

We also tested routing interventions with backbone weights fixed:

Table R5 — Routing interventions (PPL)

Condition220M scratch, 10KQwen3-0.6B FT, 1K, router 1×
DAR original40.29515.64766
Half branch41.49815.64784
Branch off59.18915.64780
Uniform routing42.230—

Uniform routing zeros the queries. The scratch checkpoint depends on routing, whereas the 1× continuation checkpoint changes little when routing is disabled. All PPLs use the same held-out validation set.

Matched curves now include the baseline and both gated controls. We will revise the ~2K recovery claim: recovering the initial loss differs from matching an adapted baseline, and late catch-up supports an adaptation-cost explanation. The original +0.6 downstream average is ARC-Easy-driven, with mixed results elsewhere. Our loader audit reproduced gate resets; the new checkpoints pass reload checks, while the original Table 5 provenance remains unverified.

Presentation

We will remove repeated L and duplicate Figs. 1/4, define softmax once, compact Eq. 1/Fig. 2, fix pseudocode wrapping, specify Qwen3 models/0.6B LR, unify 7.57B naming, and clarify Table 3 scale/AttnRes block counts.

Reviewer_Q2AROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes additive depth-wise routing that preserves the residual stream, with per-sublayer and block variants and zero-initialized gates for pretrained adaptation. It reports lower perplexity across 220M–7.57B models and a modest downstream improvement after fine-tuning.

Critical strengths
  1. Preserving the residual stream while adding routed information is a simple, useful idea. Zero-initialized gating facilitates pretrained adaptation.
  2. The cross-scale results, two-architecture ablation, and efficiency measurements provide useful evidence about the design choices.
Critical weaknesses
  1. The implemented block method does not match its main motivation. Equation (3) describes adjacent-boundary differences, but Appendix C and the code use sk=pk−sk−1s_k=p_k-s_{k-1}. The authors acknowledge that true block differences were not tested and that redundancy statistics concern per-sublayer sources. Thus, the main Delta Block results do not directly support the explanation based on low-redundancy block increments. This also makes the presentation difficult to reconcile.
  2. Evaluation validity and robustness need clarification. Both scratch-training scripts construct training and validation streams from the same train split, changing only the shuffle seed. This does not guarantee disjoint documents; actual overlap remains unknown. The archive contains run-name mappings but no training logs. Single-seed comparisons also leave uncertainty around smaller gains, including the 0.6-point downstream improvement. Appendix D’s limited repeat-run comparisons do not replace systematic uncertainty estimates.
Questions for authors
  1. What exact disjoint validation partition was used? If the supplied pipeline was used unchanged, can the checkpoints be evaluated on a verified held-out set?
  2. Does the implemented block recurrence retain the claimed low redundancy, and how does it compare with true adjacent-boundary differences?
  3. Can a representative comparison be repeated across independent seeds to establish the stability of the gains?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Q2AR

Thank you for identifying these issues. We have rerun the 220M comparison with verified document exclusion, tested both block constructions, and repeated the main comparison across three independent seeds.

Q1 — Validation integrity and re-evaluation

You are right that different shuffles do not establish disjoint documents. The original overlap rate is unknown, and the checkpoints behind the original reported scores are unavailable for direct re-evaluation. We therefore retrained with document exclusion enforced before tokenization.

We reserve crc32(id) mod 1000 < 10 for validation. Training excludes this partition and all validation document-ID and full-text SHA256 matches. An audit of the actually consumed training manifests against the 19,346 validation documents gives:

Table R1 — Train/validation overlap audit

SeedConsumed training documentsID overlapsExact-text overlaps
42319,59900
1314,84200
2309,43000

Near-duplicates are outside this audit. All PPLs below use the same 19,980,288 scored validation tokens. The comparisons use 220M models, 10K updates (327.68M training tokens), and peak LR 6e-4.

Q2 — Recurrence and block-source redundancy

The implemented alternating recurrence and adjacent-boundary differences are not equivalent. With pkp_k the state at block boundary kk and s0=d0=p0s_0=d_0=p_0 the embedding, alternating uses sk=pk−sk−1s_k=p_k-s_{k-1}, while adjacent uses dk=pk−pk−1d_k=p_k-p_{k-1}. We tested both, checking the adjacent construction against boundary states up to BF16 rounding.

Table R2 — Matched comparison (seed 42; B=4 for routing methods)

MethodValidation PPL
Standard residual42.94
AttnRes42.18
Delta Block, alternating40.30
Delta Block, adjacent40.24

All methods share backbone initialization and training tokens. Both constructions improve over standard residuals in this run; their 0.06-PPL gap does not establish a reliable ordering.

We also measured the actual stored block sources over 1,995,873 validation positions, extending the comparison to all three non-embedding source boundaries:

Table R3 — Block-source redundancy across depth

Relative depthConstructionMean abs. cosineProjection R²Shuffled R²
0.25Alternating0.0500.0040.002
0.25Adjacent0.0510.0040.002
0.50Alternating0.1080.0200.006
0.50Adjacent0.1070.0200.006
0.75Alternating0.2730.4410.011
0.75Adjacent0.2810.0950.012

Cosine compares each source with its predecessor. R² is the per-token uncentered squared-norm fraction explained by all earlier sources, averaged over tokens. Shuffled predictors come from another sequence in the batch, which may share a document.

The two constructions have similar redundancy at the first two boundaries. At the last boundary, alternating is substantially more predictable from all earlier sources, despite similar predecessor cosine. Thus the per-sublayer analysis cannot establish low redundancy for the implemented block recurrence. These checkpoint diagnostics do not isolate a causal effect of redundancy on PPL. We will correct Eq. (3), the pseudocode, and Appendix C, and identify the recurrence behind each result.

Q3 — Independent seeds and auditability

Across seeds, both initialization and the sampled training stream change; within each seed, baseline and alternating share both.

Table R4 — Independent training seeds

SeedStandard residual PPLAlternating Delta Block PPLPPL decrease
4242.9440.302.64
142.6140.292.32
243.1640.742.42
Mean ± sample SD42.90 ± 0.2840.44 ± 0.262.46 ± 0.17

Alternating improves in all three pairs, reducing mean PPL by 5.7%. The SD measures variation across independent training runs. This supports repeatability of the 220M result at this budget. Table 5's original single-seed +0.6 downstream gain remains inconclusive pending verified checkpoint loading and matched adaptation controls.

For auditability, we have prepared a bundle containing configurations, actual training and evaluation logs, partition audits, code/data hashes, and per-window losses for the new runs. Full document manifests and checkpoints are retained separately.

Reviewer_xdxMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Delta Attention Residuals, which improves the original Attention Residuals (AttnRes) by preserving the standard residual connection instead of replacing it. Instead of replacing, it adds an attention-weighted mixture of previous layers' outputs (deltas) to the residual stream.

They validated this method with model sizes up to 8B parameters. They found that AttnRes shows diminishing improvements and even degradation on larger model sizes. At 8B parameters, Delta Attention Residual achieves 8.2% lower perplexity than standard residuals, while the original AttnRes performs 6.6% worse.

Critical strengths
  • Simple and intuitive idea: Preserving the residual stream maintains the identity path and avoids disrupting information flow.
  • Strong and comprehensive experimental results: Consistently improves language modeling capabilities and downstream performance across model sizes from 220M to 7.6B.
  • Good ablations: Separates the effects of additive routing and delta representations, including experiments on Qwen and LLaMA architectures.
  • Supports pretrained models: Zero-initialized gating allows existing checkpoints to be converted without changing their initial behavior.
Critical weaknesses
  • Very weird batch size and training token count. They use 64 batch size with 2K context lengths, which gives 128K tokens per batch. Even worse, the authors only trained with 10K steps, which gives 1.28B tokens, which is too few for model sizes such as 8B. It is very important to validate the method on more typical configurations (more commonly used token-to-parameter ratios).
  • Significant computational overhead: The 7.6B model has 35% lower training throughput than baseline, despite having only 0.008% additional parameters.
  • Implementation inconsistency: The Delta Block experiments use an alternating-sum recurrence rather than the adjacent-state differences described in the method. The intended formulation is not tested. The authors should have discussed why there is such an inconsistency.
Questions for authors
  • Would a multiplicative gate further improve performance?
4 comments
AuthorsOpenReview ↗

Response to Reviewer xdxM

Thank you for the precise review. We add longer training and gating controls, and clarify the computational cost and block definition.

W1 — Training budget

The original 7.57B run uses 10,000 updates, global batch 64, and sequence length 2,048: 1.31B input tokens, or 0.17 tokens per parameter. We will present it explicitly as a finite-budget Delta Block comparison.

Our new 220M comparisons are:

Table R1 — Training budget and validation PPL

Updates / seedsInput tokens/runStandard residualAlternating Delta Block
10K / 42,1,2327.68M42.90 ± 0.2840.44 ± 0.26
33.6K / 424.404B (~20 TPP)23.3374822.99913

All use peak LR 6e-4 and a full-budget cosine schedule. Within each seed, methods share backbone initialization and training tokens. The 10K row uses batch 32 and reports mean ± sample SD; alternating improves in all three pairs (5.7% lower mean PPL). The new B200 pair uses batch 128, sequence length 1024 and lowers PPL by 1.45% at ~20 tokens/parameter. This extends the result to a longer 220M budget with one seed; the two budgets use different batch sizes and schedules.

W2 — Computational overhead

The reported 35% throughput reduction at 7.57B is material. Extra parameter count does not capture the cost of reading, normalizing, and aggregating source activations. Our new seed-42, 220M/10K-step measurements on an A100 are:

Table R2 — PPL and training cost (220M, seed 42)

MethodPPLms/updatePeak allocated GiB
Standard residual42.9463020.02
Delta Block, alternating40.3095823.31
Delta Block, adjacent40.2496023.31
AttnRes, B=442.18110425.01

All rows use LR 6e-4; routed methods use B=4. Times are median updates after step 500 with unfused routing, excluding evaluation/checkpoints; memory is peak PyTorch allocation. Alternating has 34% lower throughput and 16% more allocated memory than standard residuals, but is faster and uses less memory than AttnRes here. We will distinguish full-model training costs from routing-operator speedups.

W3 — Implemented versus stated block recurrence

You are right: the main-text definition and implemented source bookkeeping describe different constructions, not algebraically equivalent forms. Let pkp_k denote the residual state at block boundary k, with p0p_0 the embedding. The implementation uses s0=p0s_0=p_0 and sk=pk−sk−1s_k=p_k-s_{k-1}; adjacent differences are d0=p0d_0=p_0 and dk=pk−pk−1d_k=p_k-p_{k-1}.

We implemented the adjacent variant, checked it against actual boundary differences, and trained it under the same new protocol. Its PPL is 40.24 versus 40.30 for alternating above. Both improve over the matched baseline in this seed; the 0.06-PPL difference establishes neither a consistent ordering nor statistical equivalence.

All methods share a fixed validation file scoring 19,980,288 tokens. An audit of the consumed data finds zero document-ID and exact full-text overlap with training; near-duplicate exclusion is not guaranteed. We measured stored block sources over 1,995,873 validation positions:

Table R3 — Last-source projection R²

ConstructionProjection R²Shuffled-token reference
Alternating0.4410.011
Adjacent0.0950.012

Alternating is more redundant by this diagnostic. These block measurements do not justify describing the implemented recurrence as low-redundancy adjacent increments, or equating lower redundancy with better modeling quality.

We will correct Eq. (3), the pseudocode, and Appendix C, and label the reported construction as alternating.

Q — Would a multiplicative gate improve performance?

Following your suggestion, we ran additional experiments with a multiplicative scalar gate on Qwen3-0.6B. At each routing site, we use p + g*r, where p is the residual stream, r is the routed mixture, and g is a learnable scalar shared across tokens and channels. We initialize each gate to zero, preserving pretrained logits exactly, and train it jointly with the router and backbone.

Table R4 — Gated continuation (full-validation PPL)

ConditionPPLFinal median abs(g)
Standard residual15.64745—
DAR, router LR 1×15.647660.0025
DAR, router LR 100×15.643590.0281
DAR 1×, branch off after training15.64780—

The higher router LR produces larger learned gates and a small PPL improvement of 0.025% over standard residuals. At 1×, disabling the routed branch after training barely changes PPL. The PPL differences remain small in this single-seed comparison.

We verified that trained weights and gates are preserved on reload. Our loader audit reproduced gate resets in the released code; these new controls use verified loading and are separate from the original Table 5, whose checkpoint/runtime provenance remains unverified.

Thank you for your efforts.

It looks like the text in your response to my question regarding multiplicative gate is irrelevant to the question. You might have mistakenly pasted the response to another reviewer in its place. Would you kindly double check this?

AuthorsOpenReview ↗

Clarification on multiplicative gating

Thank you for pointing this out. We misunderstood your question in our previous response.

When taking about multiplicative gating, we mean gating the routed contribution before adding it to the residual stream. If p is the current residual state and r = sum_i alpha_i v_i is the softmax-weighted mixture of earlier deltas, ungated DAR uses h = p + r. We implemented two variants:

  • Scalar gate: h = p + sigmoid(a) * r, with a learned scalar at each routing site, shared across tokens and channels.
  • Input-dependent vector gate: h = p + sigmoid(W_g p + b_g) ⊙ r, where the gate varies across tokens and channels and ⊙ denotes elementwise multiplication.

In both cases, the gate scales the routed mixture after aggregation, while the residual path retains coefficient 1. The gate is learned jointly with the model. Does this match the multiplicative gating you had in mind, or were you suggesting a different placement or parameterization?

For reference, the Scalar gate results are:

Routed-branch multiplier rValidation PPL
1.0 (original)40.295
0.541.498
0.059.189

These interventions show that multiplicative gating does not help;

Thank you for the additional experiments. Some of my concerns have been resolved, and I have decided to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.