acceptodds
Under review as a conference paper at ICLR 2027

Deep Delta Learning

Abstract

Transformer residual streams are updated by addition. A sufficiently expressive residual block can represent content replacement, but the residual update itself has no operation that reads, compares, and replaces content. We introduce Deep Delta Learning (DDL), which applies the delta rule over network depth. Each layer reads the residual state along a learned direction, compares the readout with a learned target, and writes a gated rank-1 correction back along the same direction. A closed gate gives the identity map, and a unit gate overwrites the selected readout with the target. DDL works with the usual vector state or with an expanded state that stores several value channels, while attention and MLP blocks keep the original model width. We pretrain decoder-only models with approximately GPT-2 small and medium sizes on the FineWeb-Edu dataset. At both scales, every DDL variant has lower validation loss and higher average one-shot accuracy than the additive baseline, and the best expanded variants raise that average by 0.91 and 1.18 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model architectures, residual stream

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pscEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper applies the existing delta rule across network depth as an alternative to additive residual updates in Transformers. At each layer, the model reads a selected component of the residual state and makes a gated correction toward a learned target value. The authors propose both scalar and expanded residual-state implementations and report lower validation loss than an additive-residual baseline when pretraining language models at two scales.

Critical strengths
  1. Clear research question. The paper focuses on how to update the residual state in Transformers. Rather than simply increasing model capacity, it proposes an explicit read–compare–write interface.

  2. Clearly defined method. Equation (2.2) specifies the roles of the update direction, target value, and gate strength, making the method straightforward to implement and amenable to targeted ablation studies.

  3. Experiments across two model scales and multiple implementations. The paper evaluates both scalar and expanded residual states, two compressors, and variants without embedding convolution. It reports validation loss as well as task-level results.

Critical weaknesses
  1. Lack of a control that isolates the core update rule. The paper does not compare full DDL with a write-only update that removes the readout-error term while holding other settings fixed. It is therefore unclear whether erasing the old readout contributes to the observed gains.

  2. The scalar variant does more than change the update rule. It also introduces direction normalization, a target branch, and a gating branch. The effects of these components should be examined separately before attributing the scalar variant’s modest improvement to the read–compare–write mechanism.

  3. The scope of novelty is relatively narrow. The authors correctly acknowledge that their update corresponds term by term to the earlier delta rule. The main contribution is thus its use as a residual interface across network depth, rather than a new delta-rule formulation. More targeted cross-layer experiments would help establish the value of this application.

  4. Downstream results and the evaluated scope remain limited. Although the average one-shot score improves, the small DDL-CC model’s average zero-shot score is 47.18, slightly below the baseline’s 47.30. The experiments also cover only two approximately GPT-2-sized models trained with a sequence length of 1,024. The generalization claims should be qualified, and evaluations at other scales or context lengths would strengthen them.

Questions for authors

Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.

2 comments

We sincerely thank you for your valuable time and insightful feedback, for recognizing the clear research question and the clearly defined method, and for stating what would change your score. We ran the write-only control you asked for. We address all of your concerns and questions in detail below, and we will integrate this feedback in the revised version.

Setup of the write-only control. We use the small-scale model (12 layers, hidden size 768), 100,000 training steps, a global batch of 480 sequences of 1,024 tokens, the optimizer settings of Section 4.1, four H200 GPUs, and one seed per configuration. The parameterization and target v = 4σ(W_v x^in + b_v) follow the submission. Accuracy is the mean over the paper's eight tasks; the per-task standard error is 0.5 to 1.8 points.

W1 (control for the update rule). The write-only control keeps the state size, EC, the compressor, the k, v, and β branches, the initialization, the data order, and the optimizer, and replaces Eq. (2.2) with X_{l+1} = X_l + β_l k_l v_l^T.

ModelUpdateValidation lossZero-shot avg.One-shot avg.
Baselinex + h2.854347.3048.56
DDL-CC (d_v = 4)Eq. (2.2)2.832947.1849.29
DDL-CC, write-onlyX + β k v^T2.840646.2647.20
Scalar DDL (d_v = 1)Eq. (2.2)2.848247.3248.73
Scalar DDL, write-onlyx + β k v2.860846.9747.37

DDL-CC reaches a validation loss of 2.8329 and its write-only variant 2.8406, a difference of 0.0077. Scalar DDL reaches 2.8482 and its write-only variant 2.8608, a difference of 0.0126. Relative to the baseline at 2.8543, DDL-CC and scalar DDL reduce validation loss by 0.0214 and 0.0061; the write-only DDL-CC variant is also 0.0137 below the baseline, while the scalar write-only variant is 0.0065 above it. These observations motivate further controlled ablation of the erase term, rather than establishing its contribution in isolation.

W2 (components of the scalar variant). Scalar DDL adds four things to the baseline: direction normalization, the target branch, the gate branch, and the erase term. The scalar DDL validation loss is 2.8482, versus 2.8608 for the write-only variant and 2.8543 for the baseline. The direction normalization, target branch, and gate branch have not been separately ablated, and we will make this limitation explicit in the paper.

W3 (scope of novelty). We agree with this description. Section 2.4 states that the rule is prior work and that the correspondence with DeltaNet is term by term. The contribution is its use as the residual interface across depth, together with analysis of that interface and empirical evaluation. For a cross-layer diagnostic, we examined the write-only checkpoint on 16 validation documents of 1,024 tokens across 24 sublayers: its mean gate β exceeds 1.5 in 12 sublayers, and the final state column norm is 14.3. This measurement characterizes the control, but does not isolate the erase mechanism in the full model.

W4 (downstream results and scope). As Section 4.2 notes, the zero-shot results are mixed: at the small scale DDL-CC scores 47.18 and DDL-CC w/o EC scores 47.07, against the baseline's 47.30 (Table 4), while at the medium scale all five DDL variants exceed the baseline's 51.92 (Table 5). At the small scale DDL-CC improves the one-shot average from 48.56 to 49.29 (Table 1), but its zero-shot average is 0.12 points below the baseline. The abstract currently reports the one-shot average only. We will include the mixed zero-shot evidence in the abstract and introduction. The experiments cover two model sizes roughly corresponding to GPT-2 small and medium, at sequence length 1,024, and we will state this in the abstract and conclusion.

Thank you for your response, which addresses most of my concerns. Since my original score was positive, I will keep it unchanged.

Reviewer_pienOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Deep Delta Learning replaces the usual residual update with a gated rank-one correction toward a learned target. Tests at two small language-model scales show lower validation loss, with larger gains when the residual state is expanded.

Critical strengths
  • The update has a clear interpretation, and the local algebra is easy to check.
  • The experiments cover two model sizes, scalar and expanded states, and several compression choices.
  • The paper reports quality and hardware costs, and supplies model code for the variants.
Critical weaknesses
  • The main result does not isolate the delta correction from extra state and compression. A matched write-only control is missing, and related expanded-residual methods are not compared fully.
  • Each configuration is trained once. The scalar validation-loss gains are only 0.0061 and 0.0014, while downstream gains are uneven, so their reliability is unclear.
  • The comparisons match tokens rather than compute. DDL is slower, so the current results do not show a better quality-versus-training-cost tradeoff.
  • The overwrite interpretation is not tested in trained models. Gate values and actual readout-error reductions are not measured, so the analysis explains the update rather than why it helps in practice.
Questions for authors
  • How does DDL compare with a write-only update using the same state, branches, compressor, initialization, and training budget?
  • Do the scalar and expanded gains persist across seeds and under matched training compute, including a full Hyper-Connections comparison?
  • What gate regimes do trained models use, and how much does each update reduce the selected readout error?
1 comment

We sincerely thank you for your valuable time and insightful feedback, and for recognizing that the update has a clear interpretation and that the experiments cover two model sizes, scalar and expanded states, and the hardware costs. We ran the write-only control you asked for and measured what the trained gates do. We address all of your concerns and questions in detail below, and we will integrate this feedback in the revised version.

Setup of the write-only control. We use the small-scale model (12 layers, hidden size 768), 100,000 training steps, a global batch of 480 sequences of 1,024 tokens, the optimizer settings of Section 4.1, four H200 GPUs, and one seed per configuration. The parameterization and target v = 4σ(W_v x^in + b_v) follow the submission. Accuracy is the mean over the paper's eight tasks; the per-task standard error is 0.5 to 1.8 points.

Write-only control. The control uses the same state, branches, compressor, initialization, data order, and training budget, as you specify, and replaces Eq. (2.2) with X_{l+1} = X_l + β_l k_l v_l^T.

ModelUpdateValidation lossZero-shot avg.One-shot avg.
Baselinex + h2.854347.3048.56
DDL-CC (d_v = 4)Eq. (2.2)2.832947.1849.29
DDL-CC, write-onlyX + β k v^T2.840646.2647.20
Scalar DDL (d_v = 1)Eq. (2.2)2.848247.3248.73
Scalar DDL, write-onlyx + β k v2.860846.9747.37

DDL-CC reaches a validation loss of 2.8329 and its write-only variant 2.8406, a difference of 0.0077. Scalar DDL reaches 2.8482 and its write-only variant 2.8608, a difference of 0.0126. Relative to the baseline at 2.8543, DDL-CC and scalar DDL reduce validation loss by 0.0214 and 0.0061; the write-only DDL-CC variant is also 0.0137 below the baseline, while the scalar write-only variant is 0.0065 above it. These observations motivate further controlled ablation of the erase term, rather than establishing its contribution in isolation.

Single runs. The scalar improvement of 0.0061 in validation loss is small, and Section 4.2 notes that single runs cannot distinguish a gain of this size from seed variation. We did not add seeds during the discussion period; every configuration uses one seed.

Compute and Hyper-Connections. The comparison in the paper is token-matched, and its cost is reported next to the quality: training throughput is lower by 23.3% for DDL-CC at the small scale and by 21.4% at the medium scale (Tables 7 and 8). HC was run at the small scale only, which is why it appears in Figure 2 and not in the tables.

Is the overwrite interpretation used in trained models? The DDL rule supports partial overwriting by construction, but demonstrating its precise contribution requires checkpoint-level measurement. In the write-only control, we measured 16 validation documents of 1,024 tokens across all 24 sublayers: the mean gate β exceeds 1.5 in 12 sublayers, close to its ceiling of 2, and the final state column norm is 14.3. The validation loss of full DDL-CC is 2.8329, versus 2.8406 for the write-only variant. These numbers do not, on their own, establish the contribution of the erase operation independently of other experimental factors.

Reviewer_qh9AOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper replaces the additive residual with a delta rule update over the depth. The sublayer output h is normed k=h/||h||, a gate β∈(0,2)\beta \in (0,2) and a target v∈(0,4)v \in (0,4) come from small linear layers on the sublayer input. The update moves the readout k⊤xk^\top x toward vv by step β\beta: xl+1=xl−β(k⊤xl)k+βvkx_{l+1} = x_l - \beta(k^\top x_l)k + \beta v k. An expanded version keeps dvd_v residual channels and uses a learned compressor (TC or CC, optionally with embedding convolutions) to feed the width-d sublayer.

The experiments were done on a nanoGPT small and medium (GPT-2 sized) on 49B FineWeb-Edu tokens, one run per config, against the additive baseline.

Critical strengths
  1. A simple and well defined mechanism that replaces additive residuals with a rank-1 erase/write along the sublayers output direction. It has an identity limit when β\beta goes to 0 and an exact overwrite when β\beta goes to 1. The method is easy to implement and compare against alternatives.
  2. Every DDL variant lowers final validation loss at both scales and the authors clearly state their limitations of single seeds, no write only control, not being compute matched and the throughput and memory cost.
Critical weaknesses
  1. The delta rule is not separated from learned gating. Scalar DDL can be written as x′=x+α(x)hx' = x + \alpha(x) h where α(x)=β(v−k⊤x)∣∣h∣∣\alpha(x) = \frac{\beta (v - k^\top x)}{|| h ||}, or as a highway update x′=(I−T)x+T(vk)x' = (I - T)x + T(vk) with T=βkk⊤T=\beta kk^\top. The expanded variant is the same with X′=(I−T)X+Tkv⊤X' = (I - T)X + T kv^\top. So DDL mainly differs from Highway in using a rank-1 projector gate with gain β∈(0,2)\beta \in (0,2) instead of a diagonal gate in (0,1)(0,1). But no gated or scaled residual is compared (Highway, GTrXL-style gating, ReZero, etc.), so the gain cannot be attributed to the delta rule rather than learned gating.
  2. Evidence is too weak to attribute or rank gains. One seed per config and it is hard to judge if some of the smaller improvements are within noise. Expanded gains also mix in multi-stream state (close to Hyper-Connections, which is missing from the tables). For TC/EC the short causal token convolutions might help on their own and are missing as a control. CC without EC has no token conv and still improves, so this mainly concerns the TC/EC rows. Comparisons are token-matched while DDL has 12–55% lower training throughput, so equal-compute gains are unknown.
Questions for authors
  1. How does DDL compare with the gated residual x′=x+β(c)hx' = x + \beta (c) h using the same gate branch and initialization, with Highway or ReZero-style residual, and with the write only update x′=x+βvkx' = x + \beta v k? Without these, the gain cannot be attributed to the erase term.
  2. Figure 2 shows HC (rate 4) at small scale only, why is it absent from the Tables 1-8? And how do the expanded variants compare with HC in loss, throughput and memory at both scales? How does a baseline with only the short token convolution perform for TC/EC?
  3. Training throughput drops by 12-55%, does DDL still beat a baseline trained for equal wall-clock time or FLOPs?
  4. What are the per layer distributions of β\beta, vv and k⊤xk^\top x? Is ∣k⊤x∣≫4|k^\top x| \gg 4 typical and is vv close to constant? This would show whether the update writes content or mostly pulls the stream back along kk.
  5. What is the seed to seed spread of validation loss over at least 3 seeds for the baseline, scalar DDL and the expanded variants (at least at small scale)? Are the scalar gains and the differences between expanded variants outside it?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We sincerely thank you for your valuable time and insightful feedback, and for recognizing that the mechanism is simple and well defined and that the paper states its limitations clearly. Your rewriting of the update as a gated residual is correct, and we take the point it makes. We ran the write-only control you asked for. We address all of your concerns and questions in detail below, and we will integrate this feedback in the revised version.

Setup of the write-only control. We use the small-scale model (12 layers, hidden size 768), 100,000 training steps, a global batch of 480 sequences of 1,024 tokens, the optimizer settings of Section 4.1, four H200 GPUs, and one seed per configuration. The parameterization and target v = 4σ(W_v x^in + b_v) follow the submission. Accuracy is the mean over the paper's eight tasks; the per-task standard error is 0.5 to 1.8 points.

W1 and Q1 (delta rule against learned gating). Scalar DDL is x + βk(v − k^T x) = (I − βkk^T)x + βvk, a gated update. It differs from a Highway layer in what the gate acts on. A Highway gate is diagonal with entries in (0, 1) and mixes x and the sublayer output coordinate by coordinate. The DDL shortcut I − βkk^T changes the state along one direction, the normalized sublayer output, with gain 1 − β in (−1, 1), and leaves the orthogonal complement untouched (Proposition 2.1). To measure what the erase term adds on top of a gated write, we ran a write-only control. It keeps the state size, EC, the compressor, the k, v, and β branches, the initialization, the data order, and the optimizer, and replaces Eq. (2.2) with X_{l+1} = X_l + β_l k_l v_l^T.

ModelUpdateValidation lossZero-shot avg.One-shot avg.
Baselinex + h2.854347.3048.56
DDL-CC (d_v = 4)Eq. (2.2)2.832947.1849.29
DDL-CC, write-onlyX + β k v^T2.840646.2647.20
Scalar DDL (d_v = 1)Eq. (2.2)2.848247.3248.73
Scalar DDL, write-onlyx + β k v2.860846.9747.37

DDL-CC reaches a validation loss of 2.8329 and its write-only variant 2.8406, a difference of 0.0077. Scalar DDL reaches 2.8482 and its write-only variant 2.8608, a difference of 0.0126. Relative to the baseline at 2.8543, DDL-CC and scalar DDL reduce validation loss by 0.0214 and 0.0061; the write-only DDL-CC variant is also 0.0137 below the baseline, while the scalar write-only variant is 0.0065 above it. These observations motivate further controlled ablation of the erase term, rather than establishing its contribution in isolation. We did not run the gated-residual or ReZero baselines.

W2 (strength of evidence).

  • Seeds: each configuration uses one seed, and we did not add seeds in the discussion period. Scalar DDL improves validation loss over the baseline by 0.0061 (Table 6), which is small enough that seed variation remains a concern.
  • Hyper-Connections: HC was run at the small scale only, which is why it appears in Figure 2 and not in the tables. The HC control is reported under Q2 in part 2/2.
  • Token convolution: we did not run a baseline with embedding convolution only. As you note, DDL-CC w/o EC has no token convolution and still lowers validation loss by 0.0222 at the small scale and 0.0263 at the medium scale (Table 6), so this control concerns the TC and EC rows.
  • Compute: see Q3 in part 2/2.
AuthorsOpenReview ↗

Official Comment by Authors (2/2)

This continues our response (1/2). The experimental setup is given there.

Q2 (Hyper-Connections). HC appears in Figure 2 at the small scale and not in the tables. A 4-stream HC run reached a validation loss of 2.8364; DDL-CC and the baseline report 2.8329 and 2.8543, respectively. The HC run took 19 h 56 min. We have no HC run at the medium scale.

Q3 (equal wall-clock or FLOPs). The comparison in the paper is token-matched, and its cost is reported next to the quality: training throughput is lower by 11.8% for scalar DDL and by 23.3% for DDL-CC at the small scale, and by 21.4% for DDL-CC at the medium scale (Tables 7 and 8).

Q4 (distributions of β, v, and k^T x). We inspected the write-only checkpoint using 16 validation documents of 1,024 tokens across its 24 sublayers. The mean gate β exceeds 1.5 in 12 of 24 sublayers, near its ceiling of 2, and the state column norm at the final sublayer is 14.3. These measurements describe the learned write-only dynamics, but do not by themselves determine how much of the full DDL improvement is due to partial overwriting. The parameterization permits v = 4σ(·) in (0, 4), with v = 2 at zero pre-activation, and gates are initialized near β = 1 (Section 3).

Q5 (seed spread). See W2 in part 1/2: one seed per configuration.

Thank you for the additional write only runs. My main concern from my review however remains, apart from the single small scale HC run, the paper does not compare DDL against other residual connection methods such as a gated residual x′=x+β(c)hx' = x + \beta(c)h with the same gate branch, Highway, GTrXL-style gating or ReZero. Therefore I cannot judge whether it improves on existing alternatives, which is why I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.