- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper applies the existing delta rule across network depth as an alternative to additive residual updates in Transformers. At each layer, the model reads a selected component of the residual state and makes a gated correction toward a learned target value. The authors propose both scalar and expanded residual-state implementations and report lower validation loss than an additive-residual baseline when pretraining language models at two scales.
Critical strengths
-
Clear research question. The paper focuses on how to update the residual state in Transformers. Rather than simply increasing model capacity, it proposes an explicit read–compare–write interface.
-
Clearly defined method. Equation (2.2) specifies the roles of the update direction, target value, and gate strength, making the method straightforward to implement and amenable to targeted ablation studies.
-
Experiments across two model scales and multiple implementations. The paper evaluates both scalar and expanded residual states, two compressors, and variants without embedding convolution. It reports validation loss as well as task-level results.
Critical weaknesses
-
Lack of a control that isolates the core update rule. The paper does not compare full DDL with a write-only update that removes the readout-error term while holding other settings fixed. It is therefore unclear whether erasing the old readout contributes to the observed gains.
-
The scalar variant does more than change the update rule. It also introduces direction normalization, a target branch, and a gating branch. The effects of these components should be examined separately before attributing the scalar variant’s modest improvement to the read–compare–write mechanism.
-
The scope of novelty is relatively narrow. The authors correctly acknowledge that their update corresponds term by term to the earlier delta rule. The main contribution is thus its use as a residual interface across network depth, rather than a new delta-rule formulation. More targeted cross-layer experiments would help establish the value of this application.
-
Downstream results and the evaluated scope remain limited. Although the average one-shot score improves, the small DDL-CC model’s average zero-shot score is 47.18, slightly below the baseline’s 47.30. The experiments also cover only two approximately GPT-2-sized models trained with a sequence length of 1,024. The generalization claims should be qualified, and evaluations at other scales or context lengths would strengthen them.
Questions for authors
Please refer to the Critical Weaknesses section. If the authors can address these concerns satisfactorily, I would consider raising my score.
2 comments
Thank you for your response, which addresses most of my concerns. Since my original score was positive, I will keep it unchanged.
We sincerely thank you for your valuable time and insightful feedback, for recognizing the clear research question and the clearly defined method, and for stating what would change your score. We ran the write-only control you asked for. We address all of your concerns and questions in detail below, and we will integrate this feedback in the revised version.
Setup of the write-only control. We use the small-scale model (12 layers, hidden size 768), 100,000 training steps, a global batch of 480 sequences of 1,024 tokens, the optimizer settings of Section 4.1, four H200 GPUs, and one seed per configuration. The parameterization and target v = 4σ(W_v x^in + b_v) follow the submission. Accuracy is the mean over the paper's eight tasks; the per-task standard error is 0.5 to 1.8 points.
W1 (control for the update rule). The write-only control keeps the state size, EC, the compressor, the k, v, and β branches, the initialization, the data order, and the optimizer, and replaces Eq. (2.2) with X_{l+1} = X_l + β_l k_l v_l^T.
DDL-CC reaches a validation loss of 2.8329 and its write-only variant 2.8406, a difference of 0.0077. Scalar DDL reaches 2.8482 and its write-only variant 2.8608, a difference of 0.0126. Relative to the baseline at 2.8543, DDL-CC and scalar DDL reduce validation loss by 0.0214 and 0.0061; the write-only DDL-CC variant is also 0.0137 below the baseline, while the scalar write-only variant is 0.0065 above it. These observations motivate further controlled ablation of the erase term, rather than establishing its contribution in isolation.
W2 (components of the scalar variant). Scalar DDL adds four things to the baseline: direction normalization, the target branch, the gate branch, and the erase term. The scalar DDL validation loss is 2.8482, versus 2.8608 for the write-only variant and 2.8543 for the baseline. The direction normalization, target branch, and gate branch have not been separately ablated, and we will make this limitation explicit in the paper.
W3 (scope of novelty). We agree with this description. Section 2.4 states that the rule is prior work and that the correspondence with DeltaNet is term by term. The contribution is its use as the residual interface across depth, together with analysis of that interface and empirical evaluation. For a cross-layer diagnostic, we examined the write-only checkpoint on 16 validation documents of 1,024 tokens across 24 sublayers: its mean gate β exceeds 1.5 in 12 sublayers, and the final state column norm is 14.3. This measurement characterizes the control, but does not isolate the erase mechanism in the full model.
W4 (downstream results and scope). As Section 4.2 notes, the zero-shot results are mixed: at the small scale DDL-CC scores 47.18 and DDL-CC w/o EC scores 47.07, against the baseline's 47.30 (Table 4), while at the medium scale all five DDL variants exceed the baseline's 51.92 (Table 5). At the small scale DDL-CC improves the one-shot average from 48.56 to 49.29 (Table 1), but its zero-shot average is 0.12 points below the baseline. The abstract currently reports the one-shot average only. We will include the mixed zero-shot evidence in the abstract and introduction. The experiments cover two model sizes roughly corresponding to GPT-2 small and medium, at sequence length 1,024, and we will state this in the abstract and conclusion.