- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Dynamic Masking (DyM), which scales the output of a residual branch before it is added to the residual stream, therefore scaling the layer’s contribution. The motivation is to make the update strength of each residual update input-dependent, and to make this scaling interpretable as an internal control signal. The paper investigates the method across three domains: language, protein modelling and vision (classification and segmentation). In those experiments not only performance is investigated, but also interventional studies are performed to understand the interpretation of the learned scaling values.
Critical strengths
The breadth of the study. The authors investigate the method in three different domains and on five different models in total. This is much appreciated for understanding the generality of the claims made in the paper. Furthermore, I appreciate the honest reporting in the results sections. Where results are weak or inconsistent, the authors do not hide these issues but mention them often clearly.
Critical weaknesses
Missing a “takeaway”. This is the main weakness in the writing. The authors test a small modification to residual architectures, but performance does not increase consistently, nor can I observe a consistent interpretation. I would also appreciate a negative story, but it is hard to formulate a key takeway of this paper or what it would contribute to the literature after reading.
The gate cannot be read on its own. Since is a function of , a lower might just mean their updates are larger, and the gate compensates or vice versa. The authors introduce as some scaling of features of , but the overall scale could easily be absorbed into itself.
Limited insights from some of the interventional experiments. The neutralisation experiment does not provide the interpretation the authors claim. Neutralising the tokens with the largest gate magnitude means you make the largest change to the gate values post-hoc, while the group with gates closest to 1 is barely changed. Hence, one would assume that the top group gives the largest changes. This is, in my opinion, a major issue, since many of their language-model claims rely on this experiment. I see a similar issue with their post-hoc token scalar, branch scalar identity and shuffled experiments. Any trained network can break if you swap out a part it learned to rely on. So yes, these experiments show that the model depends on its learned gating structure, but such dependence is not surprising. I am missing an actual insight beyond this dependence. The authors do acknowledge the limitations, but the abstract and contributions still present it as a key result.
Limited novelty. Learned scaling of residual branch has already been investigated in previous works, and the methods closest to DyM are in my opinion not discussed. ReZero [1] is similar to the branch scalar condition in Table 4 but trained, and therefore a natural baseline. Gated Attention [2] applies an input-dependent sigmoid gate after attention, shows consistent performance gains at the 1.7B–15B scale, and analyzes the learned gating scores. Neither is cited or compared against, so the novelty of DyM is not clearly discussed.
Hence, given the limited insights provided by their experiments, a missing clear takeaway and limited novelty, I am concerned that the paper does contribute little to the current literature.
[1]: Bachlechner, Thomas, et al. "Rezero is all you need: Fast convergence at large depth." Uncertainty in artificial intelligence. PMLR, 2021.
[2]: Qiu, Zihan, et al. "Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free." Advances in Neural Information Processing Systems 38 (2026)
Questions for authors
- What is the main takeaway of the paper? In the conclusion, the authors write “DYM as a general mechanism for adaptive representation refinement”, however, how is this different from any residual layer or modification to it? To me, this is unfortunately just an obvious statement, not a conclusion. Same about “inspectable control variable that can be tested directly through intervention”, this is true for most “control variables”.
- Can the authors address the weaknesses of the neutralisation and gate-ablation experiments? These are key results of the paper, but, as they are performed, they can only support very limited conclusions.
- Could the authors discuss their contribution in the context of the existing work I have mentioned above? Specifically, what insights are gained beyond ReZero and Gated Attention?
Thank you for the detailed review. We agree that the paper needed to state its central takeaway more sharply, and that the neutralization and ablation experiments support a narrower interpretation than a full causal reading of the gate values. At the same time, we believe the contribution is broader than generic residual scaling, and we have revised the manuscript to make this distinction clearer.
1. Main takeaway and gate interpretation
Our main takeaway is not that residual scaling exists, nor that DYM consistently improves predictive performance. The central finding is that learned regulation of residual updates develops structured, task-dependent behavior: computationally consequential representations may be either attenuated or amplified.
A standard residual layer determines what update to compute, but normally incorporates it with a fixed coefficient. DYM adds a separate, input-dependent control over how strongly that newly computed update is incorporated. This provides an explicit computational variable for studying update regulation rather than only the resulting representation.
We agree that the absolute numerical gate value should not be interpreted in isolation. Because the candidate update and gate are learned jointly, some overall scaling can be redistributed between them. Our analysis instead focuses on relative modulation within the same trained layer. The same DYM module, with shared parameters, is applied to all tokens in that layer, and differences across tokens reflect how the model differentially regulates their updates under a common modulation function. Thus, we do not interpret values such as 0.8 or 1.2 as absolute measures of feature importance. Rather, the gate reflects learned allocation of update strength across tokens, whose relation to feature relevance is evaluated empirically. We have clarified this interpretation in the revised manuscript.
This is also why we view DYM as potentially useful beyond post-hoc interpretation: a learned relevance-related signal can participate directly in computation and influence how newly computed information is incorporated during training and inference.
2. Neutralization and gate-ablation experiments
We agree that the neutralization analysis is confounded by intervention size. If ut is the candidate update and gt its learned gate, resetting gt to one changes the local update by (gt−1)⊙ut. Tokens farther from identity can therefore receive larger perturbations. The current result supports the narrower conclusion that removing learned modulation from strongly modulated tokens has a larger effect than the tested position-matched controls; it does not establish that the ranking remains informative independently of perturbation magnitude. A stronger control would also match the norm of the induced update change. We have incorporated this qualification and narrowed the claim accordingly.
Likewise, the scalar, identity, and shuffled-gate experiments are inference-time diagnostics of what structure the trained DYM checkpoint relies on; they do not prove that DYM is uniquely necessary. The informative observation is that performance is sensitive not only to average residual scaling, but also to feature-wise structure and token assignment. We have revised the abstract and contribution statements so these experiments are presented as dependency analyses rather than stronger causal evidence.
3. Relation to ReZero and Gated Attention
We agree that both are important precedents. We have revised the related-work discussion to address them directly. ReZero learns a residual scaling parameter that is not input-dependent, making it conceptually closer to our branch-scalar condition. Gated Attention is closer to DYM because it uses an input-dependent gate on attention output.
We do not claim that DYM invents residual gating. The contribution is to use input-dependent gating specifically as update modulation and to study the resulting control signal across language, vision, structural biology, and medical segmentation through attenuation/amplification analysis, external annotations, perturbations, and direct interventions.
The Pythia-160M comparison is matched within our setup: Original and DYM were both trained from scratch on the same Pile data with the same tokenizer, schedule, and budget. Training used 4 GPUs, micro-batch 4/GPU, 8 gradient-accumulation steps, sequence length 2048, and 143,000 iterations, corresponding to approximately 37.5B tokens.
We have therefore sharpened the paper’s summary: DYM exposes residual-update strength as an explicit, input-dependent control signal. Across domains, this signal develops structured but task-dependent behavior, is not equivalent to conventional feature importance, and trained models become sensitive to its token- and feature-level structure.