acceptodds
Under review as a conference paper at ICLR 2027

Learning When to Update: Dynamic Masking for Adaptive Representation Refinement

Abstract

Deep neural networks typically incorporate the output of each residual transformation with a fixed contribution, even though the relevance and computational role of individual content representations can vary substantially across inputs, features, and network depth. We introduce Dynamic Masking (DyM), a lightweight mechanism that learns an input-dependent coefficient for each content-bearing representation and uses it to regulate how strongly a newly computed update enters the residual stream. DyM therefore serves simultaneously as a computational control mechanism and an inspectable signal of representation regulation, while avoiding the assumption that its coefficients are direct measures of feature importance. We evaluate DyM across language modeling, structural biology, computer vision, and 3D medical image segmentation. In language models, DyM modulation is structured by linguistic category and context, and neutralizing the most strongly modulated tokens increases target negative log-likelihood substantially more than neutralizing position-matched random or weakly modulated tokens. Notably, strongly attenuated tokens are particularly consequential, demonstrating that lower DyM coefficients do not imply lower relevance. In antibody–antigen interface prediction, biologically relevant interface residues exhibit systematically stronger attenuation of representation updates than non-interface residues. DyM also maintains competitive predictive performance, including modest improvements in ImageNet classification and average brain-tumor segmentation. Across domains, the results show that task-relevant representations may be amplified, attenuated, or dynamically reallocated depending on context. These findings position DyM as a general mechanism for exposing and controlling adaptive representation refinement, and not just a conventional feature-importance estimator.

Primary area:
interpretability and explainable AI
Keywords:
Dynamic Masking, Interpretable Deep Learning, Representation Modulation, Residual Networks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5BwxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Dynamic Masking (DyM), which scales the output of a residual branch before it is added to the residual stream, therefore scaling the layer’s contribution. The motivation is to make the update strength of each residual update input-dependent, and to make this scaling interpretable as an internal control signal. The paper investigates the method across three domains: language, protein modelling and vision (classification and segmentation). In those experiments not only performance is investigated, but also interventional studies are performed to understand the interpretation of the learned scaling values.

Critical strengths

The breadth of the study. The authors investigate the method in three different domains and on five different models in total. This is much appreciated for understanding the generality of the claims made in the paper. Furthermore, I appreciate the honest reporting in the results sections. Where results are weak or inconsistent, the authors do not hide these issues but mention them often clearly.

Critical weaknesses

Missing a “takeaway”. This is the main weakness in the writing. The authors test a small modification to residual architectures, but performance does not increase consistently, nor can I observe a consistent interpretation. I would also appreciate a negative story, but it is hard to formulate a key takeway of this paper or what it would contribute to the literature after reading.

The gate cannot be read on its own. Since mm is a function of Δh\Delta h, a lower mm might just mean their updates are larger, and the gate compensates or vice versa. The authors introduce mm as some scaling of features of Δh\Delta h, but the overall scale could easily be absorbed into Δh\Delta h itself.

Limited insights from some of the interventional experiments. The neutralisation experiment does not provide the interpretation the authors claim. Neutralising the tokens with the largest gate magnitude means you make the largest change to the gate values post-hoc, while the group with gates closest to 1 is barely changed. Hence, one would assume that the top group gives the largest changes. This is, in my opinion, a major issue, since many of their language-model claims rely on this experiment. I see a similar issue with their post-hoc token scalar, branch scalar identity and shuffled experiments. Any trained network can break if you swap out a part it learned to rely on. So yes, these experiments show that the model depends on its learned gating structure, but such dependence is not surprising. I am missing an actual insight beyond this dependence. The authors do acknowledge the limitations, but the abstract and contributions still present it as a key result.

Limited novelty. Learned scaling of residual branch has already been investigated in previous works, and the methods closest to DyM are in my opinion not discussed. ReZero [1] is similar to the branch scalar condition in Table 4 but trained, and therefore a natural baseline. Gated Attention [2] applies an input-dependent sigmoid gate after attention, shows consistent performance gains at the 1.7B–15B scale, and analyzes the learned gating scores. Neither is cited or compared against, so the novelty of DyM is not clearly discussed.

Hence, given the limited insights provided by their experiments, a missing clear takeaway and limited novelty, I am concerned that the paper does contribute little to the current literature.

[1]: Bachlechner, Thomas, et al. "Rezero is all you need: Fast convergence at large depth." Uncertainty in artificial intelligence. PMLR, 2021.
[2]: Qiu, Zihan, et al. "Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free." Advances in Neural Information Processing Systems 38 (2026)

Questions for authors
  1. What is the main takeaway of the paper? In the conclusion, the authors write “DYM as a general mechanism for adaptive representation refinement”, however, how is this different from any residual layer or modification to it? To me, this is unfortunately just an obvious statement, not a conclusion. Same about “inspectable control variable that can be tested directly through intervention”, this is true for most “control variables”.
  2. Can the authors address the weaknesses of the neutralisation and gate-ablation experiments? These are key results of the paper, but, as they are performed, they can only support very limited conclusions.
  3. Could the authors discuss their contribution in the context of the existing work I have mentioned above? Specifically, what insights are gained beyond ReZero and Gated Attention?
1 comment

Thank you for the detailed review. We agree that the paper needed to state its central takeaway more sharply, and that the neutralization and ablation experiments support a narrower interpretation than a full causal reading of the gate values. At the same time, we believe the contribution is broader than generic residual scaling, and we have revised the manuscript to make this distinction clearer.

1. Main takeaway and gate interpretation

Our main takeaway is not that residual scaling exists, nor that DYM consistently improves predictive performance. The central finding is that learned regulation of residual updates develops structured, task-dependent behavior: computationally consequential representations may be either attenuated or amplified.

A standard residual layer determines what update to compute, but normally incorporates it with a fixed coefficient. DYM adds a separate, input-dependent control over how strongly that newly computed update is incorporated. This provides an explicit computational variable for studying update regulation rather than only the resulting representation.

We agree that the absolute numerical gate value should not be interpreted in isolation. Because the candidate update and gate are learned jointly, some overall scaling can be redistributed between them. Our analysis instead focuses on relative modulation within the same trained layer. The same DYM module, with shared parameters, is applied to all tokens in that layer, and differences across tokens reflect how the model differentially regulates their updates under a common modulation function. Thus, we do not interpret values such as 0.8 or 1.2 as absolute measures of feature importance. Rather, the gate reflects learned allocation of update strength across tokens, whose relation to feature relevance is evaluated empirically. We have clarified this interpretation in the revised manuscript.

This is also why we view DYM as potentially useful beyond post-hoc interpretation: a learned relevance-related signal can participate directly in computation and influence how newly computed information is incorporated during training and inference.

2. Neutralization and gate-ablation experiments

We agree that the neutralization analysis is confounded by intervention size. If utu_t is the candidate update and gtg_t its learned gate, resetting gtg_t to one changes the local update by (gt−1)⊙ut(g_t-1)\odot u_t. Tokens farther from identity can therefore receive larger perturbations. The current result supports the narrower conclusion that removing learned modulation from strongly modulated tokens has a larger effect than the tested position-matched controls; it does not establish that the ranking remains informative independently of perturbation magnitude. A stronger control would also match the norm of the induced update change. We have incorporated this qualification and narrowed the claim accordingly.

Likewise, the scalar, identity, and shuffled-gate experiments are inference-time diagnostics of what structure the trained DYM checkpoint relies on; they do not prove that DYM is uniquely necessary. The informative observation is that performance is sensitive not only to average residual scaling, but also to feature-wise structure and token assignment. We have revised the abstract and contribution statements so these experiments are presented as dependency analyses rather than stronger causal evidence.

3. Relation to ReZero and Gated Attention

We agree that both are important precedents. We have revised the related-work discussion to address them directly. ReZero learns a residual scaling parameter that is not input-dependent, making it conceptually closer to our branch-scalar condition. Gated Attention is closer to DYM because it uses an input-dependent gate on attention output.

We do not claim that DYM invents residual gating. The contribution is to use input-dependent gating specifically as update modulation and to study the resulting control signal across language, vision, structural biology, and medical segmentation through attenuation/amplification analysis, external annotations, perturbations, and direct interventions.

The Pythia-160M comparison is matched within our setup: Original and DYM were both trained from scratch on the same Pile data with the same tokenizer, schedule, and budget. Training used 4 GPUs, micro-batch 4/GPU, 8 gradient-accumulation steps, sequence length 2048, and 143,000 iterations, corresponding to approximately 37.5B tokens.

We have therefore sharpened the paper’s summary: DYM exposes residual-update strength as an explicit, input-dependent control signal. Across domains, this signal develops structured but task-dependent behavior, is not equivalent to conventional feature importance, and trained models become sensitive to its token- and feature-level structure.

Reviewer_njVaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript studies Dynamic Masking (DyM), a learned mechanism that regulates the incorporation of newly computed information into a residual representation. Its core operation multiplies a candidate residual update by an input-dependent coefficient in (0, 2), allowing attenuation and amplification. The implementation uses scalar gates for vision and structural biology, vector-valued gates for Pythia, and spatial block-output gates for medical segmentation. The experiments investigate predictive performance and the computational interpretation of the learned coefficients across four domains. The principal finding is that modulation has task-dependent structure: computationally consequential representations can receive either attenuation or amplification. The Pythia interventions additionally demonstrate that the trained model depends on token alignment and feature-specific gating.

Critical strengths
  1. Careful conceptual framing. The distinction between update regulation and feature importance is useful. The paper separates predictive performance, association with external annotations, and sensitivity to interventions, helping readers interpret each result appropriately.
  2. Informative intervention experiments. Across 200 held-out sequences, neutralizing the most strongly modulated tokens raises target NLL by 4.291 versus 2.645 for position-matched random tokens; the paired difference has a reported 95% interval of [1.352, 1.956]. The gate ablations support dependencies within the trained checkpoint. The ImageNet patch interventions cover 5,000 validation images.
  3. Breadth and transparent reporting. Four domains provide complementary tests, and the distinction between attenuation and importance is useful. Pythia-160M LAMBADA accuracy increases from 36.3% to 48.6%, with perplexity falling from 29.040 to 16.851, about 42.0%. This is a substantial reported gain. The authors candidly describe mixed outcomes elsewhere and the limits of the three-pair linguistic sanity check.
  4. Simple construction and explicit parameter accounting. The mechanism is straightforward to implement. Appendix A gives useful formulas and parameter counts, and the Pythia initialization preserves the original residual computation at initialization.
Critical weaknesses

1 The incremental contribution requires clearer positioning The related-work discussion covers broad gating approaches but omits close residual-gating precedents. GTrXL evaluates input-dependent gating of residual branch outputs, and GateSkip, published at ICLR 2026, gates attention and MLP contributions before residual addition. These precedents do not establish that the present empirical findings are already known. The paper should explain which specific observations or analyses add new knowledge and how conditioning on the update, permitting amplification, and the cross-domain study support that contribution. A focused mathematical and empirical positioning would help. [1, 2] The already-cited Liu et al. (2026) preprint introduces Dynamic Masking with sigmoid-linear modulation. Clarifying inherited components and biological results would make the present contribution easier to assess. This is an attribution and scope question; I do not require a new benchmark comparison to that preprint as a condition of acceptance. [3] 2 The training protocol is insufficiently specified to assess the comparisons The supplied PDF leaves essential questions unresolved: whether each backbone was trained from scratch or further trained from a checkpoint; which parameters were updated; the training data and budgets; optimization settings; checkpoint selection; and evaluation configurations. These details matter for reproducing both the performance results and the mechanistic patterns. In particular, the LAMBADA gain should be compared under matched data and training budgets. The PDF promises supplementary code and configurations; their absence from the material available to me limits this assessment, and does not establish their absence from the conference submission. Clear configurations and sampling details could resolve much of this concern. 3 The mechanistic interpretation would benefit from targeted controls Table 4 legitimately tests which structure one trained checkpoint uses, consistent with the authors’ explicit scope. It supports sensitivity to token assignment and hidden-dimension structure. A static vector for each branch, retaining dimension-specific calibration while averaging across tokens, would help separate content dependence from channel-specific scaling. The reported branch-scalar replacement discards both structures. A position-stratified gate shuffle would also help distinguish alignment to content from alignment to token position. These are targeted inference controls within the existing study. Comparisons among separately trained gate families would address a broader architectural question and are useful follow-on work. The extreme identity-ablation perplexity is compatible with coadaptation and is not, by itself, an implementation failure. 4 The neutralization analysis couples token ranking with intervention size Tokens are ranked by departure from the identity gate, and the intervention then resets their gates to identity. Consequently, selected groups can receive systematically different perturbation magnitudes. The results support the stated conclusion that resetting learned gates affects computation. To establish what the ranking reveals beyond sensitivity to that particular reset, it would help to match or condition on the actual induced update change, or to compare responses across controlled perturbation doses. This would refine the interpretation of the ranking; it does not negate the measured neutralization effects. 5 Robustness and the scope of cross domain evidence The ImageNet gain of 0.35 percentage points, mean BraTS Dice gain of 0.0013, and modest biological performance changes need seed uncertainty for claims of reliable performance improvement. These gains are secondary to the paper’s mechanistic contribution. More central is whether the reported spatial patterns hold across independent complexes and volumes; per-complex and per-volume estimates would address correlated residues and voxels. The BraTS implementation scales the complete Swin block output, whereas Equation (2) gates the candidate residual update. The authors disclose this distinction. BraTS therefore provides complementary evidence about spatial modulation, with a broader operational scope. Matching the implementation across domains would be a useful extension; I do not treat the disclosed difference as invalidating the segmentation observations.

Questions for authors
  1. What precisely is new relative to the DyM mechanism in Liu et al. (2026), and how does the contribution differ from GTrXL output gating and GateSkip?
  2. Can you provide the training and evaluation configurations, including checkpoint provenance, backbone updates, data budgets, baseline matching, and random seeds?
  3. Can a static per-dimension branch vector and a position-stratified gate shuffle clarify which aspects of content alignment are used by the trained checkpoint?
  4. Does the token-neutralization ranking remain informative after controlling for the magnitude of the induced update perturbation?
  5. How stable are the mechanistic patterns across independent complexes, volumes, and training runs? Please distinguish the support for these patterns from uncertainty in the smaller task-performance gains.
2 comments

Thank you for the careful and constructive review. We appreciate the reviewer’s recognition of the distinction between update regulation and feature importance. We agree that the positioning, training details, and scope of several mechanistic conclusions needed to be clearer, and we have revised the manuscript accordingly.

1. Contribution relative to Liu et al. (2026), GTrXL, and GateSkip

Liu et al. (2026) introduced Dynamic Masking in the antibody–antigen setting. The present work builds on that formulation by studying learned update modulation across multiple domains and by making the modulation itself a central object of analysis. The exact gate parameterization differs from the earlier formulation, but we view this as an implementation evolution rather than the main conceptual contribution.

We also agree that GTrXL and GateSkip are important residual-gating precedents. We have revised the related-work section to position DYM more explicitly with respect to these methods. Our contribution is not the generic use of a gate, but the study of an input-dependent mechanism that regulates how strongly newly computed information is incorporated into an existing representation. Across language, vision, structural biology, and medical segmentation, we examine attenuation versus amplification, relationships with external annotations, input perturbations, and direct interventions on the learned modulation.

We do not assume that a DYM coefficient is itself a universal importance score. Rather, DYM provides one way for a learned, input-dependent modulation signal to participate directly in model computation, while its relationship to feature relevance is evaluated empirically. This distinction is now stated more explicitly.

2. Training and evaluation configuration

Both Original and DYM Pythia models were trained from scratch using the same Pile dataset, original Pythia tokenizer, training schedule, and evaluation procedure.

For Pythia-160M, training used 4 GPUs, micro-batch 4/GPU, 8 gradient-accumulation steps, global batch 128, sequence length 2048, and 143,000 iterations, corresponding to approximately 37.5B training tokens. We used cosine learning-rate decay with 1% warmup. All Transformer parameters were trained end-to-end in both conditions, and Original received the same training budget as DYM.

The reported results use one full training run per condition. Because available computational and storage resources did not permit multiple full-scale runs, we cannot estimate training-seed variability. We have therefore made this limitation explicit and no longer treat the smaller performance differences as statistically established improvements.

3. Targeted controls and neutralization

We agree that a fixed per-layer/per-branch vector would better separate static hidden-dimension calibration from input dependence, because the current branch scalar removes both. Likewise, unrestricted gate shuffling changes both content and positional alignment; a position-stratified shuffle would provide a cleaner test.

We have therefore revised the interpretation of Table 4: the current ablations show that the trained checkpoint depends on token assignment and feature-wise gate structure, but they do not fully decompose those dependencies or establish superiority over independently trained gating alternatives.

For neutralization, if utu_t is the candidate update and gtg_t its learned gate, resetting the gate to identity changes the local update by (gt−1)⊙ut(g_t-1)\odot u_t. Because ranking by gate departure can correlate with perturbation magnitude, the current result supports the narrower conclusion that strongly modulated tokens are more consequential under the tested intervention than the position-matched controls. It does not establish that this ranking is independent of intervention magnitude. We have incorporated this qualification and narrowed the claim accordingly.

4. Robustness and cross-domain scope

We agree that the small ImageNet, BraTS, and biological performance changes are secondary to the mechanistic analysis. We have revised the manuscript to avoid presenting these small gains as robust improvements without repeated seeds.

Residues and voxels within the same complex or volume are also correlated, so independent-complex and independent-volume summaries would provide stronger uncertainty estimates. We have added this limitation to the discussion.

Finally, the BraTS wrapper scales the complete Swin block output, whereas Pythia, ViT, and structural biology modulate candidate residual updates. We have therefore revised the manuscript to interpret BraTS as complementary evidence about learned spatial modulation more generally, rather than as a direct test of the exact residual-update formulation in Eq. (2).

Thanks you for the reply of the Authors. I will consider during the next phase of discussion with the AC and reviewers whether I should improve the score.

Reviewer_pt8eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Dynamic Masking (DYM), which predicts input-dependent coefficients for modulating the incorporation of candidate updates into residual representations. Its principal form multiplies each candidate update by a coefficient between zero and two before residual addition. The implementation uses scalar gates for ViT and antibody antigen prediction, feature wise bottleneck gates for Pythia, and a distinct spatial block-output gate for Swin UNETR. The study evaluates language modeling, antibody-antigen interface prediction, ImageNet classification, and brain-tumor segmentation. Predictive effects are mixed: the largest reported gain is on Pythia-160M LAMBADA, while several other language metrics decrease and improvements elsewhere are modest. The main analytical contribution is the characterization of the learned control signals through external annotations, input perturbations, and direct gate interventions. The authors distinguish modulation from conventional feature importance and interpret the inference-time ablations as evidence about dependencies of the trained model.

Critical strengths
  1. A clear distinction between computational control and importance. The paper defines a directly observable coefficient with a specific local meaning and avoids treating it as a universal attribution score. It explicitly reports weak deletion based associations, non-monotonic unsigned patch ablation results, and mixed predictive effects. It also appropriately notes that averaged log coefficients are descriptive, and that post training ablations do not compare independently trained architectures. These qualifications make the analysis more credible.
  2. Intervention evidence beyond visual inspection. The study combines external spatial annotations with direct gate interventions and fixed-target input perturbations. The language analysis uses position-matched random controls and reports paired uncertainty estimates; the ImageNet perturbations hold the clean predicted class fixed. The modules are straightforward to integrate, and their parameter costs are explicitly derived. Together, these experiments provide evidence that the learned gates are functionally used by the evaluated checkpoints.
Critical weaknesses
  1. The training protocol and comparative contribution are not sufficiently established. The manuscript and appendix do not specify which Pythia checkpoints were used, whether DYM was trained from scratch or by continued training, the training corpus and token budget, which parameters were updated, or whether Original received the same training budget. Appendix A mainly reports parameter counts. These omissions make the LAMBADA improvement in Table 1 and the formation of the learned gate profiles difficult to assess. The other domains likewise need clearer training configurations, splits, and training-seed variability. The reproducibility statement mentions an anonymous code/configuration supplement; the relevant details should be provided or precisely located there. More broadly, the study lacks a matched trained residual-scaling or gating alternative. This matters because SE already modulates residual-branch outputs. DYM has identifiable differences, but the incremental value is not isolated by disrupting a single trained checkpoint.
  2. The interventions do not fully isolate the adaptive structure behind gate dependence. In Table 3, tokens are ranked by gate departure from identity and then their gates are reset to identity. The ranking is therefore coupled to the size and structure of the intervention. Matching token position does not match the norm of the induced hidden-state change, which depends on both the gate and the candidate update. Consequently, the top-versus-random and attenuation versus amplification differences need a perturbation matched control. In Table 4, replacing a vector gate by one scalar per branch removes both input dependence and static hidden dimension specific scaling. A fixed per-branch, per-dimension vector would retain the latter while removing the former. Token shuffling additionally changes positional alignment, which matters given the reported position correlation. I agree that the existing interventions demonstrate dependency of the trained checkpoint; stronger controls are needed to identify the content-adaptive mechanism responsible for that dependency.
Questions for authors
  1. Were Original and DYM trained from the same initialization or checkpoint on the same data, number of updates, and optimization schedule? Please specify the trained parameters, corpus/token budget, evaluation setup, and seed counts, or identify the exact supplementary configurations. For the small gains outside LAMBADA, what variability is observed across training runs?
  2. How much performance is retained by a fixed gate vector for each layer and branch, averaging over examples and tokens but retaining the hidden dimension? Do the selective-neutralization effects persist after matching the norm of the actual hidden-state perturbation as well as position? A position-matched shuffle would further clarify the token alignment result.
  3. Which difference from SE-style recalibration, LayerScale, and attention-output gating is central to the contribution: conditioning on the candidate update, gate granularity, or the resulting empirical analysis? Can one representative trained alternative under the same protocol clarify this? Since the BraTS wrapper gates complete block outputs, please also delimit which cross domain conclusions concern residual-update-only modulation and which concern modulation more generally.
1 comment

Thank you for the careful review. We agree that the training protocol and the scope of some mechanistic claims needed to be clearer, and we have revised the manuscript accordingly.

  1. Training protocol and Original vs. DYM

Both Original and DYM Pythia models were trained from scratch using the same Pile dataset, original Pythia tokenizer, schedule, and evaluation procedure. DYM was not added by continued training of a pretrained Original model.

For Pythia-160M, training used 4 GPUs, micro-batch 4/GPU, 8 gradient-accumulation steps, global batch 128, sequence length 2048, and 143,000 iterations, corresponding to ~37.5B training tokens. We used cosine LR decay over all 143,000 iterations with 1% warmup. All model parameters were trained end-to-end in both conditions, and Original received the same training budget as DYM.

We have one full training run per condition. Because of computational and storage constraints, we cannot estimate training-seed variability for the smaller gains, so these should not be interpreted as statistically established improvements. We have made this limitation explicit in the revised manuscript.

  1. Scope of the intervention results

We agree that the current interventions show that the trained checkpoint depends on learned gate structure but do not fully isolate which component is responsible.

The branch-scalar replacement removes both input dependence and hidden-dimension-specific scaling. A fixed per-layer/per-branch vector averaged across examples and tokens would separate these effects more cleanly. Likewise, unrestricted gate shuffling changes both content and positional alignment; a position-stratified shuffle would be a stronger control. We have therefore revised the manuscript to interpret these experiments as dependency analyses of the trained checkpoint rather than as a complete decomposition of the adaptive mechanism.

For selective neutralization, resetting gate gtg_t to identity changes the local update by (gt−1)⊙ut(g_t-1)\odot u_t. Ranking by gate departure can therefore correlate with intervention magnitude. We have narrowed the corresponding claim: strongly modulated tokens are more consequential under the tested neutralization than the position-matched controls, but the current experiment does not show that this ranking is independent of perturbation magnitude.

  1. Relation to SE, LayerScale, and other gating methods

We agree that gating itself is not new. We have revised the related-work discussion to make this distinction explicit. The distinction we emphasize is using a learned gate to regulate how strongly newly computed updates are incorporated into an existing representation. DYM therefore treats gating as an input-dependent update-control mechanism rather than only feature recalibration, channel selection, or static residual scaling.

This also motivates the connection to feature importance. We do not assume that the DYM coefficient is itself a universal importance score. Instead, DYM provides one way for a learned, input-dependent modulation signal to participate directly in model computation; its relationship to feature relevance is then tested empirically rather than assumed. This distinction is now stated more explicitly in the revised manuscript.

The current inference-time ablations address what structure the trained DYM checkpoint uses. They are not intended to establish superiority over independently trained SE-, LayerScale-, or other gating alternatives; that is a broader architectural comparison. We have clarified this scope in the revised text.

Finally, the BraTS wrapper scales the complete Swin block output, whereas Pythia, ViT, and structural biology modulate candidate residual updates. We have therefore revised the manuscript to interpret BraTS as evidence about learned spatial modulation more generally, rather than as a direct test of the exact residual-update formulation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.