acceptodds
Under review as a conference paper at ICLR 2027

No Channels Left Behind: Understanding Diffusion Acceleration through Residual Gates

Abstract

Training acceleration methods for diffusion transformers, such as REPA and dispersive regularization, are commonly explained by additional supervision, spatial-structure priors, or improved feature geometry. We revisit this view and uncover a striking internal change: the AdaLN-Zero residual gate , which directly controls how conditioned updates enter the residual stream, is substantially reorganized by these methods. This observation motivates our hypothesis that acceleration arises, at least in part, from reshaping the residual gating pathway rather than solely from the feature structures explicitly imposed by auxiliary objectives. Supporting this view, accelerated models are markedly more sensitive to perturbations of , and their gate geometry reflects semantic relationships among classes. We further introduce the normalized participation ratio (nPR) to characterize the channel-wise organization of the gates and find that, across REPA teachers, projector architectures, and radial-objective targets, broader gate participation accompanies better generation quality. Guided by this insight, we propose a simple teacher-free radial objective that promotes broader gate participation through feature-norm regularization, without pairwise feature comparisons. On ImageNet with SiT-B/2, our method improves FID over dispersive loss while increasing mean gate participation over timesteps and classes, further supporting the link between gate organization and training acceleration. Our results identify residual-gate reorganization as a measurable axis of diffusion training acceleration and suggest a simple direction for further acceleration.

Primary area:
generative models
Keywords:
diffusion transformer, training acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p5K9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper calls for the community to pay attention to the gate participation in Diffusion Transformer. It motivates its direction from the finding that the REPA performance and other models are sensitive to its learned gate distribution. Based on the finding that nPR, which the authors use to measure the breadth of participations of the channels, the paper device a regularization loss that indirectly promotes nPR. They demonstrate the efficacy of this objective on benchmark datasets, and claims strength especially on teacher-free domain.

Critical strengths

The paper focuses on hindsight that can possibly explain the strength of the traububg accelration method sfor diffusion transformers, and provides several perspectives of their measured quantities to the perforamnce as well as for token representation (Section 3). It is an insight that warrants further investigations.

Critical weaknesses

I feel that there are several links that are missing, and that the works seems incomplete, remaining in the scope of identifying that turned out to work well. Please see the question sections.

Questions for authors

I am not convinced with the necessity of this objective and I think my concern warrants ablations. Firstly, this papers' nPR seems to be a case of Renyi-2 entropy of

p_d = \frac{\alpha^2_d}{\sum_{j} alpha_j^2

I wonder how much of this study is specific to this mode of entropy. How is the other forms of entropy related to the performance?

In a related note, I would like to know better why not much is mentioned regarding the direct regularization of entropies for α\alpha. For example, what would happen if the loss as below was used?
$L_{velocity}(\theta) + \lambda E_{t, Y} [KL(p(t, Y)) | Unif(D) ] $

In a similar note regarding other means of controlling the broad participation, what would happen if Muon-like optimizer was used for UU in channel direction?

Also, it is being sated that acceleration methods reorniaze the residual gates, but is this reorganization, relative to Vanilla, an increase in nPR?

4 comments
AuthorsOpenReview ↗

Response to reviewer p5K9

We appreciate your careful review and constructive feedback, as well as your positive assessment of our contributions.

Q0: I am not convinced with the necessity of this objective and I think my concern warrants ablations.

A: We clarify that the Radial objective is intended as an empirical study supporting the interpretation developed in our analysis, rather than as a necessary or unique objective. It tests whether this interpretation can guide a teacher-free design that increases nPR while improving FID. We view these results as supporting the practical value of the design principle, not as ruling out alternative objectives.

Q1: Firstly, this papers' nPR seems to be a case of Renyi-2 entropy of pd=αd2∑jαj2p_d=\frac{\alpha^2_d}{\sum_j \alpha^2_j}. I wonder how much of this study is specific to this mode of entropy. How is the other forms of entropy related to the performance?

A: Thank you for highlighting this connection. As you suggest, nPR is a normalized, exponentiated form of collision entropy: nPR=exp⁡(H2(p))/D\mathrm{nPR}=\exp(H_2(p))/D, where pd=αd2/∑jαj2p_d=\alpha_d^2/\sum_j\alpha_j^2.

Recomputing the Pearson correlations with FID in Fig. 6(a), we obtain r=−0.9840r=-0.9840 for Rényi-2 entropy (95% CI: [−0.9991,−0.9341][-0.9991,-0.9341]) and r=−0.9740r=-0.9740 for Shannon entropy (95% CI: [−0.9974,−0.8559][-0.9974,-0.8559]). Thus, the strong negative association persists with both entropy measures, suggesting that the observation in this sweep is not specific to nPR’s particular formulation. We have included this connection and comparison in the revision.

Q2: In a related note, I would like to know better why not much is mentioned regarding the direct regularization of entropies for α\alpha. For example, what would happen if the loss as below was used? Lvelocity(θ)+λEt,Y[KL(p(t,Y))∥Unif(D)]L_{\text{velocity}}(\theta)+\lambda E_{t,Y}[KL(p(t, Y)) \| Unif(D)]

A: Direct entropy regularization is a natural alternative, since KL(p∥Unif(D))=log⁡D−H1(p)\mathrm{KL}(p\|\mathrm{Unif}(D))=\log D-H_1(p). Following your suggestion, optimizing this KL objective improves Vanilla’s FID from 45.41 to 44.94. Directly optimizing nPR, the normalized exponentiated Rényi-2 entropy, improves FID to 44.89. Further tuning may yield additional gains.

However, as discussed in our response to Reviewer voVE’s Q1, higher participation alone is not sufficient: fixing α=1\alpha=\mathbf{1} maximizes participation but prevents gate reorganization during learning and worsens REPA’s FID. This fixed-gate ablation is distinct from soft regularization, but highlights the importance of learned gating beyond maximal participation alone.

Q3: In a similar note regarding other means of controlling the broad participation, what would happen if Muon-like optimizer was used for UU in channel direction?

A: Thank you for this insightful suggestion. To examine whether applying Muon to the residual branches producing UU could increase gate participation, we retained AdamW for the AdaLN modulation parameters. Muon used Nesterov momentum of 0.95, five Newton–Schulz iterations, and no weight decay. Updates were scaled by 0.2max⁡(m,n)0.2\sqrt{\max(m,n)} [1], allowing us to reuse the constant AdamW learning rate of 10−410^{-4} without tuning. All other settings followed Tab. 1 (100K steps, batch size 256). Replacing AdamW with Muon increases nPR from 0.33 to 0.696 and improves FID from 45.41 to 33.23. This provides an additional optimizer-based example consistent with the association between broader gate participation and improved generation quality. However, Muon’s update magnitudes at the measured layer are 1.21.2-1.71.7 times those of AdamW. The FID improvement may therefore also reflect differences in update scale and optimization dynamics, rather than isolating the contribution of broader participation.

Q4: Also, it is being stated that acceleration methods reorganiaze the residual gates, but is this reorganization, relative to Vanilla, an increase in nPR?

A: Yes. The gate reorganization we highlight involves increased nPR relative to Vanilla, indicating broader gate participation. The Vanilla model trained under the same conditions as Fig. 6(a) has an nPR of 0.036, while the Vanilla baseline for Fig. 6cc is already shown in the figure.

[1] Liu, Jingyuan, et al. "Muon is scalable for llm training." arXiv preprint arXiv:2502.16982 (2025).

We appreciate your time and constructive feedback. We hope our additional analyses and experiments help address your concerns. If you have any further thoughts, we would be grateful to hear whether any concerns remain or whether further clarification would be helpful.

Reviewer_p5K9OpenReview ↗

Thank you for the response

Thank you for addressing my concern. It is to be hoped that the additional studies done in our discussion would be reflected in the revised / camera ready version. I keep my positive score as is.

AuthorsOpenReview ↗

Thank you for your positive and constructive feedback

We appreciate your careful review and continued positive assessment.

We have incorporated the constructive feedback from our discussion into the revised manuscript.

Reviewer_QrirOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper starts with a key observation that diffusion training acceleration methods tend to make the AdaLN-Zero residual gate α\alpha distribute more evenly across different channels after averaging over classes and timesteps. Then, the authors confirm that there is a positive correlation between how even the distribution is and the generation quality. Finally, a new acceleration method is proposed to directly promote an even distribution without relying on external supervision, which leads to improvement.

Critical strengths
  1. This paper does provide a very interesting and insightful observation. The residual gate α\alpha might be a key to revealing a deeper mechanism behind REPA-like methods.
  2. Many highly detailed empirical results are provided.
Critical weaknesses
  1. The authors rush too much to derive a novel generation acceleration method, leaving the mechanism not fully investigated. Therefore, the potential impact and insights of this paper are limited. In my opinion, the community does not need more acceleration methods. What is truly meaningful is to understand why the existing methods can work. This paper observes correlation between residual gate α\alpha, acceleration methods, and generation quality, but stops there. Two important questions remain not answered: why does an acceleration method make α\alpha distribute more evenly, and why does a more even distribution improve generation (or are these things really causally linked)?
  2. The final experiment design is not informative enough. It simply demonstrates the new method achieves SOTA in the teacher-free family. A more informative design would be to carefully tune the hyper-parameter (potentially requiring per-iteration hyper-parameter setting), so that the nPR values of Radial and REPA align at each training step, isolating the excess gain from the external feature itself. Even within the teacher-free family, aligning the nPR values of Radial and Dispersive and seeing if the generation quality also aligns might provide more meaningful information than simply outperforming Dispersive.
Questions for authors

No.

6 comments
AuthorsOpenReview ↗

Response to reviewer Qrir

We appreciate your careful review and constructive feedback, as well as your positive assessment of our contributions.

W1: The authors rush too much to derive a novel generation acceleration method, leaving the mechanism not fully investigated. Therefore, the potential impact and insights of this paper are limited. In my opinion, the community does not need more acceleration methods. What is truly meaningful is to understand why the existing methods can work. This paper observes correlation between residual gate, acceleration methods, and generation quality, but stops there. Two important questions remain not answered: why does an acceleration method make distribute more evenly, and why does a more even distribution improve generation (or are these things really causally linked)?

A: We appreciate the reviewer's emphasis on understanding why existing acceleration methods work, which is also a central motivation of our study. Our primary goal is to provide mechanistic insights into accelerated diffusion models, with Radial serving as a practical demonstration of how these insights can inform model design.

Our interpretation is that auxiliary objectives in acceleration methods encourage the model to learn useful representations across a broader set of residual channels, which leads to a more distributed allocation of learned gates. Because α\alpha controls channel-wise residual contributions, broader gate participation may reflect greater use of these features and less reliance on a few channels. This may explane for both the gate redistribution and its association with generation quality, without implying that uniformity alone improves generation.

The recurring nPR–FID association across our sweeps supports this interpretation. Beyond this, our additional experiments provide direct evidence for the functional importance of learned gating. In our 100K-step ablation, fixing α=1\alpha=\mathbf{1} worsens REPA’s FID from 27.17 to 35.32, while barely changing Vanilla’s (45.41 to 45.57). The results demonstrate that REPA's improvements depend in part on adaptive gating, rather than simply higher gate participation.

We believe these findings offer useful insights into how acceleration methods influence learned gating behavior and its relationship with generation quality. We share the reviewer's view that understanding these mechanisms is important, and hope our work contributes to further progress in this direction.

W2: The final experiment design is not informative enough. It simply demonstrates the new method achieves SOTA in the teacher-free family. A more informative design would be to carefully tune the hyper-parameter (potentially requiring per-iteration hyper-parameter setting), so that the nPR values of Radial and REPA align at each training step, isolating the excess gain from the external feature itself. Even within the teacher-free family, aligning the nPR values of Radial and Dispersive and seeing if the generation quality also aligns might provide more meaningful information than simply outperforming Dispersive.

A: We agree that comparing methods at matched nPR is informative. Matching nPR throughout training would require additional trajectory-controlled experiments; our existing teacher-free results provide a complementary endpoint comparison.

At 100K steps, Radial with k=8k=8 and Dispersive have nearly identical nPR (0.5243 vs. 0.5252) and similar FID (40.02 vs. 40.43), despite using different objectives. Moreover, a linear fit using only the Radial target sweep predicts Vanilla’s and Dispersive’s FID as 45.75 and 40.39, close to their observed values of 45.41 and 40.43.

These results suggest that the nPR–FID relationship extends beyond Radial objective, rather than merely reflecting its target sweep.

It's good to see that the authors also intend to dig deeper into the mechanism. I read the responses to other reviewers and see that the authors also regard the proposed loss as an experimental tool rather than a directly practical method contribution. After all, the title also suggests a diagnosis-centric paper.

In this case, I feel the original manuscript could be seen as suffering from a presentation problem that gives readers a feeling it's making practical method contribution, which is easier to fix. I hope the authors could refine the writing such that more in-depth diagnosis could be done. Also, the new observations and results obtained during the rebuttal period might be worth including in the manuscript.

That said, since the refined manuscript couldn't be reviewed, the refinement quality couldn't be technically guaranteed. Therefore, I feel it might be suitable if I keep my rating unchanged. In this way, the average rating is still positive, but more slightly.

Wait... I realized ICLR does allow revising the PDF. I will update my followup response later.

We thank the reviewer for the thoughtful follow-up and constructive suggestions. We appreciate your understanding of our intended contribution and your helpful suggestions for improving the paper’s presentation.

Following your feedback, we have carefully revised the manuscript to better reflect its intended contributions. Specifically:

  1. We have revised the manuscript throughout to place greater emphasis on mechanistic understanding rather than practical method contributions. In particular, we have refined the Abstract, Introduction, Sections 3–5, and Conclusion to highlight residual-gate analysis as our central contribution, with nPR serving as a diagnostic measure and Radial illustrating how these insights can guide objective design..
  2. We have incorporated additional analyses from the rebuttal, including the ablations of learned residual gating, and clarified the interpretation and limitations of our findings, particularly the distinction between correlation and causation..

We hope these revisions better convey the intended contributions of our work, and we sincerely appreciate your willingness to consider the updated manuscript. We would also greatly appreciate any further suggestions and would be happy to carefully address them in further revision.

Reviewer_voVEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the underlying mechanism of recent training acceleration methods for diffusion transformers. Moving beyond standard representation-level explanations, the authors discover that acceleration is deeply tied to the reorganization of the AdaLN-Zero residual gates (α\alpha). They introduce the Normalized Participation Ratio (nPR) to quantify how broadly gate magnitudes are distributed across channels, revealing a strong correlation between higher nPR and lower FID. Motivated by this insight, the authors propose a simple, teacher-free, and pairwise-comparison-free Radial Objective. This objective penalizes the deviation of sample-wise flattened feature norms from a target constant, which mathematically (under certain symmetry assumptions) encourages higher gate participation. Empirical validations on ImageNet 256x256 using SiT-B/2 show that the Radial objective improves FID over the Vanilla baseline and the Dispersive loss, while consistently increasing nPR.

Critical strengths
  1. A new, cheap, well-defined diagnostic. nPR is computed from the conditioning pathway alone, with no image forward pass. Its association with FID recurs across three different kinds of sweeps, including a fixed-teacher projector sweep that removes teacher quality as a confounder. The finding that accelerated models differ drastically in gate scale and allocation is new and worth knowing. A cheap correlate of generation quality is useful even if it is not causal.
  2. A simple teacher-free objective with a real gain, carefully documented. Radial needs no teacher and no pairwise comparisons. It improves over a properly configured dispersive baseline by about 1.4 FID at 400K steps. The paper is also unusually explicit about protocol details and about the limited scope of its theory.
Critical weaknesses
  1. The mechanistic claim outruns the evidence. The abstract and Sec. 2 ("training acceleration is governed primarily by α") suggest a mechanism. However, every piece of evidence is either correlational or fails to isolate gate participation:
    • Small, possibly confounded correlations. The correlations rest on n = 8, 4 and 5 points. With n = 4, r = −0.99 carries little statistical weight, and no confidence intervals or rank correlations are reported. Each sweep also admits a plausible common cause. For example, in the projector sweep a weaker projector forces the diffusion features themselves to absorb more of the alignment. That could lower FID and change the gates independently.
    • Pilot 1 does not isolate α. It removes α, β and γ together and changes the architecture.
    • Pilot 2 is confounded by gate scale. Its perturbations are matched relative to ‖α‖, but REPA's gates are roughly 16× larger than Vanilla's. A matched δ is therefore a much larger absolute change to the residual stream, so higher sensitivity may follow from gate scale alone. Moreover, the uniformizing direction does not improve FID (Sec. 5), whereas a causal role for participation would predict that it does.
  2. The Radial evaluation is too thin to support a method contribution.
    • Narrow comparison. The comparison uses a single model size (SiT-B/2), a single training length and a single CFG scale. There is no FID-vs-steps curve, despite the "acceleration" framing.
    • Target selection and saturation. The target c was chosen by lowest FID under the same evaluation protocol. The gain saturates (FID 39.43 vs. 39.44 for k = 10 vs. 14), and the lowest target (45.72) is worse than Vanilla (45.41).
    • Inconsistent λ. The λ in the 400K run (5×10⁻⁸, App. C.2) differs from the sweep value (6.032×10⁻⁸, App. C.1) without explanation.
Questions for authors
  1. Please report results with an explicit nPR(α) regularizer (both increasing and decreasing it), or with gate allocation fixed, for Vanilla and REPA.
  2. For Pilot 2, please add three controls: perturbations matched in absolute magnitude (or relative to ‖H‖); random-direction perturbations of equal norm; and analogous perturbations of β and γ.
  3. Please explain the λ discrepancy between App. C.1 and C.2. Was target selection done on a held-out evaluation, or on the reported FID?
5 comments
AuthorsOpenReview ↗

Response to reviewer voVE (1/3)

We appreciate your careful review and constructive feedback, as well as your positive assessment of our contributions.

W1: The mechanistic claim outruns the evidence. The abstract and Sec. 2 ("training acceleration is governed primarily by α\alpha") suggest a mechanism. However, every piece of evidence is either correlational or fails to isolate gate participation:

A: We thank the reviewer for raising this important distinction. Our goal is to provide mechanistic insight into the role of α in accelerated models, rather than to claim that α alone causally determines acceleration.

Importantly, our evidence goes beyond correlation. For example, in Fig. 2, we directly perturb α while keeping all other model parameters fixed. The greater sensitivity observed in accelerated models demonstrates that their learned gates are functionally consequential, rather than merely correlated with generation quality. While this intervention does not isolate every aspect of gating behavior, it provides direct experimental evidence that α plays a significant functional role in these models.

Together with the consistent relationship between nPR and FID across different acceleration methods, these findings motivate our Radial method, which translates our mechanistic observations into empirical improvements. We therefore view nPR as a useful diagnostic of gating behavior, whose consistent association with FID across different acceleration methods provides empirical guidance for model design.

Finally, we agree that the phrase "governed primarily by α" is too strong and have revised it to better reflect our findings that emphasize the functional importance of α without implying that its contribution has been fully isolated from other factors.

W1-a: Small, possibly confounded correlations. The correlations rest on n=8,4n = 8, 4 and 55 points. With n=4n = 4, r=−0.99r = −0.99 carries little statistical weight, and no confidence intervals or rank correlations are reported. Each sweep also admits a plausible common cause. For example, in the projector sweep a weaker projector forces the diffusion features themselves to absorb more of the alignment. That could lower FID and change the gates independently.

A: We agree that the small sweeps warrant caution, but we believe the observed associations are informative. The calculated Pearson confidence intervals are [−0.99,−0.80][−0.99, −0.80], [−0.99,−0.61][−0.99, −0.61], and [−0.99,−0.18][−0.99, −0.18] for the three sweeps, with corresponding Spearman correlations of −1.0, −1.0, and −1.0. All three intervals exclude zero, and the Spearman correlations further support the consistency of the observed trends. We have reported these statistics to make the strength and uncertainty of the associations explicit in the revised version.

As clarified above, we do not interpret the nPR–FID association as establishing a direct causal effect. A weaker projector could influence both quantities, as the reviewer suggests; we believe this explanation is compatible with nPR serving as a diagnostic of gating behavior associated with generation quality. The association also recurs beyond the projector sweep, supporting its diagnostic relevance across the examined settings.

AuthorsOpenReview ↗

Response to reviewer voVE (2/3)

W1-b: Pilot 1 does not isolate α\alpha. It removes α\alpha, β\beta and γ\gamma together and changes the architecture.

A: Pilot 1 examines the modulation pathway as a whole; as noted in L146–147, Pilot 2 studies α\alpha directly. To further isolate the contribution of α\alpha gating from shift/scale, we have conducted two additional training ablations under the Table 1 setting for 100K steps: fixing α=1\alpha = \mathbf{1}, or fixing β=0\beta = \mathbf{0} and γ=1\gamma = \mathbf{1}, while retaining the other modulation components.

Fixing α\alpha changes FID from 45.41 to 45.57 for Vanilla, but substantially degrades REPA's FID from 27.17 to 35.32. In contrast, fixing β/γ\beta/\gamma yields 47.29 and 29.54, respectively. Thus, fixing α\alpha substantially reduces REPA’s advantage over Vanilla, whereas fixing β/γ\beta/\gamma largely preserves it. These controlled ablation results provide direct evidence for the important role of learned α\alpha gating in REPA’s performance gains, beyond the modulation-pathway evidence provided by Pilot 1.

W1-c: Pilot 2 is confounded by gate scale. Its perturbations are matched relative to ∥α∥\|\alpha\|, but REPA's gates are roughly 16× larger than Vanilla's. A matched δ\delta is therefore a much larger absolute change to the residual stream, so higher sensitivity may follow from gate scale alone. Moreover, the uniformizing direction does not improve FID (Sec. 5), whereas a causal role for participation would predict that it does. Q2: For Pilot 2, please add three controls: perturbations matched in absolute magnitude (or relative to ∥H∥\|\text{H}\|); random-direction perturbations of equal norm; and analogous perturbations of β\beta and γ\gamma.

A: We appreciate the reviewer's careful observation regarding gate scale. The actual change to the residual stream also depends on the residual branch activations, so differences in gate scale alone do not determine the effective perturbation magnitude.

To address the concern, we have conducted additional experiments with absolute-magnitude-matched perturbations, including random-direction controls for α\alpha, β\beta, and γ\gamma. Even under matched absolute magnitude, concentration-directed and uniformizing α\alpha perturbations produce larger FID changes (increases of 0.290 and 0.074, respectively) than random α\alpha perturbations, while random perturbations of α\alpha, β\beta, and γ\gamma yield comparatively small changes (increases of 0.013, 0.029, and 0.020, respectively). These results suggest that the sensitivity to learned α\alpha depends on the perturbation direction, rather than simply its magnitude, and is not specific to the relative-magnitude normalization used in the original experiment.

AuthorsOpenReview ↗

Response to reviewer voVE (3/3)

W2: The Radial evaluation is too thin to support a method contribution.

A: The Radial objective is intended primarily as a constructive test of the design principle suggested by our analysis, rather than solely as a standalone acceleration method. It provides another way to increase nPR, alongside the existing methods we study, while achieving lower FID. These results offer additional empirical support for our interpretation and demonstrate its practical value, without establishing nPR as a causal driver of generation quality.

W2-a: Narrow comparison. The comparison uses a single model size (SiT-B/2), a single training length and a single CFG scale. There is no FID-vs-steps curve, despite the "acceleration" framing.

A: Our comparisons use the same fixed CFG scale across methods to ensure a matched evaluation protocol. We acknowledge that this does not establish robustness across CFG scales. We have added FID-versus-training-step curves to the revised version to better assess acceleration. XL-scale training is also underway, and we will report the results once training and evaluation are complete.

W2-b: Target selection and saturation. The target cc was chosen by lowest FID under the same evaluation protocol. The gain saturates (FID 39.43 vs. 39.44 for k = 10 vs. 14), and the lowest target (45.72) is worse than Vanilla (45.41). Q3-b: Was target selection done on a held-out evaluation, or on the reported FID?

A: The target hyperparameter cc was selected based on preliminary 100K-step experiments evaluated on held-out data and fixed before the 400K-step experiments. Importantly, no further hyperparameter selection or tuning was performed based on the final reported 400K FID results. We have clarified this procedure in the revised paper.

Importantly, the improvements are not limited to the selected target. All tested settings with k≥8k \geq 8 improve over Vanilla (FID 40.02/39.43/39.44 vs. 45.41), indicating gains across multiple targets, with saturation at larger values. The smallest target yields both worse FID and the lowest nPR (0.381), consistent with the observed association. One possible explanation is that smaller targets provide weaker outward pressure at a given sub-target feature norm and switch to inward pressure at a lower norm, potentially limiting early feature expansion. We regard this as a possible mechanism rather than an established explanation.

W2-c: Inconsistent λ\lambda. The λ\lambda in the 400K run (5×10⁻⁸, App. C.2) differs from the sweep value (6.032×10⁻⁸, App. C.1) without explanation. Q3-a: Please explain the λ\lambda discrepancy between App. C.1 and C.2. Was target selection done on a held-out evaluation, or on the reported FID?

A: For the sweep (App. C.1), we set λ=6.032×10−8\lambda = 6.032 \times 10^{-8} based on the gradient-norm ratio. For the 400K experiment (App. C.2), we simplified this to λ=5×10−8\lambda = 5 \times 10^{-8} for ease of presentation and retrained from scratch, rather than continuing a sweep checkpoint. We have clarified this change in the paper.

Q1: Please report results with an explicit nPR(α)\mathrm{nPR}(\alpha) regularizer (both increasing and decreasing it), or with gate allocation fixed, for Vanilla and REPA.

A: We thank the reviewer for this constructive suggestion. Direct optimization of nPR(α)\mathrm{nPR}(\alpha) through an additional loss term increases nPR\mathrm{nPR} to 0.99 and improves Vanilla's FID from 45.41 to 44.89. Further tuning may yield larger gains.

To investigate the role of gate allocation, we conducted an additional 100K-step ablation for both Vanilla and REPA, fixing α=1\alpha=\mathbf{1} throughout training while retaining the remaining modulation components.

However, enforcing maximal gate participation does not reproduce the benefits of learned gating. Fixing α=1\alpha=\mathbf{1} gives maximal participation but prevents such reorganization. This worsens REPA’s FID from 27.17 to 35.32, while leaving Vanilla nearly unchanged (45.41 to 45.57). These results suggest that how gates are learned matters beyond simply maximizing their participation.

We therefore interpret higher nPR as a signature of gate reorganization during learning, rather than a standalone target whose maximization necessarily improves generation. We further believe that fixing α\alpha does not isolate the causal effect of nPR itself, it provides complementary evidence for the importance of learned gating behavior though.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

Dear Area Chair,

We appreciate your time and effort in managing the review process. We would like to summarize the strengths recognized by the reviewers, the discussion outcomes, and the revisions made in response to their suggestions. Our revised manuscript emphasizes residual-gate understanding as the central contribution, with nPR serving as a diagnostic and the Radial objective illustrating how these insights can guide objective design.

Strengths recognized by the reviewers

  • Reviewer Qrir described our residual-gate observation as “very interesting and insightful,” while Reviewer p5K9 recognized it as an insight warranting further investigation.
  • Reviewer Qrir acknowledged the detailed empirical results. Reviewer voVE specifically noted: “The paper is also unusually explicit about protocol details and about the limited scope of its theory.”
  • Reviewer voVE highlighted nPR as a “new, cheap, well-defined diagnostic,” computed without an image forward pass, and recognized the value of its recurring association with FID across different sweeps.

Updated reviewer feedback

Following the discussion, all three reviewers currently assign a rating of 3 (Weak acceptance).

  • Reviewer voVE: Rating 3, Confidence 3. On 18 November, the reviewer stated: “Most of my concerns have been solved. I will keep my positive score as is.”

  • Reviewer Qrir: Rating 2 → 3, Confidence 3. On 19 November, after reviewing the revised manuscript, the reviewer noted: “This version presents its diagnostic focus more clearly, which is a good sign.” The reviewer also confirmed: “I've raised my rating.”

  • Reviewer p5K9: Rating 3, Confidence 2. On 19 November, the reviewer wrote: “Thank you for addressing my concern” and “I keep my positive score as is.”

Main manuscript revisions

  1. Clarified contribution and claim scope: We revised the Abstract, Introduction, Sections 2–5, and Conclusion to emphasize mechanistic understanding rather than standalone method contributions, and clarified the distinction between association and causal evidence.

  2. Added controlled training ablations: Section 3.1 and Table 1 now separately examine fixed residual gates and fixed shift/scale modulation. Fixing the gates substantially reduces REPA’s advantage over Vanilla, whereas fixing shift and scale largely preserves it.

  3. Expanded statistical reporting: Appendix A.3 and Table A report Pearson confidence intervals and Spearman correlations for the teacher and projector sweeps and the target sweep for the Radial objective.

  4. Added an entropy perspective: Appendix E and Table B explain the connection between nPR and Rényi-2 entropy and show that the strong negative association with FID also persists for Shannon entropy in the teacher sweep.

  5. Clarified training protocols: Appendices C.1–C.2 explain target selection using preliminary 100K-step evaluations and the auxiliary-loss coefficient used for the separate 400K-step experiment.

  6. Extended evaluation across training steps and model size: Appendix C.3 (Fig. D) adds FID-versus-training-steps curves. Appendix C.4 (Fig. E) reports XL-scale training dynamics and the 100K-step result: the Radial objective achieves FID 37.18 versus Vanilla’s 43.51.

We appreciate the reviewers’ constructive engagement and your consideration of the revised manuscript and discussion results.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.