- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper calls for the community to pay attention to the gate participation in Diffusion Transformer. It motivates its direction from the finding that the REPA performance and other models are sensitive to its learned gate distribution. Based on the finding that nPR, which the authors use to measure the breadth of participations of the channels, the paper device a regularization loss that indirectly promotes nPR. They demonstrate the efficacy of this objective on benchmark datasets, and claims strength especially on teacher-free domain.
Critical strengths
The paper focuses on hindsight that can possibly explain the strength of the traububg accelration method sfor diffusion transformers, and provides several perspectives of their measured quantities to the perforamnce as well as for token representation (Section 3). It is an insight that warrants further investigations.
Critical weaknesses
I feel that there are several links that are missing, and that the works seems incomplete, remaining in the scope of identifying that turned out to work well. Please see the question sections.
Questions for authors
I am not convinced with the necessity of this objective and I think my concern warrants ablations. Firstly, this papers' nPR seems to be a case of Renyi-2 entropy of
p_d = \frac{\alpha^2_d}{\sum_{j} alpha_j^2
I wonder how much of this study is specific to this mode of entropy. How is the other forms of entropy related to the performance?
In a related note, I would like to know better why not much is mentioned regarding the direct regularization of entropies for .
For example, what would happen if the loss as below was used?
$L_{velocity}(\theta) + \lambda E_{t, Y} [KL(p(t, Y)) | Unif(D) ] $
In a similar note regarding other means of controlling the broad participation, what would happen if Muon-like optimizer was used for in channel direction?
Also, it is being sated that acceleration methods reorniaze the residual gates, but is this reorganization, relative to Vanilla, an increase in nPR?
4 comments
We appreciate your time and constructive feedback. We hope our additional analyses and experiments help address your concerns. If you have any further thoughts, we would be grateful to hear whether any concerns remain or whether further clarification would be helpful.
Thank you for the response
Thank you for addressing my concern. It is to be hoped that the additional studies done in our discussion would be reflected in the revised / camera ready version. I keep my positive score as is.
Thank you for your positive and constructive feedback
We appreciate your careful review and continued positive assessment.
We have incorporated the constructive feedback from our discussion into the revised manuscript.
Response to reviewer p5K9
We appreciate your careful review and constructive feedback, as well as your positive assessment of our contributions.
A: We clarify that the Radial objective is intended as an empirical study supporting the interpretation developed in our analysis, rather than as a necessary or unique objective. It tests whether this interpretation can guide a teacher-free design that increases nPR while improving FID. We view these results as supporting the practical value of the design principle, not as ruling out alternative objectives.
A: Thank you for highlighting this connection. As you suggest, nPR is a normalized, exponentiated form of collision entropy: nPR=exp(H2(p))/D, where pd=αd2/∑jαj2.
Recomputing the Pearson correlations with FID in Fig. 6(a), we obtain r=−0.9840 for Rényi-2 entropy (95% CI: [−0.9991,−0.9341]) and r=−0.9740 for Shannon entropy (95% CI: [−0.9974,−0.8559]). Thus, the strong negative association persists with both entropy measures, suggesting that the observation in this sweep is not specific to nPR’s particular formulation. We have included this connection and comparison in the revision.
A: Direct entropy regularization is a natural alternative, since KL(p∥Unif(D))=logD−H1(p). Following your suggestion, optimizing this KL objective improves Vanilla’s FID from 45.41 to 44.94. Directly optimizing nPR, the normalized exponentiated Rényi-2 entropy, improves FID to 44.89. Further tuning may yield additional gains.
However, as discussed in our response to Reviewer voVE’s Q1, higher participation alone is not sufficient: fixing α=1 maximizes participation but prevents gate reorganization during learning and worsens REPA’s FID. This fixed-gate ablation is distinct from soft regularization, but highlights the importance of learned gating beyond maximal participation alone.
A: Thank you for this insightful suggestion. To examine whether applying Muon to the residual branches producing U could increase gate participation, we retained AdamW for the AdaLN modulation parameters. Muon used Nesterov momentum of 0.95, five Newton–Schulz iterations, and no weight decay. Updates were scaled by 0.2max(m,n) [1], allowing us to reuse the constant AdamW learning rate of 10−4 without tuning. All other settings followed Tab. 1 (100K steps, batch size 256). Replacing AdamW with Muon increases nPR from 0.33 to 0.696 and improves FID from 45.41 to 33.23. This provides an additional optimizer-based example consistent with the association between broader gate participation and improved generation quality. However, Muon’s update magnitudes at the measured layer are 1.2-1.7 times those of AdamW. The FID improvement may therefore also reflect differences in update scale and optimization dynamics, rather than isolating the contribution of broader participation.
A: Yes. The gate reorganization we highlight involves increased nPR relative to Vanilla, indicating broader gate participation. The Vanilla model trained under the same conditions as Fig. 6(a) has an nPR of 0.036, while the Vanilla baseline for Fig. 6c is already shown in the figure.
[1] Liu, Jingyuan, et al. "Muon is scalable for llm training." arXiv preprint arXiv:2502.16982 (2025).