- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes GrowDiT, a method to improve the training efficiency of Diffusion Transformers. The method starts with a smaller model and gradually activates more attention heads and MLP groups. It uses gradient-based scores to decide which units to activate at different noise levels. Under similar compute budgets, GrowDiT achieves better FID scores than fixed-width training, random growth, and train-then-prune. However, the improvement is relatively small when comparing actual training time rather than FLOPs.
Critical strengths
GrowDiT avoids the computational waste of training a full model before pruning it and achieves better generation quality under a fixed training budget. This idea has potential practical value for training generative models with high computational costs. The paper finds that different noise levels require different amounts of model capacity, with lower noise levels needing more computational resources. The capacity allocation learned by the model is also consistent with the theoretical requirements. This suggests that the gradient-based score can identify where more computation is needed to some extent, rather than simply increasing the model width.
Critical weaknesses
The novelty of this method is relatively limited, as gradient-guided network growth and timestep-dependent compute allocation have both been explored in previous work. In the experiments, Random Growth already achieves good results, while the additional improvement from the Need Score is relatively small. Also, the improvement in actual training time is much smaller than the FLOPs-based improvement.
Questions for authors
The Need Score is computed using first-order gradients at gate = 0, but many inactive Attention Heads and MLP Groups may not have been sufficiently trained. Why can their gradient magnitudes reflect how useful they will be after training? Also, gradient magnitudes may be affected by the scale of unit outputs. How do the authors account for this?
GrowDiT combines progressive growth with noise-dependent capacity allocation, but the paper does not clearly separate their contributions. What if we directly use the final allocation of Attention Heads and MLP Groups learned by GrowDiT and train a fixed architecture from scratch? Could it achieve similar results? If so, what is the benefit of progressive growth itself?
3 comments
On the Novelty (Weakness). While GrowDiT shares the goal of noise-dependent capacity allocation with methods such as DyDiT, it addresses a fundamentally different learning problem. DyDiT determines which units to retain after training a full model, whereas GrowDiT learns how to allocate capacity across noise levels during training, without access to a fully trained network. Crucially, because weights are shared across noise levels, GrowDiT can evaluate whether units already trained at one noise level should be activated at another. This enables adaptive capacity allocation without the cost of first training a dense model. Gradient-guided growth such as GradMax (Evci et al., 2022) and Firefly (Wu et al., 2020) uses gradients to decide which units to add to a network that serves every input. GrowDiT instead uses its gradient-based need score to decide which noise levels each unit should serve, which gives an adaptive, noise-dependent allocation of capacity. This allocation yields a measurable advantage: in seed-matched experiments, it consistently outperforms fixed or unguided growth under constrained training budgets. The advantage is also interpretable: the need score automatically finds an allocation with more capacity at low noise levels, the same overall pattern as one learned from a fully trained model. The same mechanism also improves depth growth. We believe this gives a new way to train diffusion models and a reference for later work.
On the Gain over Random Growth (Weakness). The improvement over random growth is consistent across all nine seed-matched comparisons (Appendix D, Table 9) and across all three model scales. While the absolute FID reduction varies with model size, the relative improvement increases from 2.3% on DiT-S to 3.9% on DiT-L. More importantly, the additional ablations isolate the source of this improvement: the benefit arises primarily from the noise-dependent capacity allocation learned by the need score, rather than from the selection of individual units (Sec. 5.3).
| GrowDiT over random growth | FID gain | relative gain |
|---|---|---|
| DiT-S | 1.52 | 2.3% |
| DiT-B | 1.27 | 3.0% |
| DiT-L | 0.93 | 3.9% |
Gain of GrowDiT over random growth on ImageNet-256 (1× budget), averaged over seeds. Relative gain: gain divided by the FID of random growth.
On the Wall-clock Time (Weakness). We recomputed the wall-clock comparison using FID averaged over seeds, consistently with Table 2, and with all growth-related operations included in the training time (Table 6; Appendix G). As expected, the practical gains are smaller than those measured under equal FLOP budgets. Nevertheless, GrowDiT achieves lower FID at equal wall-clock time on DiT-B and DiT-L, with improvements of 3.8% and 7.1%, respectively, while matching uniform training on DiT-S. Importantly, the gap between FLOP-based and wall-clock improvements becomes smaller for larger models, suggesting that the benefits of adaptive growth become increasingly relevant at scale. These results demonstrate practical training-time advantages beyond theoretical compute savings.
| FID gain over uniform | DiT-S | DiT-B | DiT-L |
|---|---|---|---|
| GrowDiT, same FLOPs | 2.1% | 6.0% | 9.6% |
| GrowDiT, same wall-clock time | 0.0% | 3.8% | 7.1% |
| wall-clock gain / FLOP gain | 0% | 63% | 74% |
FID improvement over uniform on ImageNet-256 (Table 6), averaged over seeds as in Table 2. Same wall-clock time: every method trains for the time that uniform needs for the 1× FLOP budget.
On Presentation. We acknowledge the Reviewer's concerns about clarity. We have revised the presentation to better distinguish progressive growth from noise-dependent capacity allocation, clarify the role of the need score, and make the relationship with prior approaches more explicit. We have also reorganized the ablation discussion to isolate the contribution of each component.
Thank you for the detailed response. The additional experiments address my main concerns. I will raise my rating to 3.
We thank the Reviewer for their careful evaluation and for their questions, which have helped us state more precisely what the need score contributes.
Question 1 (usefulness after training). Weights are shared across noise levels and only the gates are per noise level, so most inactive candidates are units already trained at other noise levels. Most growth decisions therefore ask whether a trained unit should also serve another noise level. Although the need score is a local, first-order estimate (Proposition 1), our experiments show that it provides a useful signal for learning noise-dependent capacity allocations over the course of training. Two findings support this conclusion. First, without access to a fully trained model, GrowDiT recovers the same overall allocation trend as the routers learned by DyDiT on a pretrained dense model (Fig. 4a). Second, the learned allocation consistently outperforms a hand-designed allocation, with the advantage becoming more pronounced for DiT-B (table below). The choice of units, in contrast, matters little. At low noise levels, where fine details are refined, most units end up active, so the choice of units mainly decides when a unit is activated, not whether it is (Appendix E, Table 11). The activation time of a unit, and hence the locality of the need score, has little effect on the final FID (Sec. 5.3, Table 5).
Gain over random growth on ImageNet-256 (1× budget), averaged over seeds. All three methods choose the units at random. Relative gain: gain divided by the FID of random growth.
Question 1 (scale of unit outputs). The need score measures the local sensitivity of the loss to gate activation under the current parameterization (Eq. 3). This sensitivity naturally depends on both the magnitude of the unit's output and its alignment with the loss gradient, since both determine the first-order loss variation when the gate is activated. We acknowledge that the score is parameterization-dependent. However, our objective is not to estimate an intrinsic, scale-invariant importance of each unit, but to identify beneficial gate activations within the current model. The empirical results support its effectiveness for this purpose.
Question 2. We trained the baseline suggested by the Reviewer (final allocation without growth; Sec. 5.3, Table 5). It takes the final allocation of heads and MLP groups from a GrowDiT run with the same seed and keeps it fixed throughout training. This architecture is trained from random initialization with the same budget. With the full budget its FID is still higher than that of GrowDiT (65.0 against 64.3 on DiT-S; table below), and progressive growth has two further benefits. (1) It finds the allocation within one run, whereas the fixed architecture needs a GrowDiT run first. (2) With a limited budget it also gives a lower FID: growth is ahead until about 75% of the budget (Appendix E, Fig. 6), because the early narrow model takes more steps for the same FLOPs.
FID-50k on DiT-S (ImageNet-256, 1× budget), averaged over seeds. Gain: FID reduction relative to random growth.