- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a joint denoising and sparsity scheduling strategy for diffusion model training. It progressively introduces lower noise levels while activating more neurons. The authors provide theoretical analysis of feature learning and generalization using a two-layer ReLU network, and validate the method on several diffusion architectures and image datasets.
Critical strengths
- Meaningful theoretical contribution. The paper provides an interesting perspective on coarse-to-fine feature learning in diffusion models and attempts to explain the benefits of high-to-low noise scheduling.
- Broad experimental evaluation. Experiments across U-ViT, DiT, SiT, and U-Net demonstrate improvements over standard training in several settings.
Critical weaknesses
- Insufficient ablation studies. Much of the performance improvement comes from noise scheduling alone, while the additional gains from sparsity scheduling are sometimes marginal or inconsistent. The sparsity-only baseline is missing, and the sensitivity to different sparsity ratios and scheduling strategies is not sufficiently explored.
- Limited practical validation. Experiments are restricted to low-resolution datasets (32×32 and 64×64), leaving scalability to larger diffusion models unclear. Moreover, masking embedding channels does not necessarily reduce actual computation, and wall-clock training time is not reported.
Questions for authors
- Could the authors provide a sparsity-only baseline and analyze the sensitivity to different sparsity ratios and uniform schedules?
- Why does joint scheduling provide limited or no improvement over noise-only scheduling in some settings?
- Could the authors report actual training time and GPU memory usage to demonstrate the practical efficiency of the proposed method?
5 comments
2. Sensitivity to sparsity and noise scheduling
Thank you for suggesting sensitivity analyses of the sparsity and noise schedules.
Sensitivity to initial sparsity. We vary the initial fraction of masked embedding channels from 5% to 75% on CIFAR-10 and CelebA-64. All Joint configurations share the original noise curriculum reported below, the same stage boundaries, and the same training settings within each dataset. The table summarizes the sparsity schedules and final FID results after 200k training steps. We have included these results in Appendix A.2, Table 3.
Initial sparsity of the joint scheduling method
| Training steps | 5% | 15% | 30% | 45% | 60% | 75% |
|---|---|---|---|---|---|---|
| 0–40k | 5% | 15% | 30% | 45% | 60% | 75% |
| 40–50k | 4.33% | 13% | 26% | 39% | 52% | 65% |
| 50–60k | 3.33% | 10% | 20% | 30% | 40% | 50% |
| 60–80k | 2.67% | 8% | 16% | 24% | 32% | 40% |
| 80–100k | 1.67% | 5% | 10% | 15% | 20% | 25% |
| 100–200k | 0% | 0% | 0% | 0% | 0% | 0% |
| CIFAR-10 FID ↓ | 10.9884 | 7.8260 | 8.4835 | 7.7745 | 8.0107 | 7.2865 |
| CelebA-64 FID ↓ | 3.6860 | 3.8843 | 4.0471 | 4.3725 | 4.3579 | 4.1325 |
| Baseline | CIFAR-10 FID ↓ | CelebA-64 FID ↓ |
|---|---|---|
| Uniform | 8.2956 | 5.0526 |
| Noise-only | 8.5611 | 3.8915 |
| Both baselines use zero sparsity throughout training. Uniform samples timesteps uniformly from [0,1], while Noise-only follows the original noise curriculum. |
Joint scheduling improves over Uniform across multiple initial sparsity settings on both datasets, indicating that its benefits do not require a precisely tuned sparsity value. The preferred sparsity depends on the dataset and may also depend on the noise schedule, rather than following a universally optimal setting.
Sensitivity to the noise curriculum.
On CIFAR-10, we fix the entire sparsity schedule to the initial-15% configuration above and vary only the lower bound of the timestep sampling interval. The weaker and stronger curricula scale the original lower bounds by 0.5 and 1.5, respectively. All variants use uniform sampling within each interval, identical stage boundaries, and an upper bound of 1.
| Training steps | Weaker | Original | Stronger |
|---|---|---|---|
| 0–10k | [0.15,1] | [0.30,1] | [0.45,1] |
| 10–20k | [0.10,1] | [0.20,1] | [0.30,1] |
| 20–40k | [0.05,1] | [0.10,1] | [0.15,1] |
| 40–50k | [0.035,1] | [0.07,1] | [0.105,1] |
| 50–60k | [0.025,1] | [0.05,1] | [0.075,1] |
| 60–80k | [0.015,1] | [0.03,1] | [0.045,1] |
| 80–100k | [0.005,1] | [0.01,1] | [0.015,1] |
| 100–200k | [0,1] | [0,1] | [0,1] |
| Final FID-50k ↓ | 9.7706 | 7.3371 | 13.0506 |
All three variants are trained for 200k steps and evaluated using EMA weights, a 1000-step SDE sampler, and 50,000 generated images. The original curriculum is independently rerun for this comparison. It achieves the best FID among the three tested schedules. The noise schedules and their FID results are included in Appendix A.2, Table 4.
Summary. These experiments show that both initial sparsity and the noise curriculum affect generation quality. Joint scheduling improves over Uniform across multiple initial sparsity levels, but the preferred setting differs between datasets. Changing the noise curriculum also materially affects FID, even with the sparsity schedule fixed. We therefore treat both schedules as hyperparameters to be selected for the dataset and model, rather than claiming a universally optimal configuration.
3. Why can Joint and Noise-only perform similarly?
Thank you for highlighting the cases in which Joint and Noise-only achieve similar performance.
In settings where Joint and Noise-only perform similarly, most of the improvement over the Uniform baseline comes from noise scheduling, while sparsity scheduling provides an additional, complementary benefit. This additional gain depends on the sparsity hyperparameters: a setting that is not well matched to the dataset and model may yield performance close to Noise-only. As the table above shows, suitable sparsity choices can further improve generation quality, although this additional improvement is not uniform across all settings. The corresponding comparisons with Uniform and Noise-only are included in Appendix A.2, Table 3.
4. Validation at 128×128 and scope regarding latent diffusion
Thank you for suggesting evaluation at higher resolution and clarification of the scope regarding latent diffusion.
We have completed additional experiments on pixel-space CelebA-128, comparing Uniform, Noise-only, and Joint scheduling under matched settings. All three methods were trained for 200k steps and evaluated using EMA weights, a 50-step DPM-Solver sampler, and FID-50k. We have included these results and the discussion of latent diffusion in Appendix A.3, Table 5.
CelebA-128 (128×128), 200k steps
| Method | Final FID-50k ↓ |
|---|---|
| Uniform | 11.6497 |
| Noise-only | 12.9538 |
| Joint 15% | 8.8125 |
Joint scheduling achieves the lowest FID among the three methods, reducing FID by approximately 24.4% relative to Uniform and demonstrating its generation-quality benefit at a higher resolution.
For substantially higher resolutions, diffusion models are often trained in a learned latent space using a pretrained variational autoencoder (VAE). In such settings, generation quality depends not only on the diffusion training strategy but also on the representation and reconstruction quality of the VAE. Since our theoretical analysis focuses on diffusion learning directly in data space, we evaluate our method in pixel space to more directly isolate the effect of denoising and sparsity scheduling, without introducing the additional influence of a pretrained latent representation. Extending our analysis and evaluation to latent diffusion models is an interesting direction for future work.
5. Training time and GPU memory
Thank you for requesting measurements of practical training efficiency.
To measure practical computational benefits, we implemented progressive widening, using smaller embedding, attention, and MLP dimensions when fewer channels are active. Here, current sparsity denotes the fraction of inactive channels at the measured model width, relative to the full width of 256. This differs from initial sparsity in the sensitivity experiments, which specifies the sparsity at the start of training.
The following measurements use CelebA-64, a single H100 GPU, FP16, and a batch size of 128. All configurations use the same implementation. These measurements and their evaluation setup are included in Appendix A.4, Table 6.
| Current sparsity | Active channels | Peak allocated memory (GiB) ↓ | Throughput (images/s) ↑ |
|---|---|---|---|
| 50% | 128/256 | 2.21 | 4,209 |
| 37.5% | 160/256 | 2.95 | 3,423 |
| 25% | 192/256 | 3.23 | 3,400 |
| 0% (full width) | 256/256 | 4.31 | 3,065 |
Reducing the active width lowers memory consumption and increases training throughput. At a current sparsity of 50%, peak allocated memory decreases by 48.7% and throughput increases by approximately 37.3% relative to the full-width configuration. These benefits apply while the model operates at reduced width. After all channels are activated, its memory use and throughput match the full-width configuration.
Response to Reviewer MNR4
We thank the reviewer for recognizing the theoretical contributions and the breadth of our experimental evaluation. Below, we address the motivation for joint scheduling, sensitivity to the sparsity schedule, higher-resolution validation, and practical training efficiency.
1. Motivation for joint scheduling and comparison with sparsity-only scheduling
Thank you for asking us to clarify the motivation for joint scheduling and the role of sparsity-only training.
Our method aims to match active representation capacity to the noise curriculum. Early in training, the model primarily processes high-noise inputs with restricted capacity. As lower-noise inputs are progressively introduced, additional capacity becomes available. Capacity scheduling is therefore a component of the noise-dependent curriculum, and its effect should be understood in conjunction with the corresponding noise distribution.
In preliminary exploration, we also tried sparsity-only scheduling while retaining uniform sampling over the full noise range, but observed poorer generation quality. This setting restricts active capacity without adjusting the training tasks: the capacity-constrained model must handle the entire noise range from the beginning. We believe this mismatch between available capacity and training tasks is one contributing factor. The original manuscript therefore focused on joint scheduling, which follows the proposed design principle. The preliminary sparsity-only observation is noted in the footnote to Section 5.1, and the additional schedule sensitivity results are provided in Appendix A.2.