acceptodds
Under review as a conference paper at ICLR 2027

Enhanced Generation in Diffusion Models via Joint High-to-Low Denoising Schedule and Model Sparsity with Theoretical Guarantees

Abstract

Diffusion models require learning denoising tasks across a wide range of noise levels. Standard training selects noise levels uniformly across the forward process, leading to entangled feature representations and degraded generation. Several recent methods adopt a high-to-low noise schedule during training, showing empirically faster convergence and improved generalization, yet no theoretical guarantees have been established. This paper develops a novel joint high-to-low denoising schedule and model sparsity framework and provides the first theoretical analysis linking noise level and model sparsity to the training dynamics and generalization in diffusion models. Our theoretical analysis shows that scheduling noise from high to low strengthens the implicit coarse-to-fine learning dynamics in diffusion training: the high-noise stage learns coarse features first, while the subsequent low-noise stage refines fine-grained details. Beyond the denoising schedule, our method introduces a high-to-low model sparsity strategy that progressively activates neurons: it only updates a subset of neurons during the high-noise stage and gradually activates the remaining neurons, which preserves capacity for fine-grained features without interference from early-stage high noise. Our method and analysis are supported by both theoretical guarantees and empirical results on multiple practical datasets and models, demonstrating clear advantages over standard diffusion training with uniform noise sampling.

Primary area:
learning theory
Keywords:
Training Dynamics, Generalization Guarantees, Learning Theory, Diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MNR4OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a joint denoising and sparsity scheduling strategy for diffusion model training. It progressively introduces lower noise levels while activating more neurons. The authors provide theoretical analysis of feature learning and generalization using a two-layer ReLU network, and validate the method on several diffusion architectures and image datasets.

Critical strengths
  1. Meaningful theoretical contribution. The paper provides an interesting perspective on coarse-to-fine feature learning in diffusion models and attempts to explain the benefits of high-to-low noise scheduling.
  2. Broad experimental evaluation. Experiments across U-ViT, DiT, SiT, and U-Net demonstrate improvements over standard training in several settings.
Critical weaknesses
  1. Insufficient ablation studies. Much of the performance improvement comes from noise scheduling alone, while the additional gains from sparsity scheduling are sometimes marginal or inconsistent. The sparsity-only baseline is missing, and the sensitivity to different sparsity ratios and scheduling strategies is not sufficiently explored.
  2. Limited practical validation. Experiments are restricted to low-resolution datasets (32×32 and 64×64), leaving scalability to larger diffusion models unclear. Moreover, masking embedding channels does not necessarily reduce actual computation, and wall-clock training time is not reported.
Questions for authors
  1. Could the authors provide a sparsity-only baseline and analyze the sensitivity to different sparsity ratios and uniform schedules?
  2. Why does joint scheduling provide limited or no improvement over noise-only scheduling in some settings?
  3. Could the authors report actual training time and GPU memory usage to demonstrate the practical efficiency of the proposed method?
5 comments

Response to Reviewer MNR4

We thank the reviewer for recognizing the theoretical contributions and the breadth of our experimental evaluation. Below, we address the motivation for joint scheduling, sensitivity to the sparsity schedule, higher-resolution validation, and practical training efficiency.

1. Motivation for joint scheduling and comparison with sparsity-only scheduling

Thank you for asking us to clarify the motivation for joint scheduling and the role of sparsity-only training.

Our method aims to match active representation capacity to the noise curriculum. Early in training, the model primarily processes high-noise inputs with restricted capacity. As lower-noise inputs are progressively introduced, additional capacity becomes available. Capacity scheduling is therefore a component of the noise-dependent curriculum, and its effect should be understood in conjunction with the corresponding noise distribution.

In preliminary exploration, we also tried sparsity-only scheduling while retaining uniform sampling over the full noise range, but observed poorer generation quality. This setting restricts active capacity without adjusting the training tasks: the capacity-constrained model must handle the entire noise range from the beginning. We believe this mismatch between available capacity and training tasks is one contributing factor. The original manuscript therefore focused on joint scheduling, which follows the proposed design principle. The preliminary sparsity-only observation is noted in the footnote to Section 5.1, and the additional schedule sensitivity results are provided in Appendix A.2.

2. Sensitivity to sparsity and noise scheduling

Thank you for suggesting sensitivity analyses of the sparsity and noise schedules.

Sensitivity to initial sparsity. We vary the initial fraction of masked embedding channels from 5% to 75% on CIFAR-10 and CelebA-64. All Joint configurations share the original noise curriculum reported below, the same stage boundaries, and the same training settings within each dataset. The table summarizes the sparsity schedules and final FID results after 200k training steps. We have included these results in Appendix A.2, Table 3.

Initial sparsity of the joint scheduling method

Training steps5%15%30%45%60%75%
0–40k5%15%30%45%60%75%
40–50k4.33%13%26%39%52%65%
50–60k3.33%10%20%30%40%50%
60–80k2.67%8%16%24%32%40%
80–100k1.67%5%10%15%20%25%
100–200k0%0%0%0%0%0%
CIFAR-10 FID ↓10.98847.82608.48357.77458.01077.2865
CelebA-64 FID ↓3.68603.88434.04714.37254.35794.1325
BaselineCIFAR-10 FID ↓CelebA-64 FID ↓
Uniform8.29565.0526
Noise-only8.56113.8915
Both baselines use zero sparsity throughout training. Uniform samples timesteps uniformly from [0,1], while Noise-only follows the original noise curriculum.

Joint scheduling improves over Uniform across multiple initial sparsity settings on both datasets, indicating that its benefits do not require a precisely tuned sparsity value. The preferred sparsity depends on the dataset and may also depend on the noise schedule, rather than following a universally optimal setting.

Sensitivity to the noise curriculum.

On CIFAR-10, we fix the entire sparsity schedule to the initial-15% configuration above and vary only the lower bound of the timestep sampling interval. The weaker and stronger curricula scale the original lower bounds by 0.5 and 1.5, respectively. All variants use uniform sampling within each interval, identical stage boundaries, and an upper bound of 1.

Training stepsWeakerOriginalStronger
0–10k[0.15,1][0.30,1][0.45,1]
10–20k[0.10,1][0.20,1][0.30,1]
20–40k[0.05,1][0.10,1][0.15,1]
40–50k[0.035,1][0.07,1][0.105,1]
50–60k[0.025,1][0.05,1][0.075,1]
60–80k[0.015,1][0.03,1][0.045,1]
80–100k[0.005,1][0.01,1][0.015,1]
100–200k[0,1][0,1][0,1]
Final FID-50k ↓9.77067.337113.0506

All three variants are trained for 200k steps and evaluated using EMA weights, a 1000-step SDE sampler, and 50,000 generated images. The original curriculum is independently rerun for this comparison. It achieves the best FID among the three tested schedules. The noise schedules and their FID results are included in Appendix A.2, Table 4.

Summary. These experiments show that both initial sparsity and the noise curriculum affect generation quality. Joint scheduling improves over Uniform across multiple initial sparsity levels, but the preferred setting differs between datasets. Changing the noise curriculum also materially affects FID, even with the sparsity schedule fixed. We therefore treat both schedules as hyperparameters to be selected for the dataset and model, rather than claiming a universally optimal configuration.

3. Why can Joint and Noise-only perform similarly?

Thank you for highlighting the cases in which Joint and Noise-only achieve similar performance.

In settings where Joint and Noise-only perform similarly, most of the improvement over the Uniform baseline comes from noise scheduling, while sparsity scheduling provides an additional, complementary benefit. This additional gain depends on the sparsity hyperparameters: a setting that is not well matched to the dataset and model may yield performance close to Noise-only. As the table above shows, suitable sparsity choices can further improve generation quality, although this additional improvement is not uniform across all settings. The corresponding comparisons with Uniform and Noise-only are included in Appendix A.2, Table 3.

4. Validation at 128×128 and scope regarding latent diffusion

Thank you for suggesting evaluation at higher resolution and clarification of the scope regarding latent diffusion.

We have completed additional experiments on pixel-space CelebA-128, comparing Uniform, Noise-only, and Joint scheduling under matched settings. All three methods were trained for 200k steps and evaluated using EMA weights, a 50-step DPM-Solver sampler, and FID-50k. We have included these results and the discussion of latent diffusion in Appendix A.3, Table 5.

CelebA-128 (128×128), 200k steps

MethodFinal FID-50k ↓
Uniform11.6497
Noise-only12.9538
Joint 15%8.8125

Joint scheduling achieves the lowest FID among the three methods, reducing FID by approximately 24.4% relative to Uniform and demonstrating its generation-quality benefit at a higher resolution.

For substantially higher resolutions, diffusion models are often trained in a learned latent space using a pretrained variational autoencoder (VAE). In such settings, generation quality depends not only on the diffusion training strategy but also on the representation and reconstruction quality of the VAE. Since our theoretical analysis focuses on diffusion learning directly in data space, we evaluate our method in pixel space to more directly isolate the effect of denoising and sparsity scheduling, without introducing the additional influence of a pretrained latent representation. Extending our analysis and evaluation to latent diffusion models is an interesting direction for future work.

5. Training time and GPU memory

Thank you for requesting measurements of practical training efficiency.

To measure practical computational benefits, we implemented progressive widening, using smaller embedding, attention, and MLP dimensions when fewer channels are active. Here, current sparsity denotes the fraction of inactive channels at the measured model width, relative to the full width of 256. This differs from initial sparsity in the sensitivity experiments, which specifies the sparsity at the start of training.

The following measurements use CelebA-64, a single H100 GPU, FP16, and a batch size of 128. All configurations use the same implementation. These measurements and their evaluation setup are included in Appendix A.4, Table 6.

Current sparsityActive channelsPeak allocated memory (GiB) ↓Throughput (images/s) ↑
50%128/2562.214,209
37.5%160/2562.953,423
25%192/2563.233,400
0% (full width)256/2564.313,065

Reducing the active width lowers memory consumption and increases training throughput. At a current sparsity of 50%, peak allocated memory decreases by 48.7% and throughput increases by approximately 37.3% relative to the full-width configuration. These benefits apply while the model operates at reduced width. After all channels are activated, its memory use and throughput match the full-width configuration.

Reviewer_pk7jOpenReview ↗
Rating
2(+1)
Confidence
1(−2)
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework that jointly schedules noise levels and model sparsity from high to low, and provides a theoretical analysis linking these factors to the training dynamics of diffusion models. During the high-noise training stage, the method updates only a subset of neurons. As lower-noise tasks are introduced, it progressively activates the remaining neurons, aiming to reserve capacity for learning fine-grained features and reduce interference from early-stage high-noise training. Experiments on multiple datasets show improvements in generation quality and training efficiency over standard diffusion training with uniform noise sampling.

Critical strengths
  1. The proposed delayed unfreezing of a subset of neurons is well motivated, as it aims to preserve capacity for learning fine-grained features at later stages while reducing interference from early high-noise training.

  2. The experiments include a high-to-low denoising schedule only baseline, helping assess the additional benefit of the proposed model sparsity schedule.

Critical weaknesses
  1. The paper contains multiple theoretical errors. For example, the Gaussian projection bound in Eq. (117) is incompatible with the specified noise distribution: the probability of this event tends to zero as (d) increases, contradicting the claimed high-probability guarantee. The argument in Appendix G.5 for reaching the activation threshold is inconsistent with the bias and feature growth rates given in Eq. (196) and (208).

  2. The feature-alignment values in Figure 3 are problematic. Section 5.1 sets (m=50) and (d=20), averages the top 20 normalized neuron projections for each orthogonal feature, and then averages across all features. The upper bound of this statistic precludes the near-one values shown by the (M_1) curves and Joint’s (M_2) curve in Figure 3.

Questions for authors
  1. Fixing the ReLU activation pattern does not guarantee that the loss is jointly strongly convex in the weights of both layers. Could the authors prove the local PL condition in Eq. (252) with (μ=Θ(1)\mu=\Theta(1)), specify the neighborhood where it holds, and justify why subsequent training iterates remain within that neighborhood?

  2. How do the progressively expanding noise range and embedding-output masking used in the experiments correspond to the stage-specific noise sampling and neuron-weight freezing analyzed in the theory?

  3. Section 5.2 specifies (t∼Unif(0.5,1)t\sim\mathrm{Unif}(0.5,1)) for the initial stage, whereas Table 2 in Appendix A.3 uses (t∼Unif(0.3,1)t\sim\mathrm{Unif}(0.3,1)). Which timestep sampling schedule was actually used to obtain the reported experimental results?

4 comments

Response to Reviewer pk7j

We thank the reviewer for the careful reading and constructive comments. We address each concern below.

W1.1: Gaussian projection bound in Eq. (117)

Thank you for raising this concern about the Gaussian projection bound.

Clarification of Eq. (117). We believe this comes from a misunderstanding. We intent to say that the event in Eq. (117) happens with high probability 1−e−Ω(d1)1-e^{-\Omega(d_1)}.

For a fixed nonzero wi(t)w_i^{(t)} independent of ξ∼N(0,σξ2Id1)\xi\sim\mathcal N(0,\sigma_\xi^2I_{d_1}), the Gaussian tail bound gives

Pr⁡ξ!(∣⟨wi(t),τξ⟩∣>Cτ∣wi(t)∣2d(1+c0)/2)≤2exp⁡!(−C22σξ2d1+c0)≤2e−Ω(d1).\Pr_\xi\\!\left( |\langle w_i^{(t)},\tau\xi\rangle| > \frac{C\tau\\|w_i^{(t)}\\|_2}{d^{(1+c_0)/2}} \right)\le 2\exp\\!\left(-\frac{C^2}{2\sigma_\xi^2d^{1+c_0}}\right) \le 2e^{-\Omega(d_1)}.

Therefore, the corresponding projection bound holds with high probability 1−e−Ω(d1)1-e^{-\Omega(d_1)}.

We have added a clarification in Appendix E.1.1, immediately after Eq. (117), distinguishing the stated probability level from the scale of the projection bound.

W1.2: Activation threshold in Appendix G.5

Thank you for pointing out the need to clarify the bias schedule and activation threshold.

Bias growth and the activation threshold. We agree that the multiplicative bias growth rate in Eq. (196) exceeds the feature-alignment growth rate in Eq. (208). However, the bias does not grow indefinitely. It grows multiplicatively during Stage I and reaches the cap bmax⁡=Θ(1)b_{\max}=\Theta(1) during Stage II, t∈[T1,T1+T2]t\in[T_1,T_1+T_2], after which it remains fixed. This cap is consistent with the stated scales of ∣wi∣2\\|w_i\\|_2 and ∣vi∣2\\|v_i\\|_2. An excessively large bias would suppress the activations needed for feature learning.

After the bias reaches its cap, feature alignment continues to grow toward the fixed threshold. This explains “the condition holds when Stage II is sufficiently long” at line 3318 of the original submission and the duration in Eq. (209). We have stated the cap in Appendix G.1, Eq. (196), and explained its role near Eq. (209) in Appendix G.5.

W2: Feature-alignment statistic in Figure 3

Clarification of the feature-alignment metric. We thank the reviewer for identifying an inaccurate description of the metric in Section 5.1. The averaging is indexed by features. For each feature MkjM_{kj}, with j=1,…,dj=1,\ldots,d, we find its largest normalized alignment among the mm neurons. We then average these dd feature-specific maxima:

AMk=1d∑j=1dmax⁡1≤i≤m∣⟨wi,Mkj⟩∣∣wi∣2.A_{M_k}=\frac{1}{d}\sum_{j=1}^{d}\max_{1\le i\le m}\frac{|\langle w_i,M_{kj}\rangle|}{\\|w_i\\|_2}.

With d=20d=20 and m=50m=50, we average 20 feature-specific maxima, each taken over 50 neurons. Different features may select different neurons, and the metric can approach 1 when every feature has a highly aligned neuron. We have corrected “average over the top 20 neurons” in Section 5.1 and clarified the Figure 3 caption. The plotted curves remain unchanged.

Q1: Local PL condition in Eq. (252)

Thank you for highlighting the need to justify the local PL condition.

We agree that fixing the ReLU activation pattern alone does not guarantee joint strong convexity. The convergence argument in Appendix I concerns only the final phase, from t=T1+T2t=T_1+T_2 to t=T1+T2+TDMt=T_1+T_2+T_{\mathrm{DM}}, after the feature-learning stages. At its starting point, Appendix G.8 gives the following alignment results:

\left| \left\langle w_i^{(T_1+T_2)},M_{1j}\right\rangle \right|^2 \ge (1-o(1))\left\\|w_i^{(T_1+T_2)}\right\\|_2^2, \qquad i\in S_{1j,\mathrm{sure}}. \tag{218}

\left| \left\langle \widehat w_i^{(T_1+T_2)},M_{2j}\right\rangle \right|^2 = (1-o(1))\left\\|\widehat w_i^{(T_1+T_2)}\right\\|_2^2, \qquad i\in S_{2j,\mathrm{sure}}. \tag{219}

These bounds show that the two groups of neurons have already learned well-aligned representations of the M1M_1 and M2M_2 features, respectively.

Write Sj,sureS_{j,\mathrm{sure}} and zjz_j for the sure set and target feature in the relevant stage. For i∈Sj,surei\in S_{j,\mathrm{sure}}, consider the fixed pattern in which the positive ReLU branch is active when zj=1z_j=1 and inactive otherwise.

On the active branch, the output is vi(wi⊤xτ−bi)v_i(w_i^\top x_\tau-b_i), which contains a bilinear term in wiw_i and viv_i. With bib_i fixed and expectations taken over the fixed distribution of samples with zj=1z_j=1, the branch loss, using x0x_0 as the reconstruction target, is

\mathcal L_{\mathrm{DM},j}(w_i,v_i) = \frac12\mathbb E \left\\|v_i(w_i^\top x_\tau-b_i)-x_0\right\\|_2^2.

The exact loss difference satisfies

\begin{aligned} &\mathcal L_{\mathrm{DM},j}(w_i,v_i) -\mathcal L_{\mathrm{DM},j}(w_i^*,v_i^*)\\\\ ={}& \langle\nabla_{w_i}\mathcal L_{\mathrm{DM},j},w_i-w_i^*\rangle + \langle\nabla_{v_i}\mathcal L_{\mathrm{DM},j},v_i-v_i^*\rangle\\\\ &- \mathbb E\\!\left[ \bigl(v_i(w_i^\top x_\tau-b_i)-x_0\bigr)^\top (v_i-v_i^*)((w_i-w_i^*)^\top x_\tau) \right]\\\\ &-\frac12\mathbb E \left\\| v_i(w_i^\top x_\tau-b_i) -v_i^*(w_i^{*\top}x_\tau-b_i) \right\\|_2^2. \end{aligned}

Using the independent isotropic noise model, the residual interaction term and the squared prediction difference satisfy

\tag{R1} \begin{aligned} & \mathbb E\\!\left[ \bigl(v_i(w_i^\top x_\tau-b_i)-x_0\bigr)^\top (v_i-v_i^*)((w_i-w_i^*)^\top x_\tau) \right]\\\\ &+\frac12\mathbb E \left\\| v_i(w_i^\top x_\tau-b_i) -v_i^*(w_i^{*\top}x_\tau-b_i) \right\\|_2^2\\\\ &\qquad\ge \frac{\lambda}{2} \left( \\|w_i-w_i^*\\|_2^2+\\|v_i-v_i^*\\|_2^2 \right), \end{aligned}

where

\lambda= \min\left\\{ \left(1-O\\!\left(\frac1d\right)\right) \\|v_i^*\\|_2^2\mathbb E[\tau^2], \quad \mathbb E[(w_i^{*\top}x_\tau-b_i)^2] \right\\}>0.

Substituting Eq. (R1) and applying Cauchy--Schwarz inequality gives

LDM,j(wi,vi)−LDM,j(wi∗,vi∗)≤∣∇LDM,j∣2∣wi−wi∗∣22+∣vi−vi∗∣22−λ2(∣wi−wi∗∣22+∣vi−vi∗∣22)≤∣∇LDM,j∣222λ.\begin{aligned} &\mathcal L_{\mathrm{DM},j}(w_i,v_i) -\mathcal L_{\mathrm{DM},j}(w_i^*,v_i^*)\\\\ \le{}& \\|\nabla\mathcal L_{\mathrm{DM},j}\\|_2 \sqrt{ \\|w_i-w_i^*\\|_2^2+\\|v_i-v_i^*\\|_2^2 }\\\\ &-\frac{\lambda}{2} \left( \\|w_i-w_i^*\\|_2^2+\\|v_i-v_i^*\\|_2^2 \right)\\\\ \le{}& \frac{\\|\nabla\mathcal L_{\mathrm{DM},j}\\|_2^2} {2\lambda}. \end{aligned}

Therefore,

∣∇LDM,j(wi,vi)∣22≥2μ[LDM,j(wi,vi)−LDM,j(wi∗,vi∗)],μ=λ>0.\boxed{ \begin{aligned} &\\|\nabla\mathcal L_{\mathrm{DM},j}(w_i,v_i)\\|_2^2\\\\ &\quad\ge 2\mu\left[ \mathcal L_{\mathrm{DM},j}(w_i,v_i) -\mathcal L_{\mathrm{DM},j}(w_i^*,v_i^*) \right], \qquad \mu=\lambda>0. \end{aligned} }

We have added the gradient calculation and the loss-difference expansion in Appendix I.3, Eqs. (254) and (263)--(264) of the revised manuscript.

Q2: Correspondence between theoretical and practical schedules

Thank you for asking us to clarify the correspondence between the theoretical and experimental schedules.

Noise scheduling. We adopt stage-specific noise sampling as a convenient formulation for controlling the decrease in the expected noise level and simplifying the theoretical analysis. The separate noise intervals provide a simplified description of the high-to-low curriculum, rather than prescribing the only way to implement it. In practice, we keep the upper sampling bound fixed and progressively lower the lower bound, introducing more low-noise samples while retaining high-noise samples. Under uniform sampling, this shifts the distribution toward lower noise levels and decreases the expected noise level. Thus, the expanding range implements the same high-to-low progression. We have clarified this correspondence in the revised manuscript.

Capacity scheduling. Embedding-output masking is the experimental counterpart of neuron freezing in our theoretical model: both delay the participation of selected feature units in learning. Specifically, we mask selected output channels of the convolutional patch embedding. These channels produce zero output and receive zero task-loss gradient through the masked outputs. Removing the masks allows these channels to participate in learning, corresponding to unfreezing additional neurons as lower-noise tasks are introduced. We implement this mechanism at the embedding-channel level to accommodate the Transformer architecture.

We have clarified these implementation choices in Appendix A.6, under Training Schedule and in Table 8. The distinction between output masking and progressive widening is discussed separately in Appendix A.4.

Q3: Actual initial timestep sampling schedule

Thank you for pointing out the inconsistency in the reported initial sampling range.

The reported experiments used t∼Unif⁡(0.3,1)t\sim\operatorname{Unif}(0.3,1) in the initial stage, as correctly specified in Table 8 of Appendix A.6. The reference to Unif⁡(0.5,1)\operatorname{Unif}(0.5,1) in Section 5.2 was a textual error. We have corrected the initial sampling range and the corresponding descriptions of tmin⁡t_{\min} in Section 5.2 to match the appendix.

We have also added schedule sensitivity experiments in Appendix A.2, Tables 3--4, showing improvements over Uniform across multiple initial sparsity settings. Appendix A.3, Table 5, reports 128×128128\times128 CelebA experiments, where Joint reduces FID by 24.4% relative to Uniform. Appendix A.4, Table 6, reports reduced memory usage and increased throughput at reduced model widths.

The authors have addressed my concerns, so I have raised my score to 2. However, I have lowered my confidence score to 1, as I have not fully verified the entire manuscript due to its length and cannot rule out the possibility of other undiscovered errors.

Reviewer_ES6wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a curriculum training framework for diffusion models that combines a high-to-low noise schedule with progressive model capacity expansion (high-to-low model sparsity). Using a two-scale orthogonal sparse coding data model and a one-hidden-layer ReLU network, the authors analyze the SGD training dynamics and generalization behavior. They show that training a subset of neurons on high noise first captures coarse features, while progressively unfreezing the remaining neurons under low noise allows pure learning of fine-grained features without early-stage interference. The method is validated on synthetic data and across four diffusion backbones on four image datasets.

Critical strengths
  1. Unlike prior diffusion theory that largely focuses on linear denoisers, this paper adapts the non-linear feature purification framework to diffusion training. It provides an intuitive theoretical explanation of how noise scheduling and progressive neuron activation interact to prevent feature entanglement between coarse and fine scales.

  2. The experiments cover four mainstream diffusion architectures—CNN (U-Net) and Transformer-based (U-ViT, DiT, SiT)—as well as multiple datasets, and integrate global SVD spectral projection and Silhouette clustering scores to provide an intuitive diagnostic analysis of feature evolution.

Critical weaknesses
  1. Gap Between Theory and Practice: The theory assumes Stage II strictly samples low noise and completely freezes hidden neurons, whereas practical experiments use full-range uniform sampling (=1.0) for most training and only mask input patch-embedding channels rather than freezing deep Transformer neurons.

  2. Missing Quantitative Tables and Minor Inconsistencies: The paper lacks a standard summary table for evaluation metrics (FID, sFID, etc.). Additionally, the CIFAR-10 schedule details in Section 5.2 conflict with Appendix Table 2, Figure 3 lacks its mentioned Silhouette Score panel, and Theorem 5 omits the baseline generalization bound.

Questions for authors
  1. Theory-Practice Alignment: Why does retaining high-noise samples (=1.0) and only masking input channels in experiments avoid the feature entanglement predicted by Theorem 4?

  2. Mathematical Clarification: In Appendices F and G, the bias grows much faster than the alignment rate during Stage II. Does this prevent features from crossing the activation threshold?

  3. Could you provide a comprehensive quantitative summary table comparing all models and datasets across evaluation metrics?

2 comments

Response to Reviewer ES6w

We thank the reviewer for recognizing the nonlinear feature-learning perspective and the breadth of our experimental evaluation. We address each concern below.

1. Correspondence between theory and practical scheduling

Thank you for raising this question about the connection between theory and implementation.

Noise scheduling. We use stage-specific noise sampling to simplify the theoretical analysis and conveniently control the decrease in the expected noise level. The key scheduling principle is the progressive reduction in the expected noise level. In practice, we implement this by keeping the upper sampling bound fixed and progressively lowering the lower bound, thereby introducing more low-noise samples while retaining high-noise samples. Therefore, retaining an upper bound of 1.0 is consistent with the intended scheduling principle.

Capacity scheduling. Embedding-output masking implements the delayed participation of feature channels that neuron-weight freezing captures in our theoretical model. Specifically, we mask selected output channels of the convolutional patch embedding. These channels contribute zero output and receive zero task-loss gradient through the masked outputs, temporarily preventing them from participating in feature learning. Progressively removing the mask reactivates these channels, corresponding to unfreezing additional neurons as lower-noise tasks are introduced. Thus, both mechanisms initially restrict the active representation capacity and subsequently release additional capacity for learning finer features. The implementation operates on embedding channels to accommodate the Transformer architecture.

We have clarified these implementation choices in Appendix A.6, under Training Schedule and in Table 8. The distinction between output masking and progressive widening is discussed separately in Appendix A.4.

2. Bias growth and the activation threshold

Thank you for drawing attention to the bias growth and activation-threshold argument.

We agree that the multiplicative growth rate of the bias in Eq. (196) is larger than the feature-alignment growth rate in Eq. (208). However, the intended bias schedule does not grow indefinitely. It is capped at bmax⁡=Θ(1)b_{\max}=\Theta(1), consistent with ∥wi∥2=Θ(1)\|w_i\|_2=\Theta(1) and ∥vi∥2=Θ(1)\|v_i\|_2=\Theta(1). An excessively large bias would suppress ReLU activations and hinder target-feature learning, motivating the use of a capped bias schedule.

Once the bias reaches this cap, feature alignment continues to grow and can cross the fixed threshold despite its slower initial growth rate. This is the intended meaning of “the condition holds when Stage II is sufficiently long” at line 3318 of the original submission, and the purpose of the training duration specified in Eq. (209). We have explicitly stated the bias upper bound and explained its role in reaching the activation threshold in Appendix G.1, Eq. (196), and Appendix G.5, in the discussion surrounding Eq. (209).

3. Comprehensive quantitative summary

Thank you for suggesting a consolidated quantitative comparison.

We agree that a consolidated numerical table would make the experimental comparisons easier to assess. We have organized the reported results by backbone, dataset, and training strategy, comparing standard training, denoising scheduling alone, and joint scheduling. These results are included in Appendix A.1, with endpoint FID values in Table 1 and metrics at the minimum-FID checkpoints in Table 2.

The tables include FID, sFID, Precision, Recall, and IS wherever evaluated, together with the training budget and checkpoint-selection rule. We have distinguished final-checkpoint results from results at the checkpoint with the lowest FID, and explicitly marked metrics that were not evaluated.

4. CIFAR-10 timestep sampling schedule

Thank you for identifying the inconsistent sampling ranges.

The reported CIFAR-10 experiments used t∼Unif⁡(0.3,1)t\sim\operatorname{Unif}(0.3,1) in the initial stage, as correctly specified in Table 8 of Appendix A.6. The reference to Unif⁡(0.5,1)\operatorname{Unif}(0.5,1) in Section 5.2 was a textual error. We have corrected the initial sampling range and the corresponding descriptions of tmin⁡t_{\min} in Section 5.2 to match the appendix.

5. Silhouette Score panel in Figure 3

Thank you for pointing out this inconsistency. We have updated the Figure 3 caption in Section 5.1 to match the displayed content, describing only the displayed feature-alignment panels.

6. Clarification of Theorem 5

Thank you for pointing out the need to make the baseline comparison in Theorem 5 explicit.

We acknowledge that the notation in Eq. (11) was insufficiently explicit. In the high-noise regime, τ∈[τ1,τmax⁡]\tau\in[\tau_1,\tau_{\max}], the intended statement is that both methods have denoising risk of order Θ(d−2c0)\Theta(d^{-2c_0}):

\qquad \mathcal L(g_{\mathrm{std}};\tau)=\Theta(d^{-2c_0}).$$ We have revised Eq. (11) to explicitly identify both $g_{\mathrm{joint}}$ and $g_{\mathrm{std}}$, replacing the ambiguous notation $g$ in Section 4.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Request for Follow-up Reviewer Discussion on Submission #35744

Dear Area Chair,

Thank you for overseeing our submission. We would like to briefly summarize the progress made during the rebuttal and respectfully request follow-up discussion with the reviewers, particularly regarding additional evidence that may not have been fully considered before the rebuttal deadline.

Our paper proposes a joint high-to-low denoising and model sparsity scheduling framework for diffusion training. Our main contribution is a theoretical characterization of how noise scheduling and progressive neuron activation jointly promote coarse-to-fine feature learning, prevent feature entanglement, and improve generalization. We complement this analysis with experiments across four diffusion architectures and multiple datasets.

During the rebuttal, we made several substantial additions and clarifications:

  1. Strengthened theoretical justification. We clarified the Gaussian projection bound, the capped bias schedule governing feature activation, and the local Polyak–Łojasiewicz (PL) condition. We also explained the correspondence between the theoretical and practical noise and capacity scheduling mechanisms.

  2. Added extensive ablation and sensitivity experiments. We evaluated initial sparsity ratios from 5% to 75% on CIFAR-10 and CelebA-64, as well as different noise curricula, demonstrating the benefits of joint scheduling across multiple configurations.

  3. Extended validation to higher resolution. We conducted new pixel-space CelebA-128 experiments, where joint scheduling achieved a 24.4% FID reduction compared with uniform training.

  4. Quantified computational efficiency. We added GPU memory and throughput measurements, demonstrating 48.7% lower memory usage and 37.3% higher throughput at 50% sparsity under progressive widening. We also consolidated quantitative comparisons and corrected presentation inconsistencies.

Reviewer feedback and score updates:

  • Reviewer ES6w (score: 3): Has not yet provided a post-rebuttal assessment.

  • Reviewer pk7j (initial score: 1, increased to 2): Explicitly acknowledged that our responses addressed their concerns and raised the score to 2. The reviewer retained some uncertainty because they had not fully verified the lengthy manuscript.

  • Reviewer MNR4 (score: 2, have not responded to the latest results): Acknowledged that our additional experiments addressed several original concerns but maintained the score, citing the lack of end-to-end training-time measurements and uncertainty regarding the complete theoretical proofs.

Following Reviewer MNR4's feedback, we submitted an additional computational-efficiency analysis estimating training-update time across the full 200k-step schedule. The results show a 6.7% reduction in total training-update time and a 13.4% reduction during the first 100k steps, directly addressing the reviewer's remaining empirical concern. However, Reviewer MNR4 may not have had an opportunity to review it. We have not received a follow-up response.

We would therefore be particularly grateful if you could facilitate a follow-up discussion during the reviewer discussion period and encourage Reviewer MNR4 to examine our final computational-efficiency analysis and provide an updated assessment. We would also appreciate it if Reviewer ES6w could review our rebuttal and share their final assessment.

We believe that the substantial theoretical clarifications, extensive new experiments, and reviewers' acknowledgment of resolved concerns have significantly strengthened the submission. We hope these developments, including our final clarification, can be fully considered in the discussion and final evaluation.

Thank you very much for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.