acceptodds
Under review as a conference paper at ICLR 2027

Conditional Diffusion without ERM Oracles: From Finite Neural Training to Sampling Guarantees

Abstract

Conditional diffusion guarantees based on empirical risk minimization do not directly describe the accuracy achieved by finite neural-network training. We establish an end-to-end sampling guarantee for a finite deep ReLU denoiser trained in all layers by finitely many steps of full-batch gradient descent, without an empirical risk minimization oracle. Under explicit conditional tail and regularity assumptions, the resulting reverse sampler achieves conditional total-variation error , averaged over the conditioning variable, for the conditional law at a fixed positive diffusion stopping time. The sufficient clean-sample size is , where combines the data and conditioning dimensions, is the integer regularity order in the conditional-law assumptions, and is the effective smoothness. Our analysis derives denoising regularity and approximation bounds from the conditional data law and transfers population learning guarantees to the actual finite network. The key step uses Duhamel-based trajectory comparisons to control how discrepancies between neural and kernel training accumulate over a finite horizon and affect predictions at unseen inputs. This yields an explicit tradeoff between training duration and sufficient network width, without a Polyak-Lojasiewicz condition or a positive lower bound on the empirical kernel spectrum. The bounds keep clipping, learning, and finite-width errors explicit. Recovering the original conditional law additionally requires smoothing control and a corresponding resource calibration.

Primary area:
generative models
Keywords:
conditional diffusion models, statistical learning theory, nonconvex optimization, finite-time analysis, neural tangent kernels, early stopping, Duhamel analysis, sample complexity

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f19DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies end to end sampling guarantees for conditional diffusion models when the denoising network is a finite deep ReLU neural network trained by a finite number of full-batch gradient descent steps. Unlike existing analyses that assume an empirical risk minimization oracle or rely on infinite-width/kernel limits, the paper establishes conditional total-variation guarantees for the actual finite neural network after finite training. Under conditional tail and regularity assumptions, the authors derive denoising regularity and approximation bounds from the conditional data distribution, establish population learning guarantees for the limiting kernel predictor, and then transfer these guarantees to the finite trained network.

Critical strengths
  1. A major strength is that the analysis moves beyond idealized empirical-risk minimization and infinite-width assumptions to the practically relevant setting of a finite deep ReLU network trained for finitely many gradient steps.
  2. The comparison among neural, finite-kernel, and limiting-kernel trajectories provides a coherent mechanism for transferring population/kernel guarantees to finite neural training. Explicitly accounting for kernel drift, nonlinear errors, and finite-width approximation over a finite training horizon is a useful theoretical framework.
  3. The paper obtains its guarantees without assuming a Polyak–Łojasiewicz condition or a positive lower bound on the empirical kernel spectrum. This is theoretically appealing because such conditions can be difficult to justify for finite neural networks.
Critical weaknesses
  1. Although the paper provides explicit dependencies on sample size, width, training duration, and sampling resources, the abstract does not make clear whether these bounds are quantitatively meaningful or potentially vacuous in realistic regimes. The interaction between network width and training horizon appears central to the result, but it is unclear whether the required width remains computationally feasible when the number of gradient steps is large.
  2. The guarantees rely on explicit conditional tail and regularity assumptions on the data law, as well as assumptions needed for the neural/kernel trajectory comparison. The paper should more clearly characterize which common conditional diffusion settings satisfy these assumptions and how restrictive the resulting smoothness and tail requirements are.
  3. A substantial part of the argument appears to proceed by obtaining guarantees for a limiting-kernel predictor and then transferring them to the finite network through trajectory comparisons. While this transfer is technically nontrivial, the paper should more clearly quantify what genuinely new statistical or sampling behavior is captured by the finite-network analysis compared with existing neural tangent/kernel-based guarantees. Otherwise, the contribution may be viewed primarily as a technical extension of kernel-regime analysis.
Questions for authors
  1. How large are the resulting width and sample-size requirements in representative parameter regimes? Can the authors provide a concrete scaling example showing whether the sufficient width remains feasible for a practically relevant training horizon?
  2. Can the authors give examples of commonly used conditional data distributions that satisfy conditional regularity and tail assumptions, as well as examples that do not?
  3. What is the precise improvement over existing kernel/NTK-based diffusion guarantees? Which part of the final bound fundamentally requires the finite-training, finite-width trajectory analysis, rather than following directly from an existing limiting-kernel result?
4 comments

We thank the reviewer for the detailed summary, and for noting that the analysis moves beyond ERM and infinite-width idealizations and avoids PL and Gram lower-bound conditions.

Weakness 1/Q1: How large are the width and sample requirements? Table 3 (App. I) gives the explicit scales at fixed t0t_0. With s=(D+2)/2s=(D+2)/2 and effective smoothness kˉ=min⁡{k,s}\bar k=\min\{k,s\}:

  • sample size N=O~(ε−[2+(D+1)/kˉ])N=\tilde O(\varepsilon^{-[2+(D+1)/\bar k]});
  • training horizon ηR⋆=O~(ε−2s/kˉ)\eta R_\star=\tilde O(\varepsilon^{-2s/\bar k});
  • final width m=O~(ε−(6+26s/kˉ))m=\tilde O(\varepsilon^{-(6+26s/\bar k)}).

Once k≥⌈s⌉k\ge\lceil s\rceil, these become N=O~(ε−(4−2/(D+2)))N=\tilde O(\varepsilon^{-(4-2/(D+2))}), ηR⋆=O~(ε−2)\eta R_\star=\tilde O(\varepsilon^{-2}) and m=O~(ε−32)m=\tilde O(\varepsilon^{-32}). The horizon and width exponents in ε\varepsilon then do not depend on DD, because the capacity exponent cancels. For example, D=2D=2 gives N=O~(ε−3.5)N=\tilde O(\varepsilon^{-3.5}) and m=O~(N64/7)m=\tilde O(N^{64/7}).

These widths are far beyond practical sizes, and we do not present them as guidance. Worst-case polynomial widths of this kind are standard in finite-width analyses of gradient training: m=Ω~(n24L12)m=\tilde\Omega(n^{24}L^{12}) (Allen-Zhu et al., 2019) and m=Ω~(n8L12)m=\tilde\Omega(n^{8}L^{12}) (Zou & Gu, 2019) for training-loss convergence alone; m≳n(6β+12)/(2β+γ)m\gtrsim n^{(6\beta+12)/(2\beta+\gamma)}, up to depth factors, for generalization in regression (Zhou et al., 2026, Cor. 6); and m≳poly(d) N26+4/(d+3)m\gtrsim\mathrm{poly}(d)\,N^{26+4/(d+3)} in the closest diffusion analysis (Han et al., Thm. 3.13). Several of these also require data-separation or Gram eigenvalue conditions. The revised abstract and Section 4 state this explicitly.

Weakness 2/Q2: Which distributions satisfy the assumptions, and how restrictive are they?

  • Tails (Asm. 3.1): EeλY∥Y∥≤MY\mathbb E e^{\lambda_Y\|Y\|}\le M_Y and sup⁡y∫eλX∥z∥μy(dz)≤MX\sup_y\int e^{\lambda_X\|z\|}\mu_y(dz)\le M_X. This allows unbounded X0X_0 and YY and exponential rather than Gaussian tails, which is weaker than the Gaussian envelopes of Fu et al. and Hu et al. The restrictive part is uniformity in yy, which excludes location families with unbounded conditional mean. As explained in our reply to Reviewer saP5 (Q1), this uniformity can be localized; Remark D.6 of the revision records the weaker condition.
  • Regularity (Asm. 3.2): ∂yαμy=Sαμy\partial_y^\alpha\mu_y=S_\alpha\mu_y with ∣Sα(z,y)∣≤Cα(1+∥y∥)aα(1+∥z∥)bα|S_\alpha(z,y)|\le C_\alpha(1+\|y\|)^{a_\alpha}(1+\|z\|)^{b_\alpha}. This is regularity in yy of the law only: smoothness in xx and tt comes from Gaussian noising, and no density of X0∣YX_0\mid Y is needed at fixed t0t_0. At fixed t0t_0, the exponents aα,bαa_\alpha,b_\alpha change only constants; they affect rates only through χ\chi in the original-law result (Cor. 1).
  • Satisfied (Section H): Gaussian location–scale families N(m(y),Σ(y))N(m(y),\Sigma(y)) with bounded smooth mean and cI⪯Σ(y)⪯CIcI\preceq\Sigma(y)\preceq CI, e.g. m(y)=atanh⁡(v⊤y)m(y)=a\tanh(v^\top y); finite mixtures with logistic weights, which are non-log-concave when the components are separated, with bα=0b_\alpha=0 (Prop. H.2); noisy conditioning Y=h(X0)+ξY=h(X_0)+\xi with compactly supported X0X_0, where aα=bα=0a_\alpha=b_\alpha=0; and fixed atomic components (stopped law only).
  • Not satisfied: X0=Y+ZX_0=Y+Z with Gaussian Y,ZY,Z, which fails Asm. 3.1 since E[eλ∣X0∣∣Y=y]≥eλ∣y∣\mathbb E[e^{\lambda|X_0|}\mid Y=y]\ge e^{\lambda|y|} (it is covered by the localized condition of Remark D.6); and conditional laws whose support moves with yy, such as δg(y)\delta_{g(y)}, which fail Asm. 3.2.
  • Trajectory comparison: this imposes no assumption on the data law. Its conditions are design choices: fixed depth L≥3L\ge3, bounded adjacent width ratios (eq. 4), η≤1/(4L)\eta\le1/(4L), and inputs mapped to the sphere.

The revision moves these examples and non-examples into Section 3.

Weakness 3/Q3: Improvement over kernel-based guarantees. The closest NTK-based diffusion guarantee (Han et al.) is unconditional and two-layer with a fixed readout, assumes a Gram eigenvalue lower bound, and controls a weighted L2L^2 score error rather than TV. A limiting-kernel result bounds only Eker\mathcal E_{\mathrm{ker}}, the error of a predictor that is never computed; the trained network also requires Eopt\mathcal E_{\mathrm{opt}} and Ewidth\mathcal E_{\mathrm{width}} (Prop. 5.1, App. C.6), which produce the trade-off above. No existing limiting-kernel result applies directly to the time-atomic training law, so the Eker\mathcal E_{\mathrm{ker}} bound is also proved here (general response, item 3). The analysis is in the kernel regime, and we do not claim feature-learning effects.

Reviewer_f19DOpenReview ↗

Thank you for the rebuttal

Thank you for the rebuttal, which addresses several specific points, but my main concern regarding the precise improvement of Q3 remains unclear. I prefer to maintain my original score.

We thank the reviewer for the reply. Below we state the Q3 improvement more precisely.

(a) What a kernel guarantee controls, and what ours controls. A limiting-kernel (NTK) guarantee bounds the error of a kernel predictor: kernel ridge regression or kernel gradient flow. The sampler, however, runs with the trained network FθR⋆−Fθ0F_{\theta_{R_\star}}-F_{\theta_0}. Our decomposition (eq. 15) separates the two: EY TV≤Eterm+Edisc+CW (Eclip+Eopt+Ewidth+Eker).\mathbb E_Y\,\mathrm{TV}\le\mathcal E_{\mathrm{term}}+\mathcal E_{\mathrm{disc}}+C\sqrt{W\,(\mathcal E_{\mathrm{clip}}+\mathcal E_{\mathrm{opt}}+\mathcal E_{\mathrm{width}}+\mathcal E_{\mathrm{ker}})}. A kernel analysis bounds Eker\mathcal E_{\mathrm{ker}} only. Eopt+Ewidth\mathcal E_{\mathrm{opt}}+\mathcal E_{\mathrm{width}} is the gap between the finite network trained by GD and the kernel predictor at the same horizon. Prop. 5.1 bounds this gap by (1+h)1/6(h+h2)m−1/6(1+h)^{1/6}(h+h^2)m^{-1/6}, where h=ηR⋆h=\eta R_\star. Without this term, a kernel bound says nothing about the network that is actually trained and sampled from.

(b) Why this term needs a finite-width analysis.

  • Growing horizon. Early stopping is the regularizer, so h=O~(Nℓ)h=\widetilde O(N^{\ell}) grows with the sample size. The network-to-kernel gap must therefore be controlled over a growing horizon, which forces m≳(1+h)13εcoup−6m\gtrsim(1+h)^{13}\varepsilon_{\mathrm{coup}}^{-6} (eq. 59). This horizon–width trade-off determines m=O~(ε−32)m=\widetilde O(\varepsilon^{-32}), and it is absent from limiting-kernel analyses.
  • No eigenvalue lower bound. Existing finite-width diffusion results control the gap through a lower bound on the empirical Gram eigenvalue. For diffusion training, the inputs (Xt,Y,t)(X_t,Y,t) come from the time-atomic design with unbounded data, so such a bound is an extra assumption on the training design. We instead use the discrete Duhamel identity: gradient descent with η≤1/(4L)\eta\le 1/(4L) does not amplify errors (∥I−ηNK∥≤1\|I-\tfrac{\eta}{N}K\|\le1), so the comparison needs no eigenvalue bound. This follows the regression analysis of Zhou et al. (2026).
  • Deep, all-trained network. All LL layers are trained, including the readout, and the widths may differ (eq. 4). The relevant kernel is therefore the full deep NTK (Prop. E.3), not the two-layer kernel with a fixed readout. The Eker\mathcal E_{\mathrm{ker}} bound is also proved for the time-atomic training law, using a capacity bound that holds for every input law (Prop. E.19).

Comparison with the closest neural diffusion guarantees.

Han et al.Strasman et al. (2026)This work
Settingunconditionalunconditionalconditional
Networktwo-layer, fixed readouttwo-layer, first layer traineddeep, all layers incl. readout trained
Spectral assumptionempirical Gram eigenvalue lower boundnone (small eigenvalues of K∞K_\infty handled by spectral truncation)none
Guaranteeweighted L2L^2 score errorW2W_2 samplingaveraged conditional TV after sampling

Relative to Han et al., the improvement is that no Gram eigenvalue lower bound is needed. Strasman et al. also avoid this bound in the two-layer case; relative to them, the improvement is the extension to deep networks with all layers trained, and to conditional guarantees in TV.

In short, the analysis is in the kernel regime, so the contribution is not a faster rate or feature learning. The guarantee applies to the deep network actually trained by finite GD, without a Gram eigenvalue lower bound, and composes with conditional sampling in TV.

Dear Reviewer f19D,

Thank you for your reply. Our follow-up above separates what kernel-based guarantees control (Eker\mathcal E_{\mathrm{ker}}) from what our analysis adds (Eopt+Ewidth\mathcal E_{\mathrm{opt}}+\mathcal E_{\mathrm{width}} for the trained deep network), and compares it with Han et al. and Strasman et al. If any part of Q3 is still unclear, we would be grateful if you could point to it, so we can clarify before the discussion closes.

Reviewer_saP5OpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether we can start from finite conditional data, train a finite deep neural network for a prescribed number of gradient-descent iterations, and obtain an end-to-end conditional sampling guarantee for the resulting network.

The paper sits between two existing lines of work. On one hand, conditional-diffusion papers (such as Fu et al., 2024; Tang et al., 2025; and Hu et al., 2025) derive statistical and sampling guarantees under regularity assumptions on the conditional distribution, but the learned score estimator is an oracle that serves as the minimizer of the ERM training objective (i.e., the effect of finite optimization is discarded). On the other hand, Han et al. (2024) and Gaur et al. (2026) study finite training of diffusion models, but in the unconditional setting. The paper tries to combine these two directions.

The main result shows that this can be done for a ReLU network trained in all layers by full-batch gradient descent. For a fixed stopping time, the authors give finite requirements on the number of training samples, gradient-descent steps, network width, and reverse-sampling steps that guarantee the learned sampler has vanishing averaged conditional total-variation error.

From my understanding, the proof strategy is to control the population denoising error of the finitely trained neural network by introducing two intermediate kernel models. The authors then decompose the learning error into three terms: the prediction error of the limiting NTK learner relative to the denoising target (Eker\mathcal{E}{\mathrm{ker}}), the discrepancy between finite-width kernel training and the limiting NTK (Ewidth\mathcal{E}\mathrm{width}), and the discrepancy between the actual nonlinear network and its initialization-kernel dynamics (Eopt\mathcal{E}\mathrm{opt}). Thus, the proof first studies whether the target can be learned in an analytically tractable RKHS, and then shows that the actual finite network stays sufficiently close to this kernel learner over the prescribed training horizon.

The first step is therefore to establish that the conditional denoising target is learnable by the limiting NTK under the actual diffusion training distribution. The authors derive regularity of the denoising target from assumptions on the conditional law and use this regularity to construct bounded RKHS approximants. They then combine these approximation bounds with capacity and early-stopping arguments to control the limiting-NTK prediction error.

The second step transfers this kernel guarantee back to the network that is actually trained. The authors first compare the finite initialization kernel with the limiting NTK, and then compare the nonlinear neural trajectory with the finite-kernel trajectory. To do this, they control parameter movement, tangent-kernel drift, and nonlinear prediction remainders throughout the finite training horizon and propagate these perturbations using a Duhamel-type trajectory comparison.

Many of the individual tools used in the proof are drawn from existing literature, as the paper itself states in Appendix A.3. The main technical contribution is in making these ingredients compatible with the conditional setting. First, the paper adapts existing NTK approximation and capacity tools to the actual diffusion training distribution, which is supported only on discrete time slices. Second, it builds on the neural-to-kernel trajectory comparison framework of Zhou et al. (2026), where the general strategy for comparing neural, finite-kernel, and limiting-kernel trajectories is already developed. The authors extend this analysis to their setting, which includes unequal hidden-layer widths, a trainable Gaussian readout, and population control at fresh inputs. Third, the paper uses an existing diffusion-sampling result from Benton et al. (2024), which requires accumulated score error rather than denoising error. Since the learning analysis instead controls population denoising error, the authors choose the distribution of training times so that the two quantities are directly related.

I am happy to be corrected if my reading of these contributions is not fairly correct. I did not check the proofs line by line, but from skimming through the arguments and given the overlap with prior work, I lean to believe that there are no major concerns about the mathematical arguments.

Critical strengths

The paper is well written and clear about its objective, methodology, and position among related work. The authors made it easy to follow their message. In fact, my summary above is, to a great extent, a rephrasing of certain parts in the intro, proof sketch section, and appendix.

Critical weaknesses

I have a few questions about the assumptions.

  1. Assumption 3.1 requires that

    \mathbb E\Big[e^{\lambda_X\vert X_0\vert}\mid Y=y\Big]<\infty,$$ while the final guarantee is only averaged over $Y$. This assumption is used in different places in the proof, but I am not sure which one requires this restrictive worst-case assumption. From my understanding, this assumption was crucial in Appendix D. However, when you look at (113), it seems the supremum is evaluated over a neighborhood $R$, not globally. My question is whether the assumption can be relaxed to local neighborhoods.
  2. Assumption 3.2 requires that ∂yαμy=Sαμy\partial_y^\alpha\mu_y=S_\alpha\mu_y. This assumption is less familiar than a Holder assumption on a conditional density. I struggled to grasp it. It would help to elaborate on the choice of this assumption and give several examples of densities that satisfy and fail it. What does it bring to the argument that alternative assumptions in the related literature do not?

  3. The training objective is not simply FθF_\theta, but Fθc=Fθ−Fθ0F_\theta^c=F_\theta-F_{\theta_0}, so the initial prediction is exactly zero even though all parameters are randomly initialized, as the paper explains. Why did the authors make such a choice? Do the authors propose to use it during training, or is it just a technicality for the proof?

Questions for authors

See above.

3 comments

We thank the reviewer for the accurate summary and the positive assessment. On informativeness, the general response now sets out our technical contributions relative to prior work.

Q1: Where is the uniform-in-yy tail bound needed, and can it be localized? The reviewer is right that the target envelope (113) is only evaluated on ∥x∥+∥y∥≤R\|x\|+\|y\|\le R. We checked every use of the supremum; each can be weakened:

  1. Posterior moments (Lemma D.5). The proof lower-bounds the evidence using μy(∥z∥≤r0)≥1/2\mu_y(\|z\|\le r_0)\ge 1/2 with r0=λX−1log⁡(2MX)r_0=\lambda_X^{-1}\log(2M_X), and bounds the posterior tail by MXe−λXuM_Xe^{-\lambda_X u}, both at the same yy. Since (113) and the spherical extension (App. E.3) use only ∥y∥≤R\|y\|\le R, MXM_X can be replaced by MX(R)=sup⁡∥y∥≤RE[eλX∥X0∥∣Y=y]M_X(R)=\sup_{\|y\|\le R}\mathbb E[e^{\lambda_X\|X_0\|}\mid Y=y]. If MX(R)≤CecRM_X(R)\le Ce^{cR}, then r0=O(1+R)r_0=O(1+R) and (110) keeps its form Cq(1+R)2q(1+t0−1)qC_q(1+R)^{2q}(1+t_0^{-1})^q, with larger constants. Since ρN=Θ(log⁡(N/δ)+T)\rho_N=\Theta(\log(N/\delta)+T), this changes only logarithmic factors.
  2. Sampling terms (Lemmas C.1–C.2). These are applied for each yy and averaged over YY, and EYdx+M2(Y)≤dx+E∥X0∥2\mathbb E_Y\sqrt{d_x+M_2(Y)}\le\sqrt{d_x+\mathbb E\|X_0\|^2} by Jensen, where M2(y)=∫∥z∥2μy(dz)M_2(y)=\int\|z\|^2\mu_y(dz). Only the marginal second moment is needed.
  3. Clipping and localization (Lemmas C.4–C.5, eq. 42). These use P(∥X0∥>a)≤E[eλX∥X0∥] e−λXaP(\|X_0\|>a)\le\mathbb E[e^{\lambda_X\|X_0\|}]\,e^{-\lambda_X a}, so only the marginal exponential moment is needed. The same holds for the original-law smoothing bias (Lemma C.13), which is averaged over YY.

The stopped-law result therefore holds under the weaker condition EeλX∥X0∥<∞\mathbb E e^{\lambda_X\|X_0\|}<\infty and MX(R)≤CecRM_X(R)\le Ce^{cR}, with Assumption 3.2 unchanged. This covers the location family X0=Y+ZX_0=Y+Z, which the current assumption excludes: there MX(R)≤2eλXR+λX2/2M_X(R)\le 2e^{\lambda_X R+\lambda_X^2/2}. The revision records this condition and argument in Remark D.6, with a pointer from Section 3.

Q2: The choice of Assumption 3.2. The assumption requires ∂yαμy=Sαμy\partial_y^\alpha\mu_y=S_\alpha\mu_y with ∣Sα(z,y)∣≤Cα(1+∥y∥)aα(1+∥z∥)bα|S_\alpha(z,y)|\le C_\alpha(1+\|y\|)^{a_\alpha}(1+\|z\|)^{b_\alpha}. When μy\mu_y has a positive density p(z∣y)p(z\mid y) on a common support, Sα=∂yαp(z∣y)/p(z∣y)S_\alpha=\partial_y^\alpha p(z\mid y)/p(z\mid y), the yy-analogue of a score. For example, μy=N(m(y),σ2)\mu_y=N(m(y),\sigma^2) gives S1(z,y)=m′(y)(z−m(y))/σ2S_1(z,y)=m'(y)(z-m(y))/\sigma^2, which is linear in zz; with bounded m′m', a1=0a_1=0 and b1=1b_1=1.

What it brings: the denoiser f⋆f^\star is a ratio of integrals against μy\mu_y, and the assumption lets yy-derivatives pass through them, ∂yβ∫g dμy=∫g Sβ dμy\partial_y^\beta\int g\,d\mu_y=\int g\,S_\beta\,d\mu_y (Lemma D.7). Combined with Gaussian derivatives in xx and tt and the tail bounds, this gives the polynomial derivative envelope (Prop. D.8) behind the approximation step.

Hölder-density assumptions (Fu et al.; Hu et al.) control the smoothness of a density jointly in (x,y)(x,y). Our assumption controls only how the law moves with yy: smoothness in xx comes from noising, no density of X0∣YX_0\mid Y is needed at fixed t0t_0 (atomic components are allowed), and no density of YY is needed.

Examples (Section H): Gaussian location–scale families with bounded smooth mean and elliptic covariance; mixtures with logistic weights, including non-log-concave ones, with bα=0b_\alpha=0 (Prop. H.2); and Bayes posteriors for Y=h(X0)+ξY=h(X_0)+\xi with compactly supported X0X_0, with aα=bα=0a_\alpha=b_\alpha=0. The assumption fails when the support moves with yy, as for δg(y)\delta_{g(y)}, because the derivative then has a component not dominated by μy\mu_y. The revision brings these examples into Section 3.

Q3: Centering. Centering is part of the algorithm, used both in training and at prediction (Algorithm 1, lines 10–15), at the cost of storing the initial network (a constant-factor overhead, App. I). With independent N(0,1)N(0,1) initialization, Fθ0F_{\theta_0} is a random function of order one. Centering makes the initial prediction exactly zero without changing any gradient, so the network can be compared with zero-start kernel GD. Without it, the kernel comparison would have the random shifted target Z−Fθ0(X)Z-F_{\theta_0}(X), and the population bound would need a separate control of the initialization function (Remark F.2). Related analyses obtain the same effect with an antisymmetric paired initialization (e.g. Zhou et al., 2026); centering achieves it while keeping every layer, including the readout, independently Gaussian and trained.

Dear Reviewer saP5,

Thank you again for your positive review and for your careful questions. As the author–reviewer discussion closes on November 18, we wanted to check whether our response addresses them. Following your question on Assumption 3.1, the revision adds Remark D.6, which shows that the uniform-in-yy tail bound can be replaced by a localized condition covering location families such as X0=Y+ZX_0=Y+Z, and Section 3 now includes examples and non-examples of Assumptions 3.1–3.2. We are glad to clarify anything further before the deadline.

I thank the authors for their detailed response, which reinforces my confidence in my score.

Reviewer_a3yqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies conditional diffusion with a finite deep ReLU network trained by gradient descent. It combines an analysis of neural-network training with regression and sampling error bounds to obtain a guarantee for the generated distribution.

Critical strengths
  1. The paper studies a relevant question by giving a guarantee for the output of finite gradient-descent training, rather than assuming an empirical risk minimizer.

  2. The choice of the training-time distribution gives a clean connection between the regression error and the score error used in the sampling analysis.

Critical weaknesses
  1. My main concern is that the methodological contribution is limited. The central proof compares neural-network training with initialization-kernel training and then with limiting-kernel training. As Appendix A.3 acknowledges, this general framework already exists in prior work. The final sampling guarantee also uses existing error bounds for reverse diffusion. The paper should state more clearly which arguments are already known and which arguments are new. Based on the current presentation, the contribution appears to be an incremental extension of existing methods.

  2. The better sample bound concerns a smoothed distribution at a fixed positive diffusion time. The guarantee becomes much weaker when recovering the original data distribution. For example, Remark B.10 gives a sufficient sample bound of O~(ε−40)\widetilde O(\varepsilon^{-40}), compared with O~(ε−5)\widetilde O(\varepsilon^{-5}) for the smoothed target in the same example. The required network width is also extremely large, so the guarantee gives little practical guidance on the resources needed for finite training.

  3. There are also no experiments, even on the examples considered in the theoretical analysis. The paper is also difficult to follow, many parts of the main theorem can only be understood by refering several appendices.

Questions for authors
  1. Beyond the existing neural-to-kernel comparison framework acknowledged in Appendix A.3, what are the main new technical ideas? Which difficulties arise specifically from conditional diffusion?

  2. How does the original-distribution guarantee compare with existing results, and can its large sample-size exponent be improved?

4 comments

We thank the reviewer for the careful reading, and for noting that the paper gives a guarantee for finite GD training rather than an ERM minimizer.

Weakness 1/Q1: New ideas, and difficulties specific to conditional diffusion. The general response sets out our technical contributions relative to prior work; we agree the submission should have stated this in the main text, and the revision does. Below we separate the difficulties by source, and for each give the obstacle and the results that resolve it.

Specific to conditioning.

  • Regularity in yy. Approximation needs joint smoothness of the denoiser f⋆(x,y,t)=E[X0∣Xt=x,Y=y]f^\star(x,y,t)=\mathbb E[X_0\mid X_t=x,Y=y], but we assume no density of YY (or of X0∣YX_0\mid Y at fixed t0t_0), with YY unbounded. Assumption 3.2 lets yy-derivatives pass through the posterior integrals (Lemma D.7); exponential tails control the posterior moments (Lemma D.5); and quotient induction gives the derivative envelope with explicit t0t_0-dependence (Prop. D.8, eq. 113).
  • Averaged conditional guarantee. The sampling bound of Benton et al. is applied for each fixed yy and then averaged over a fresh YY, which our population learning bound controls (Lemmas C.2–C.3).

Specific to diffusion.

  • Singular training law. Inputs lie on finitely many time slices, so PXP_X is singular with respect to spherical measure, and approximation in L2L^2 of that measure or capacity bounds for a fixed reference law do not control the error. We build uniform RKHS approximants (App. E.3) and prove dfP(λ)≲λ−γ\mathrm{df}_P(\lambda)\lesssim\lambda^{-\gamma} for every input law PP (Prop. E.19), which feeds the early-stopping bound (eq. 18, App. C.5).
  • Score versus denoising error. The sampler needs accumulated score error, while regression controls denoising error; the score-weighted time law (2) makes them equal (eq. 14, Lemma C.3).

Specific to the architecture.

  • With independent initialization and a trained Gaussian readout, the cancellation in Zhou et al.'s Lemma 20 is unavailable and every block enters the kernel. We prove perturbation bounds uniform over activation patterns in the movement ball (Thm. F.8), concentration of the full initialization kernel including the readout block (Thm. F.10), a movement bootstrap keeping iterates within O(1+h)O(\sqrt{1+h}) of initialization (Prop. F.13), and the two Duhamel comparisons at fresh inputs (Prop. 5.1, Lemma F.11).

Weakness 2/Q2: Original-law guarantee. The ε−40\varepsilon^{-40} in Remark B.10 is the least favorable case: D=2D = 2 and k=β=1k = \beta = 1, for original-law recovery. In the same example the fixed-t0t_0 exponent is 55, and it is 4−2/(D+2)4 - 2/(D+2) once k≥⌈(D+2)/2⌉k \ge \lceil (D+2)/2 \rceil (Cor. B.7). The larger original-law exponent has two sources. First, the denoiser's derivative envelope grows like t0−χt_0^{-\chi} as t0→0t_0 \to 0. Second, the neural–kernel comparison condition (pgatep_{\mathrm{gate}} in eq. 13) must hold at that shrinking t0t_0; with the statistical exponent alone (3434), it would fail.

On comparison with existing results: ERM-based conditional analyses (Fu et al., 2024; Tang et al., 2025; Hu et al., 2025) obtain much faster original-law rates because they assume access to the minimizer. We know of no prior original-law guarantee for a finitely trained conditional network, so the gap measures the cost of analyzing the trained network itself. The exponent may be improvable, for example by using a lower regularity order uu as in Remark B.10, or a sharper small-time regularity envelope. We state this as an open direction rather than a claim.

Weakness 3: Experiments. Our contribution and claims are theoretical: Theorem 1 gives sufficient conditions under which finitely trained networks achieve the stated TV guarantee, and the paper makes no empirical performance claim. The procedure combines standard components (denoising regression under the score-matching time weighting, gradient descent on a ReLU network, and the exponential-integrator sampler analyzed by Benton et al., 2024). It adds simplifications (one frozen noise draw per pair, full-batch GD, centering) chosen so that the analyzed procedure is exactly the one the theorem covers. As Section G.1 notes, the schedule is specified through class bounds and proof constants rather than tuned values, so a numerical run would illustrate one instantiation rather than test the result. Several of the theoretical works compared in Table 1 likewise present their guarantees without experiments, including Fu et al. (2024), Hu et al. (2025, ICLR), and Gaur et al. (2026). Where comparable works report numerical results (Yuan et al., 2023; Tang et al., 2025; Gupta et al., 2024), these illustrate qualitative effects rather than test the stated rates or sufficient conditions.

Weakness 3: Readability. Theorem 1 is now self-contained. It now gives the scalings of NN, R⋆R_\star, mm, MM, and TT directly, and the calibration references (Section G.1, eqs. 44 and 59, Lemma B.5) are collected in Remark 4.1.

Dear Reviewer a3yq,

Thank you again for your review. As the author–reviewer discussion closes on November 18, we wanted to check whether our response and the revised PDF address your concerns. In particular, the general response and the new paragraph in Section 1 set out which arguments are new relative to Zhou et al. (2026), Theorem 1 is now self-contained (Remark 4.1), and our reply explains the source of the ε−40\varepsilon^{-40} exponent in Remark B.10. If these points resolve your concerns, we would be grateful if you would consider them in your final assessment. We are glad to clarify anything further before the deadline.

Dear Reviewer a3yq,

As today is the last day of the discussion period, we wanted to check whether our response addressed your concerns. In brief, it (i) states what is new relative to Zhou et al. (2026) and the other works in Table 1, (ii) explains where the original-law exponent comes from and that it is a sufficient condition, not a prescription, and (iii) explains the theoretical scope and how it compares with the experiments in related theoretical work. The revision also makes Theorem 1 self-contained and moves the examples of the assumptions into Section 3 for readability.

If any point remains unclear, we would be glad to clarify it before the discussion closes. If our response resolves your concerns, we would appreciate it if you would consider updating your assessment.

Thank you again for your time and feedback.

Thank you for the detailed response and revisions. I appreciate the clarification of the contributions and the differences from prior work. However, I still think the main technical ideas are largely based on existing methods, and the new extensions are rather limited. The theoretical bounds also remain very conservative. Therefore, I will keep my original score.

Comments on the paper

1 comment

General response: what is new, and what we revised

We thank all reviewers. The reviews note that the paper gives a guarantee for the output of finite gradient-descent training rather than an ERM minimizer (a3yq), moves beyond ERM-oracle and infinite-width assumptions (f19D), and is clear about its position in the literature (saP5). None of the reviews points to an error in the proofs. The shared concern is which arguments are new. We answer this here and have revised the paper to state it in the main text.

Result. To our knowledge, Theorem 1 is the first sampling guarantee for conditional diffusion in which a deep ReLU denoiser, trained in all layers by finitely many GD steps, is analyzed without an ERM oracle, a PL condition, or a lower bound on the Gram spectrum. The last two properties come from the neural-to-kernel comparison approach of Zhou et al. (2026) for regression; they are new for diffusion.

Technical contributions relative to prior work.

  1. Trajectory comparison. Existing neural-to-kernel trajectory comparisons (Zhou et al., 2026), which pass through initialization-kernel and limiting-kernel GD, are developed for regression with equal widths, a paired last layer, and a fixed signed readout; their nonsymmetric extension (their Sec. 5) also keeps the readout fixed. In the paired setting only the last-layer block enters the initialization kernel, and their backward perturbation bound (Lemma 20) uses a cancellation from that initialization. Our denoiser trains every layer, including the readout, from independent Gaussian initialization. Every layer and the readout then contribute to the kernel (eq. 120) and the cancellation is unavailable, so we prove the perturbation bounds and kernel concentration for this architecture directly (Thm. F.8, F.10). Our width condition (eq. 4) also allows nonincreasing hidden widths with bounded adjacent ratios κpyr\kappa_{\mathrm{pyr}}, covering the pyramidal architectures used in prior deep-ReLU training analyses (Nguyen & Mondelli, 2020) and containing equal widths as the case κpyr=1\kappa_{\mathrm{pyr}} = 1; the perturbation bounds track this ratio explicitly.
  2. How the classical Duhamel identity is used. We apply the discrete variation-of-constants identity twice. Between the network and initialization-kernel GD, the one-step perturbations are kernel drift across all layers and the trained readout, plus the nonlinear remainder. Between the initialization and limiting kernels, the perturbation is their uniform difference, including the readout block. Because η≤1/(4L)\eta \le 1/(4L), a kernel GD step never enlarges an existing discrepancy between the two trajectories, so the total discrepancy is at most the sum of the one-step errors and grows with the horizon hh. The argument never needs earlier errors to decay, which is what a Gram lower bound or PL condition would provide; this is why neither is assumed. This accumulation gives the explicit width–horizon condition m≳(1+h)13εcoup−6m \gtrsim (1+h)^{13}\varepsilon_{\mathrm{coup}}^{-6} (eq. 59). Transferring to fresh inputs through the averaged cross-kernel operator, whose norm does not grow with NN, gives the h+h2h + h^2 dependence (Prop. 5.1, App. F.1).
  3. Learning under the diffusion training law. Zhou et al.'s theorem assumes capacity and source conditions for the input law. Here the training law is supported on finitely many time slices (zero spherical measure) and the target is a posterior mean. We prove the capacity bound for every input law (Prop. E.19, via the rank–tail argument of Vakili et al.) and build bounded uniform approximants from the conditional law in place of a source condition (App. D.2, E.3). The time slices are drawn from a score-weighted law, which makes the population denoising error equal to the accumulated score error required by the sampling bound of Benton et al. (2024) (eq. 14).
  4. Regularity from the conditional law. Prior conditional analyses assume Hölder regularity of the conditional density (Fu et al.; Tang et al.; Hu et al.). We derive the target's regularity from relative derivatives of the conditional law, with unbounded YY and no density of YY (App. D).

Existing results give guarantees either for an ERM oracle or for a kernel predictor, not for the network that is actually trained. Items 1–3 close this gap.

Revisions in the uploaded PDF.

  • Thm. 1 is self-contained; calibration details are in Remark 4.1 (a3yq, f19D).
  • Sec. 1 adds the paragraph above and revises contribution 2; App. A.3 is rewritten to match (all).
  • Sec. 3 adds examples and non-examples of Asm. 3.1–3.2 (f19D, saP5).
  • The abstract and Sec. 4 state that the resource bounds are polynomial sufficient conditions (a3yq, f19D).
  • Remark D.6 gives the localized tail condition (saP5).
  • The abstract, Sec. 1, and App. F.1 credit Zhou et al. for the absence of PL and Gram conditions; App. A.1 adds Strasman et al. (2026).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.