acceptodds
Under review as a conference paper at ICLR 2027

Spectral Bias and Finite Sample Effects in Denoising Score Matching

Abstract

Score-based diffusion models achieve remarkable performance on high-dimensional tasks, yet the inductive biases that shape their learned scores remain poorly understood. Their training can be viewed as a sequence of denoising regression problems. In the infinite-width limit, a network is described by its NNGP kernel, whose spectrum induces a well-understood spectral bias in supervised regression. In this work, we show that this spectral bias is largely accurate in high-dimensional denoising score matching, and wide networks essentially act as supervised learning on the score. At small diffusion time, we derive two correction terms, one involving a position-dependent ridge and another involving a pointwise bias to the ensemble mean score. These corrections become significant when data have low intrinsic dimension but are embedded in a high-dimensional space with many weak-variance directions. We validate these predictions using finite-width networks on seven synthetic distributions, MNIST, and CIFAR-10, finding close agreement.

Primary area:
generative models
Keywords:
spectral bias, diffusion models, generative models, denoising score matching, statistical mechanics of learning, infinite-width networks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_WBeQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Accurate score estimation is central to the reverse process in diffusion models. This paper studies how network inductive bias and finite training data learned scores in the infinite-width kernel regime. At a fixed small noise level, the authors connect denoising score matching to supervised kernel regression and derive expressions for the ensemble mean and variance of the denoiser. The mean score is represented as a kernel operator applied to the score of an uncertainty-reweighted density. Experiments on synthetic distributions and image data provide empirical support for these predictions.

Critical strengths

The paper provides a useful connection between supervised spectral bias and denoising score matching. In particular, it identifies how spatially varying posterior uncertainty changes the effective regularization and introduces an additional score correction. These predictions offer a detailed description of inductive bias.

Critical weaknesses
  1. The presentation makes the probabilistic setup and approximation regime difficult to follow. For example, in Section 3, the authors should define the parameter prior and the kernel KK more explicitly, and explain clearly the posterior mean in Eq. (3). The notation also overloads μ\mu as both a measure and a sample index.

  2. The relationship to statistical analyses of score estimation could be clarified. Existing work, including "Diffusion Models are Minimax Optimal Distribution Estimators" (Oko et al., 2023) and "Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions" (Azangulov et al., 2025), studies finite-sample guarantees for score estimators and generated distributions under specified function classes and geometric assumptions. The present paper instead characterizes pointwise bias and uncertainty in an equilibrium kernel model. What conclusions about learned scores can this characterization provide that are not captured by existing estimation-rate results, and how do the assumptions compare?

Questions for authors
  1. The experiments use very wide networks. Could the authors test, in one existing setting, whether the predicted score distortion remains qualitatively visible for a finite-width network (used in practice) trained with a standard optimizer at a fixed small noise level? If this lies outside the intended scope, please clarify which aspects of the theory are expected to transfer to practical diffusion training.

  2. Same as Weaknesses 2.

2 comments

Response to Reviewer WBeQ

We thank the reviewer for their thoughtful comments, which helped us improve the paper's clarity. In addition, their question about existing estimation-rate results helped us clarify what our pointwise analysis adds. It predicts where the learned score is systematically distorted and in which direction.

Weakness 1. The presentation makes the probabilistic setup and approximation regime difficult to follow. For example, in Section 3, the authors should define the parameter prior and the kernel KK more explicitly, and explain clearly the posterior mean in Eq. (3). The notation also overloads μ\mu as both a measure and a sample index.

Authors: We apologise for any unclarity in our introduction. We sincerely hope that the referee can re-examine the revised version.

Two sources of randomness define the ensemble: the network weights, which Langevin training samples from the posterior, and the draw of the PP clean training points. The noise draws are averaged in the loss. For a fixed training set, averaging over the weights gives the posterior-mean predictor of Eq. (3), and the ensemble mean is its average over training sets, as the reviewer describes. We now state this in the introduction before the contributions. We also explain why this average is the object of interest: it is the part of the learned score shared by all training sets of size PP, so its deviation from the population score isolates the systematic effect of finite data and of the kernel from sample-specific fluctuations. It is also the quantity that the supervised spectral-bias theory characterizes (Canatar et al., 2021), which makes the comparison direct.

Minor presentation comments:

  • Please expand neural network Gaussian process (NNGP) and neural tangent kernel (NTK) at first mention, in lines 014 and 033 respectively.
  • Please correct s=−fβs = -f\sqrt{\beta} at line 202 to match sθ=−fθ/βs_\theta = -f_\theta/\sqrt{\beta} at line 127.
  • Figure 1's caption should refer to Figure 4 for the d=16d = 16 experiment, rather than Figure 6.
  • Please revise the opening sentence of Appendix D for grammatical clarity.

Weakness 2 & Question 2. The relationship to statistical analyses of score estimation could be clarified. Existing work, including ''Diffusion Models are Minimax Optimal Distribution Estimators'' (Oko et al., 2023) and ''Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions'' (Azangulov et al., 2025), studies finite-sample guarantees for score estimators and generated distributions under specified function classes and geometric assumptions. The present paper instead characterizes pointwise bias and uncertainty in an equilibrium kernel model. What conclusions about learned scores can this characterization provide that are not captured by existing estimation-rate results, and how do the assumptions compare?

Authors: We thank the reviewer for pointing out these references, which we now include. As argued in the revised introduction and related works section, a main novel element of the current work is a directly derived connection with spectral bias theory, which contains a novel element: A gain factor. Some works assume spectral smoothing rather than derive it (Gabriel et al., 2025; Farghly et al., 2025; Chen, 2026), others derive it for linear denoisers for wider β\beta and κ\kappa ranges but do not flesh out the gain factor (Wang, Zavatone-Veth & Pehlevan, 2026, which we now cite) and its implications.

Finally, for distributions having regions with low intrinsic dimension, we further flesh out a combined effect related to having a position-dependent ridge

As the revised manuscript clarifies these novel aspects, and further shows experiments on larger real datasets where both the gain factor and spectral bias are cleanly seen. (gain in Fig. 4a and Fig. 1; spectral bias in Fig. 4b)

We also note that the assumptions and setup in these papers are different. Oko et al. analyze empirical score matching over prescribed network classes under Besov smoothness assumptions. Azangulov et al. assume a smooth distribution supported on a smooth low-dimensional manifold and construct estimators adapted to that geometry. Our setup assumes a smooth, strictly positive density in the ambient space, a specified infinite-width kernel, and Langevin equilibrium at one small noise level. The pointwise prediction uses the Gaussian-equivalence and mean-field approximations described in Section 5.

Question 1. The experiments use very wide networks. Could the authors test, in one existing setting, whether the predicted score distortion remains qualitatively visible for a finite-width network (used in practice) trained with a standard optimizer at a fixed small noise level? If this lies outside the intended scope, please clarify which aspects of the theory are expected to transfer to practical diffusion training.

Authors: We thank the referee for their questions, which motivated us to perform more extensive experiments. These now validate our spectral-bias theory for FCNs on CIFAR10 with P=1000,2500,5000,10000P=1000,2500,5000,10000 training points. We derive the spectrum analytically from the untrained networks' NNGP kernels, and we estimate the ground truth score from the data (See the newly added Fig. 4 in particular panel b).
Despite using finite-width networks and ADAM optimizers, rather than our infinite-width, Langevin-based theory, the results clearly show spectral filtering of the normalized learned score, as our theory predicts.

We further found that the score is overestimated due to the aforementioned gain term; we now emphasize this gain (Eq (7)). This gain factor can be computed analytically, given the NNGP kernel at initialization. Interestingly, dividing the trained neural network by this gain factor improves the test/held-out DSM score in these more realistic settings (see Figure 4a).

Notably, a positive gain factor in the score is equivalent to raising the implied probability by a positive power. Such a distortion increases the score in the regions where the true data distribution is higher and reduces diversity. Our theory offers a potential-theory-based route to analyzing and countering these known issues.

Reviewer_LTLNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses kernel theory to study finite-sample effects in denoising score matching. It predicts how local uncertainty changes the learned score at small noise levels.

Critical strengths
  1. Finite-sample bias, or more generally induction bias, in score estimation is an important problem.

  2. The proposed link between local uncertainty and score bias looks interesting.

Critical weaknesses
  1. The writing is difficult to follow and sloppy. Exact calculations, assumptions, and approximations are not clearly separated.

  2. The paper mixes variation across datasets with uncertainty within a fitted model. The meaning of “variance” is therefore unclear. More specifically, two distinctions matter:

  • This measures uncertainty within the fitted model. It differs from the variation of fitted mean predictions across datasets.
  • The network ff predicts noise. Since the score estimate is −f/β-f/\sqrt{\beta}, its variance is approximately R(x)/βR(x)/\beta.

The paper’s terminology makes these distinctions harder to follow

  1. The novelty beyond existing score-smoothing theory is unclear. The paper should establish what is new in the local regularization, correction D(x)D(x), and self-consistent equations, and demonstrate their accuracy and usefulness.

  2. The experimental descriptions conflict with each other. Section 6 reports 256 synthetic networks and 64 noise draws. The figure captions report 4,096 datasets and closed-form calculations. These procedures are not clearly reconciled, weakening confidence in the empirical validation.

Questions for authors
  1. How do the 256-network and 4,096-dataset experiments relate? Which results appear in the figures?

  2. The Gaussian and mean-field approximations lack clear accuracy conditions and error estimates. Can the authors please clarify and make the assumptions/theorem statement rigorous?

3 comments

Response to Reviewer LTLN

We thank the reviewer for their careful reading and constructive comments.

Weakness 1. The writing is difficult to follow and sloppy. Exact calculations, assumptions, and approximations are not clearly separated.

Authors: We apologize for these deficiencies. We have worked hard to make the revised revision more readable. In particular the introduction, Sec.3, and Sec.4. Hopefully this is now easier to follow.

Specifically our approximations and assumptions are,

  • Modeling choice (loss). The noise-averaged loss, i.e.\ M→∞M\to\infty at fixed κ2\kappa^2 (see comparison with finite MM in the revised version)
  • Modeling choice (training). Weight-decay Langevin dynamics at equilibrium (though see comparison with ADAM in revised version).
  • Modeling choice (kernel limit). The prior over functions is a centred Gaussian process with kernel Kij(x,y)=k(x,y) δijK_{ij}(x,y)=k(x,y)\,\delta_{ij}, differentiable and trace class on L2(μ)L^2(\mu) (Eq. (14)). The derivation holds for any such kernel; the NNGP kernel of a wide network is one instance.
  • Approximation (expansion of ff). f(x0+β ϵ)f(x_0+\sqrt{\beta}\,\epsilon) is Taylor-expanded in β\sqrt{\beta} (App. A.5) and βκ2\tfrac{\sqrt{\beta}}{\kappa^2} (Eq. (24)). It is exact when the expansion parameter is small, i.e., β≪κ2\sqrt{\beta}\ll\kappa^2. The experiments use β=κ2/100\sqrt{\beta}=\kappa^2/100 with β\beta rainging over O(10−3)O(10^{-3}) to O(10−8)O(10^{-8}) for the various plots.

We have added this list at the start of App. A.

Weakness 2. The paper mixes variation across datasets with uncertainty within a fitted model. The meaning of ``variance'' is therefore unclear. More specifically, two distinctions matter:

  • This measures uncertainty within the fitted model. It differs from the variation of fitted mean predictions across datasets.
  • The network (ff) predicts noise. Since the score estimate is (−f/β-f/\sqrt{\beta}), its variance is approximately (R(x)/βR(x)/\beta). The paper's terminology makes these distinctions harder to follow.

Authors: Regarding the reviewer's first bullet: in the paper, ``ensemble variance'' is the total variance of ff, over both the draws of the network weights and the draws of the training set DD. This is Eq. (8), derived in App. A.10, where it is shown that the total variance equals Ry+O(β)R_y + O(\beta). It is the only variance discussed in the paper. The revised text is more explicit about which randomness the ensemble includes: the opening of Sec. 4 now states that the mean and variance are taken over weights and data seeds; Sec. 3 lists the sources of randomness as well.

As for the reviewer's second bullet: the reviewer is correct. Eq. (8) is the variance of the noise predictor ff, and since sθ=−fθ/βs_\theta = -f_\theta/\sqrt{\beta}, the variance of the score estimate is Rx/βR_x/\beta. However, the denoiser itself, x^0(y)=y−βfθ(y)\hat{x}_0(y) = y - \sqrt{\beta} f_\theta(y), has variance βRx\beta R_x, which goes to zero in the small β\beta limit we consider.

Weakness 3. The novelty beyond existing score-smoothing theory is unclear. The paper should establish what is new in the local regularization, correction D(x)D(x), and self-consistent equations, and demonstrate their accuracy and usefulness.

Authors: Spectra smoothing : To the best of our knowledge, existing score-smoothing theory either postulates a smoother (Gabriel et al., 2025; Farghly et al., 2025; Chen, 2026) or derives it from training for linear denoisers (Wang, Zavatone-Veth & Pehlevan, 2026, which we now cite). They also do not describe the gain factor over standard spectral-bias with the score at the target, which we now emphasize in the text. Further on that latter point, multiplying networks trained on real data by an analytically pre-computed factor amounting to this gain improves DSM test/held-out loss. In short, our work fleshes out a direct and simple connection with spectral-bias theory for generic kernels and point-wise predictors, which augments the latter by a gain factor.

To answer the referee regarding what is new in the local regularization and D(x)D(x)

  • The heteroscedastic ridge is, to the best of our knowledge, new; prior works have one scalar (effective ridge) (Canatar et al., 2021, Wang et al., 2026). To demonstrate their accuracy, we refer the referee to Fig. 1 and Fig. 3 in the paper; in the anisotropic embedding, the constant ridge misplaces the mean where RxR_x departs from its average, while the local ridge does not (Fig. 1, bottom row, solid vs.\ dotted). The local ridge also reproduces the variance dip, which the constant ridge underestimates (Fig. 3).
  • The score bias is also new. The smoothed object is s+Ds+D with D=12∇log⁡(κ2+Rx)D=\tfrac12\nabla\log(\kappa^2+R_x), not ss. A fixed smoother applied to the score has no such term, and to our knowledge it does not appear in the literature. We have added to Fig. 1 and Fig. 4 a curve showing the prediction of the heteroscedastic theory without the score bias term (smoothing the score alone using the smoother predicted by the heteroscedastic theory), showing that in the anisotropic embedding the score bias term's contribution to the mean predictor is of the same order as that of the score itself. Without it, the prediction decreases monotonically through the origin for the Gaussian and Student-tt profiles, whereas the ensemble mean is non-monotonic or flat there, and for the three-component mixture it overestimates the central oscillation by roughly a factor of two.
  • Usefulness. These corrections are important when the data have low-dimensional structure, so that the predictor variance RxR_x changes across the data. We now emphasize this in the main contributions and in the opening of Sec. 4 and Sec. 4.2. This is seen in Fig. 1, bottom row. The heteroscedastic ridge is also important on real data (see Fig. 2, left panels).

Weakness 4 & Question 1. The experimental descriptions conflict with each other. Section 6 reports 256 synthetic networks and 64 noise draws. The figure captions report 4,096 datasets and closed-form calculations. These procedures are not clearly reconciled, weakening confidence in the empirical validation. How do the 256-network and 4,096-dataset experiments relate? Which results appear in the figures?

Authors: The paper shows the 4,096-dataset experiments; early in the writing, the experiments were run as Section 6 stated; however, this was later changed. Section 6 should have been updated accordingly, but some of the old text had remained unchanged. We have corrected this oversight, and now the text only refers to the 4,096-dataset experiments, which is what Figures 1 and 4 show.

Question 2. The Gaussian and mean-field approximations lack clear accuracy conditions and error estimates. Can the authors please clarify and make the assumptions/theorem statement rigorous?

Authors: We apologize for these deficiencies. We have worked hard to make the revised revision more readable. In particular the introduction, Sec.3, and Sec.4. Hopefully this is now easier to follow.

Specifically our approximations and assumptions are,

  • Modeling choice (loss). The noise-averaged loss, i.e.\ M→∞M\to\infty at fixed κ2\kappa^2 (see comparison with finite MM in the revised version)
  • Modeling choice (training). Weight-decay Langevin dynamics at equilibrium (though see comparison with ADAM in revised version).
  • Modeling choice (kernel limit). The prior over functions is a centred Gaussian process with kernel Kij(x,y)=k(x,y) δijK_{ij}(x,y)=k(x,y)\,\delta_{ij}, differentiable and trace class on L2(μ)L^2(\mu) (Eq. (14)). The derivation holds for any such kernel; the NNGP kernel of a wide network is one instance.
  • Approximation (expansion of ff). f(x0+β ϵ)f(x_0+\sqrt{\beta}\,\epsilon) is Taylor-expanded in β\sqrt{\beta} (App. A.5) and βκ2\tfrac{\sqrt{\beta}}{\kappa^2} (Eq. (24)). It is exact when the expansion parameter is small, i.e., β≪κ2\sqrt{\beta}\ll\kappa^2. The experiments use β=κ2/100\sqrt{\beta}=\kappa^2/100 with β\beta rainging over O(10−3)O(10^{-3}) to O(10−8)O(10^{-8}) for the various plots.

We have added this list at the start of App. A.

Reviewer_mNdZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how finite training data and kernel inductive bias shape denoising score matching at a fixed, small noise level. Experiments on seven synthetic distributions and downsampled MNIST and grayscale CIFAR-10 compare the predicted means and variances with finite-width network estimators, with larger corrections under anisotropic data geometry.

Critical strengths

The uncertainty-dependent correction provides a useful interpretation of DSM, described in Equations (5) and (7)–(10). The synthetic comparisons help distinguish the proposed effects. Figures 1 and 4 compare constant ridge, the additional score correction, and position-dependent ridge across several distributions.

Critical weaknesses

Concerns

  1. Scope of the image validation and CIFAR-10 preprocessing. The image experiments use downsampled MNIST and 16×1616\times16 grayscale CIFAR-10, with only P=256P=256 images per dataset draw. For CIFAR-10, removing color and downsampling reduces the input dimension from 3072 to 256 and alters the data geometry and kernel spectrum central to the analysis. Why is this simplified representation used, and does the agreement persist on native 32×3232\times32 RGB inputs? A comparison separating color removal from downsampling would clarify this issue. Section 6 also notes that the score-bias correction is small on the processed image data, limiting their evidence for this mechanism. Larger-scale datasets such as FFHQ or ImageNet would strengthen applicability.

  2. Accessibility of the introduction. The introduction explains the connection between supervised spectral bias and denoising score matching, but introduces technical terms before providing enough intuition for their role. In particular, the abstract and line 052 refer to the "ensemble mean" without explaining which sources of randomness define the ensemble. Section 3 subsequently specifies an average of posterior-mean predictors over clean datasets. Please introduce that distinction earlier and explain why this average is useful for understanding the learned score.

Questions for authors

Major questions

Please check the weaknessed above.

Minor presentation comments

  • Please expand neural network Gaussian process (NNGP) and neural tangent kernel (NTK) at first mention, in lines 014 and 033 respectively.
  • Please correct s=−fβs=-f\sqrt{\beta} at line 202 to match sθ=−fθ/βs_\theta=-f_\theta/\sqrt{\beta} at line 127.
  • Figure 1's caption should refer to Figure 4 for the d=16d=16 experiment, rather than Figure 6.
  • Please revise the opening sentence of Appendix D for grammatical clarity.
1 comment

Response to Reviewer mNdZ

1. Scope of the image validation and CIFAR-10 preprocessing. use downsampled MNIST and 16×1616 \times 16 grayscale CIFAR-10, with only P=256P = 256 images per dataset draw. For CIFAR-10, removing color and downsampling reduces the input dimension from 3072 to 256 and alters the data geometry and kernel spectrum central to the analysis. Why is this simplified representation used, and does the agreement persist on native 32×3232 \times 32 RGB inputs? A comparison separating color removal from downsampling would clarify this issue. Section 6 also notes that the score-bias correction is small on the processed image data, limiting their evidence for this mechanism. Larger-scale datasets such as FFHQ or ImageNet would strengthen applicability.

Authors: We largely agree with the referee. The limiting factor was that both doing Kernel Regression and training a network in the overparameterized regime are computationally prohibitive here, since the number of data points is PMPM where M=O(100)M = O(100) is the number of noise points. Such experiments are nonetheless needed to provide a clean comparison with theory. This is the main reason for using this simplification.

To circumvent this, the revised manuscript now contains experiments which verify spectral bias behavior on realistic datasets. Specifically, we used a pre-trained CNN as the ground truth for the score function, then trained FCNs with P=1000,2500,5000,10000P=1000,2500,5000,10000 data points, with ADAM, and spectrally filtered their results on their kernels (at initialization). Clear agreement with theory is evident. This is now presented in Figure 4b.

The revised introduction also clarifies the circumstances under which spectral bias is meaningfully augmented by score bias and position-dependent ridge. On this note, we would like to emphasize an aspect of the theory that was underemphasized, which is that even the spectral-bias results are not verbatim those of standard spectral-bias theory, as they contain an additional gain term proportional to the effective ridge over the bare ridge. This is also clarified in the revised introduction section. The effect of this gain is also shown numerically in Figure 1 and Figure 4a.

2. Accessibility of the introduction. The introduction explains the connection between supervised spectral bias and denoising score matching, but introduces technical terms before providing enough intuition for their role. In particular, the abstract and line 052 refer to the ``ensemble mean'' without explaining which sources of randomness define the ensemble. Section 3 subsequently specifies an average of posterior-mean predictors over clean datasets. Please introduce that distinction earlier and explain why this average is useful for understanding the learned score.

Authors: We apologise for any unclarity in our introduction. We sincerely hope that the referee can re-examine the revised version.

Two sources of randomness define the ensemble: the network weights, which Langevin training samples from the posterior, and the draw of the PP clean training points. The noise draws are averaged in the loss. For a fixed training set, averaging over the weights gives the posterior-mean predictor of Eq. (3), and the ensemble mean is its average over training sets, as the reviewer describes. We now state this in the introduction before the contributions. We also explain why this average is the object of interest: it is the part of the learned score shared by all training sets of size PP, so its deviation from the population score isolates the systematic effect of finite data and of the kernel from sample-specific fluctuations. It is also the quantity that the supervised spectral-bias theory characterizes (Canatar et al., 2021), which makes the comparison direct.

Minor presentation comments:

  • Please expand neural network Gaussian process (NNGP) and neural tangent kernel (NTK) at first mention, in lines 014 and 033 respectively.
  • Please correct s=−fβs = -f\sqrt{\beta} at line 202 to match sθ=−fθ/βs_\theta = -f_\theta/\sqrt{\beta} at line 127.
  • Figure 1's caption should refer to Figure 4 for the d=16d = 16 experiment, rather than Figure 6.
  • Please revise the opening sentence of Appendix D for grammatical clarity.

Authors: We thank the referee for their presentation comments; we have accepted them all and made the corrections in lines 014, 033, 202, Figure 1's caption, and the opening sentence of Appendix D.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.