- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Accurate score estimation is central to the reverse process in diffusion models. This paper studies how network inductive bias and finite training data learned scores in the infinite-width kernel regime. At a fixed small noise level, the authors connect denoising score matching to supervised kernel regression and derive expressions for the ensemble mean and variance of the denoiser. The mean score is represented as a kernel operator applied to the score of an uncertainty-reweighted density. Experiments on synthetic distributions and image data provide empirical support for these predictions.
Critical strengths
The paper provides a useful connection between supervised spectral bias and denoising score matching. In particular, it identifies how spatially varying posterior uncertainty changes the effective regularization and introduces an additional score correction. These predictions offer a detailed description of inductive bias.
Critical weaknesses
-
The presentation makes the probabilistic setup and approximation regime difficult to follow. For example, in Section 3, the authors should define the parameter prior and the kernel more explicitly, and explain clearly the posterior mean in Eq. (3). The notation also overloads as both a measure and a sample index.
-
The relationship to statistical analyses of score estimation could be clarified. Existing work, including "Diffusion Models are Minimax Optimal Distribution Estimators" (Oko et al., 2023) and "Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions" (Azangulov et al., 2025), studies finite-sample guarantees for score estimators and generated distributions under specified function classes and geometric assumptions. The present paper instead characterizes pointwise bias and uncertainty in an equilibrium kernel model. What conclusions about learned scores can this characterization provide that are not captured by existing estimation-rate results, and how do the assumptions compare?
Questions for authors
-
The experiments use very wide networks. Could the authors test, in one existing setting, whether the predicted score distortion remains qualitatively visible for a finite-width network (used in practice) trained with a standard optimizer at a fixed small noise level? If this lies outside the intended scope, please clarify which aspects of the theory are expected to transfer to practical diffusion training.
-
Same as Weaknesses 2.
2 comments
part 2/2
Question 1. The experiments use very wide networks. Could the authors test, in one existing setting, whether the predicted score distortion remains qualitatively visible for a finite-width network (used in practice) trained with a standard optimizer at a fixed small noise level? If this lies outside the intended scope, please clarify which aspects of the theory are expected to transfer to practical diffusion training.
Authors: We thank the referee for their questions, which motivated us to perform more extensive experiments. These now validate our spectral-bias theory for FCNs on CIFAR10 with training points.
We derive the spectrum analytically from the untrained networks' NNGP kernels, and we estimate the ground truth score from the data (See the newly added Fig. 4 in particular panel b).
Despite using finite-width networks and ADAM optimizers, rather than our infinite-width, Langevin-based theory, the results clearly show spectral filtering of the normalized learned score, as our theory predicts.
We further found that the score is overestimated due to the aforementioned gain term; we now emphasize this gain (Eq (7)). This gain factor can be computed analytically, given the NNGP kernel at initialization. Interestingly, dividing the trained neural network by this gain factor improves the test/held-out DSM score in these more realistic settings (see Figure 4a).
Notably, a positive gain factor in the score is equivalent to raising the implied probability by a positive power. Such a distortion increases the score in the regions where the true data distribution is higher and reduces diversity. Our theory offers a potential-theory-based route to analyzing and countering these known issues.
part 1/2
Response to Reviewer WBeQ
We thank the reviewer for their thoughtful comments, which helped us improve the paper's clarity. In addition, their question about existing estimation-rate results helped us clarify what our pointwise analysis adds. It predicts where the learned score is systematically distorted and in which direction.
Weakness 1. The presentation makes the probabilistic setup and approximation regime difficult to follow. For example, in Section 3, the authors should define the parameter prior and the kernel K more explicitly, and explain clearly the posterior mean in Eq. (3). The notation also overloads μ as both a measure and a sample index.
Authors: We apologise for any unclarity in our introduction. We sincerely hope that the referee can re-examine the revised version.
Two sources of randomness define the ensemble: the network weights, which Langevin training samples from the posterior, and the draw of the P clean training points. The noise draws are averaged in the loss. For a fixed training set, averaging over the weights gives the posterior-mean predictor of Eq. (3), and the ensemble mean is its average over training sets, as the reviewer describes. We now state this in the introduction before the contributions. We also explain why this average is the object of interest: it is the part of the learned score shared by all training sets of size P, so its deviation from the population score isolates the systematic effect of finite data and of the kernel from sample-specific fluctuations. It is also the quantity that the supervised spectral-bias theory characterizes (Canatar et al., 2021), which makes the comparison direct.
Minor presentation comments:
Weakness 2 & Question 2. The relationship to statistical analyses of score estimation could be clarified. Existing work, including ''Diffusion Models are Minimax Optimal Distribution Estimators'' (Oko et al., 2023) and ''Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions'' (Azangulov et al., 2025), studies finite-sample guarantees for score estimators and generated distributions under specified function classes and geometric assumptions. The present paper instead characterizes pointwise bias and uncertainty in an equilibrium kernel model. What conclusions about learned scores can this characterization provide that are not captured by existing estimation-rate results, and how do the assumptions compare?
Authors: We thank the reviewer for pointing out these references, which we now include. As argued in the revised introduction and related works section, a main novel element of the current work is a directly derived connection with spectral bias theory, which contains a novel element: A gain factor. Some works assume spectral smoothing rather than derive it (Gabriel et al., 2025; Farghly et al., 2025; Chen, 2026), others derive it for linear denoisers for wider β and κ ranges but do not flesh out the gain factor (Wang, Zavatone-Veth & Pehlevan, 2026, which we now cite) and its implications.
Finally, for distributions having regions with low intrinsic dimension, we further flesh out a combined effect related to having a position-dependent ridge
As the revised manuscript clarifies these novel aspects, and further shows experiments on larger real datasets where both the gain factor and spectral bias are cleanly seen. (gain in Fig. 4a and Fig. 1; spectral bias in Fig. 4b)
We also note that the assumptions and setup in these papers are different. Oko et al. analyze empirical score matching over prescribed network classes under Besov smoothness assumptions. Azangulov et al. assume a smooth distribution supported on a smooth low-dimensional manifold and construct estimators adapted to that geometry. Our setup assumes a smooth, strictly positive density in the ambient space, a specified infinite-width kernel, and Langevin equilibrium at one small noise level. The pointwise prediction uses the Gaussian-equivalence and mean-field approximations described in Section 5.