Spectral Bias and Finite Sample Effects in Denoising Score Matching

Score-based diffusion models achieve remarkable performance on high-dimensional tasks, yet the inductive biases that shape their learned scores remain poorly understood. Their training can be viewed as a sequence of denoising regression problems. In the infinite-width limit, a network is described by its NNGP kernel, whose spectrum induces a well-understood spectral bias in supervised regression. In this work, we show that this spectral bias is largely accurate in high-dimensional denoising score matching, and wide networks essentially act as supervised learning on the score. At small diffusion time, we derive two correction terms, one involving a position-dependent ridge and another involving a pointwise bias to the ensemble mean score. These corrections become significant when data have low intrinsic dimension but are embedded in a high-dimensional space with many weak-variance directions. We validate these predictions using finite-width networks on seven synthetic distributions, MNIST, and CIFAR-10, finding close agreement.