- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the theory for approximating score functions with deep neural networks. Unlike prior works that assume smooth data distributions, it proves that neural networks can approximate the score functions of arbitrary compactly supported, non-smooth distributions. It further shows that the network size is governed by the intrinsic (Minkowski) dimension, rather than the ambient dimension, and introduces a new network architecture to achieve this rate. A key innovation in the proof technique is the "EA" regularity, which is a relaxation of Ahlfors regularity and proven by the authors to hold for any compact distributions.
Critical strengths
Understanding whether neural networks can approximate score functions of non-smooth distributions and how the network complexity depends on the dimension and smoothness is an important and interesting question in learning theory for generative models.
Critical weaknesses
- The paper, to me, is relatively hard to follow. It may be easy to read for someone familiar with related works such as Zhang et al. (2026), but for a general machine learning audience, the paper can be hard to read. Below are specific questions and suggestions.
- More background on Ahlfors regularity, its relation to low-dimensional structures and connection to Minkwoski dimension will be appreciated.
- It's unclear what the order M in Theorem 1 and the "M-th order complexity bound" on Line 192 mean. They don't appear to be standard terms in approximation theory. Furthermore, the authors may want to clarify its relation with the M in "(M+1)-Holder smoothness". Do these two M's represent the same quantity?
- The error term from previous works (line 181) depends on t, but the derived network complexity (line 184) and the result in Theorem 1 depends on t0. Can the authors explain why they simplified the dependence to t0?
- Can the authors clarify whether on line 178 denotes continuous derivatives through order or -Holder smoothness? If it refers to Holder smoothness, what's the relation between and M, which is also used to define Holder smoothness?
- There are several grammatical errors. For example, "Using a discrete-mixture framework, our network avoids the exponential curse of ambient dimensionality, its complexity governed by the support’s Minkowski dimension."
Questions for authors
Besides the questions raised in the Weakness section, can the authors comment on what information does one need to construct the network architecture introduced in the paper? Do we need to calculate the number of covers from a predefiend approximation error? How can this be achieved? Is the value of Minkwoski dimension needed? Ultimately, is this network mainly a theoretical construction or a potential alternative to the networks currently used in practice?
2 comments
Response to Questions
Q1(1): Theoretical existence result, or practical alternative?
Our construction is primarily a theoretical existence result, not a new engineered architecture meant to replace U-Nets or Diffusion Transformers. The network in Theorem 4 is a standard fully connected ReLU MLP; Figure 1 simply illustrates how its weights are organized inside the proof.
In other words, our theorems prove that standard ReLU MLPs of sufficient depth and width naturally contain our construction in their hypothesis class, establishing expressivity rather than gradient-descent learnability or justification for alternative network designs.
Q1(2): Are or needed in advance?
No, one only needs to run a covering algorithm on all samples (though knowledge of the manifold dimension is needed if one wishes to invoke the tighter manifold-case bound rather than the general -dependent bound). While governs the asymptotic scaling as and , it is not required to instantiate the network for fixed . One can verify directly from Appendix B.6 (lines 1306--1323) and Theorem 1 that and enter our complexity bound through only two quantities:
- The total number of sub-distributions , which is directly proportional to the number of Taylor modules;
- The maximum possible number of far-group balls , upper-bounded by , which controls the far-group contribution and produces the term inside (whereas all remaining terms in depend only on ).
For fixed and , the definitions of (lines 646--647, 674--676) together with (lines 649--652) guarantee that in the general case (and in the manifold case). Consequently, can be upper-bounded directly by the number of balls of radius (or ) needed to cover (or the observed samples in practice), replacing in with . From , follows directly via Theorem 1, and is simply the number of radius- balls a standard clustering or greedy -net algorithm uses to cover the samples.
Q1(3): What information the construction needs
Replicating our construction requires (1) The far-group cover count at radius (or ), from which and the covering radius follow via Theorem 1 using (and in the manifold case); (2) an -ball cover of the support/samples; and (3) each ball's empirical probability mass and local moments up to order . All cover counts and local moments can be estimated directly from finite samples without knowing or .
Response to Weaknesses
Thank you for recognizing the importance of our research question, and for your detailed suggestions on improving accessibility for a broader ML audience. We clarify all ambiguities below.
W1(1): Background on Ahlfors regularity and its relationship to Minkowski dimension
We will add this concise background and the EA-vs-Ahlfors comparison to Section 3.
W1(2) & W1(4): Meaning of order M, Cβ smoothness, and how M and β relate
Thank you for pointing out this ambiguity; these two questions are closely linked.
We will change "Cβ" to "β-Hölder smooth", define "M-th order complexity bound" explicitly before Section 2.1/Section 3 and clarify this comparison in the Abstract and Section 2.1.
W1(3): Why complexity depends on t0 rather than t
One neural network must satisfy the approximation error tolerance uniformly for every t∈[t0,T], and both Zhang et al.'s bound (line 181: MSE O(n2γ+d+2δ2/t)) and our bounds are worst at the smallest noise level t=t0. Setting the worst-case MSE at t=t0 to ϵ2 in Zhang et al. yields δ of order ϵt0⋅n−(γ+1+d/2), which, upon substituting into O(δ−d/β) with 2γ+2=4 from Footnote 2, gives the t0−d/(2β) dependence in line 184. Similarly, our Theorem 1 sizes the network for the bottleneck time t0.
We will insert this explicit 2-line substitution directly after line 181.
W1(5): Grammatical errors
Thank you for catching this. In our revision, we will correct your example sentence to: "Using a discrete-mixture framework, our network avoids the exponential curse of ambient dimensionality, with its complexity governed by the support's Minkowski dimension." We have also audited the manuscript and corrected additional issues (e.g., line 23 "each at size" → "each of size"; line 93 "weighed" → "weighted"; line 147 "analysis ... focus" → "focuses"; line 298 "mass ... dominate" → "dominates"; line 452 "bounds ... predicts" → "predict"; line 461 "network ... require" → "requires"; line 488 "Deepseek" → "DeepSeek").