acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Wasserstein Rates for Score-Based Diffusion Models on Smooth Manifolds

14 views

Abstract

Many high-dimensional data distributions are supported on low-dimensional manifolds, whereas diffusion models learn score functions in the ambient space. This raises a basic question: can empirical risk minimization over ambient score-network classes attain a rate governed by intrinsic dimension without first estimating a manifold representation from the observations? We study this question for distributions with strictly positive -H\"older densities on compact -dimensional smooth manifolds embedded in . The main technical challenge is the low-noise regime, where the score becomes singular as the noise level vanishes. To address this challenge, we construct tangent-cell ReLU approximations at moderate and high noise and a projection-centered ReLU approximation at low noise, with explicit ambient-dimension dependence. For , under the stated geometry, regularity, and optimization conditions, the cube-projected output of the continuous-time sampler has expected -Wasserstein error up to logarithmic factors. The result therefore establishes the intrinsic sample-size exponent convergence rate for ambient score networks, without a preliminary data-dependent dimension-reduction step.

Primary area:
learning theory
Keywords:
Diffusion models, manifold hypothesis, statsitical rates

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tRFkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies the sample complexity of score-based diffusion models for learning distributions supported on low-dimensional smooth manifolds. Under certain regularity conditions, they show that the generated distribution has expected 1-Wasserstein error O~(DOβ(d)n−(β+1)/(d+2β)),\widetilde{O}\big(D^{O_\beta(d)}n^{-(\beta+1)/(d+2\beta)}\big), where DD denotes the ambient dimension, d>2d>2 is the intrinsic dimension, nn is the sample size, and β\beta is the H"older smoothness parameter of the density.

Critical strengths

Their overall proof strategy and theoretical analysis appear sound. Specifically, the separation of score approximation error and statistical estimation error provides a clear framework for controlling the final sampling error. Furthermore, they use different constructions for comparison scores under different noise levels. These constructions are well motivated and yield convergence rates governed by the intrinsic dimension.

Critical weaknesses
  1. The bound contains a factor DOβ(d)D^{O_\beta(d)}, which may become substantial even when the intrinsic dimension is moderate. In comparison, "Convergence of Diffusion Models Under the Manifold Hypothesis in High-Dimensions" (Azangulov et al. 2025) obtains a Wasserstein bound with only D\sqrt{D} dependence, while "Diffusion Models Are Statistically Optimal for Learning Low-Dimensional Multi-Modal Distributions" (Wu & Cai, 2026) obtains linear dependence on DD under a union-of-subspaces model. Although these results use different assumptions and estimators, the difference deserves further explanation. Can you identify which steps lead to the larger ambient dependence in your analysis and explain whether this dependence is necessary or could be improved?

  2. Novelty compared with existing literature. "Adaptivity of Diffusion Models of Manifold Structures" (Tang & Yang, 2024) and "Convergence of Diffusion Models under the manifold hypothesis in high-dimensions" (Azangulov et al., 2025) already establish intrinsic-dimensional Wasserstein rates without explicitly estimating the manifold. Although the authors discuss these connections, the additional contribution could be made clearer. The main advance appears to be the finite-anchor Gauss–Newton construction and its explicit control of ambient-dimension dependence. Can you explain more precisely what new approximation or statistical guarantee this construction provides beyond the existing results, particularly since the sample-size exponent remains unchanged?

Questions for authors
  1. The paper obtains the sample-size exponent (β+1)/(d+2β)(\beta+1)/(d+2\beta), which matches the known benchmark for smooth-density estimation. However, comparisons between upper bounds under different assumptions do not directly establish minimax optimality for the distribution class considered here. Can you provide or cite a matching lower bound under your manifold and H"older-density assumptions, and clarify which geometry and density parameters are held fixed?

  2. Please address Weaknesses 1 and 2.

  3. Can the assumption that the density is bounded away from zero on the manifold be relaxed? Smooth densities on compact smooth manifolds may vanish at some points, so this assumption excludes otherwise regular distributions. Since the lower bound is used to control the denominators in the score approximation, what are the main obstacles to removing it? Would intrinsic-dimensional convergence guarantees still hold, possibly with a different rate?

4 comments
AuthorsOpenReview ↗

Response to reviewer tRFk (Part 1)

Thank you for the positive assessment and the detailed technical questions. In response, we added Table 2, which shows where each power of DD comes from, a lower bound (Proposition J.1) and an example on positivity (Appendix J.3). Substantive changes are marked in blue in the updated manuscript. Below, εn=n−(β+1)/(2β+d)\varepsilon_n=n^{-(\beta+1)/(2\beta+d)} is the target rate, pβ,d=max⁡⌈6β⌉+8,⌈5d/4⌉+13\mathfrak p_{\beta,d}=\max\\{\lceil6\beta\rceil+8,\lceil5d/4\rceil+13\\} is the power of DD in Theorem 2.2, qq is the chart smoothness order of Eq. (4), and GG is a polynomial in the geometry and density bounds.

W1: Which steps lead to the larger ambient dependence, and is it necessary or can it be improved?

The dependence comes from two steps of our upper bound. A linear dependence of the network size on DD is necessary. We do not think that the power pβ,d\mathfrak p_{\beta,d} is necessary, and we point to one place where it can be improved.

(a) Which steps. The noise has norm about σtD\sigma_t\sqrt D, so every approximation must hold on a tube of radius σtD∨log⁡n\sigma_t\sqrt{D\vee\log n} around the manifold, and all bounds are stated in D∨log⁡nD\vee\log n. The two entries of pβ,d\mathfrak p_{\beta,d} have different sources (Table 2).

  • ⌈6β⌉+8\lceil6\beta\rceil+8 comes from the approximation error of the low-noise expansion on this tube (Lemma F.28).
  • ⌈5d/4⌉+13\lceil5d/4\rceil+13 comes from the estimation error. The score classes have width and sparsity GNk(D∨log⁡n)5d/2+10GN_k(D\vee\log n)^{5d/2+10} (Eq. (9)), a size set by the tangent-cell networks at moderate and high noise, and the entropy multiplies this size by the depth and by the logarithm of the weights.

The step to W1W_1 then takes a square root and adds one more factor D∨log⁡nD\vee\log n.

(b) What is necessary. An ambient network reads DD coordinates and outputs a score in RD\mathbb R^D, so it needs at least order DD nonzero weights (see the example in Appendix A.1), and the entropy bound of the network class grows with DD. The early-stopped sampler also has bias Ω(D,εn)\Omega(\sqrt D\\,\varepsilon_n) on manifolds padded with order DD zero coordinates, even with the exact score (Appendix J.2).

(c) What can be improved. The size of the power comes from worst-case control on the tube. For example, the low-noise expansion contains scalar products of the normal residual with chart-derivative vectors. We bound each of them by the Cauchy–Schwarz inequality (proof of Lemma F.27), which costs a factor D∨log⁡n\sqrt{D\vee\log n}, whereas the projection of Gaussian noise onto a fixed direction has size of order one (a heuristic, since these directions depend on the projection point). The proof of Lemma F.28 also notes that its bound on the low-noise integral errors is conservative. We have not carried out the sharper analysis.

(d) The rates you cite. Their estimators recover the geometry first, so the DD-dimensional linear maps are computed from the data and are not learned by a network.

  • Azangulov et al. (2024, arXiv v2, Corollary 6(ii)) use a structured estimator: anchor points chosen from the training data and a local linear projection at each anchor, both fixed before training. Standard training has no such step. They attribute their D\sqrt D factor to early stopping. Their arXiv v3 (Corollary 3.2) removes this factor with a consistency-preserving sampler and control of the denoiser tails.
  • Wu–Cai (2026, Theorem 2) obtain O~(DJ3/2n−1/(d∨2))\widetilde O(DJ^{3/2}n^{-1/(d\vee2)}) for JJ components on linear subspaces, using subspace recovery and a kernel score. Their model has no curvature and no gain from smoothness.

We learn the score directly in the ambient space, and Theorem 2.2 shows that the price is at most polynomial in DD.

AuthorsOpenReview ↗

Response to reviewer tRFk (Part 2)

(Continued; notation as in part 1.)

W2: What new approximation or statistical guarantee does the construction give beyond existing results?

As you note, the main advance is the finite-anchor Gauss–Newton construction with explicit control of the ambient dimension. It gives two approximation results and one statistical guarantee that the cited works do not have.

(a) Approximation guarantees.

(i) Projection network (Theorem F.13). It approximates the nearest-point projection to any accuracy εΠ\varepsilon_\Pi with width and sparsity GCD1+d/qεΠ−2d/qG^CD^{1+d/q}\varepsilon_\Pi^{-2d/q} up to logarithms. This count includes all costs of the construction, and the number of anchors does not depend on εΠ\varepsilon_\Pi. In comparison:

  • The Newton networks of Tang–Yang (2024, Lemma C.8) do not track these costs.
  • The construction of Cloninger–Klock (2021, Corollary 5) costs O~(DεΠ−d)\widetilde O(D\varepsilon_\Pi^{-d}).
  • The projection network of Zhang et al. (2026, Lemma D.18) approximates a function of DD variables with smoothness order at least βD/d\beta D/d, and its constant may grow exponentially in DD.

This matters because the projection error enters the low-noise score as mtεΠ/σt2m_t\varepsilon_\Pi/\sigma_t^2, so the accuracy has to increase as t→0t\to0. The exponent 2d/q2d/q keeps the score networks within the size bounds of Eqs. (8)–(9).

(ii) Low-noise score network (Lemmas F.27–F.28). The normal residual enters only through finitely many scalar products, so the coefficient functions to approximate depend only on the dd chart variables and on time. Tang–Yang (2024) expand in polynomials of the DD ambient variables instead.

(b) Statistical guarantee. The guarantee, a W1W_1 bound for ambient ERM with a fully tracked polynomial dependence on DD, is new; the argument behind it is standard. It is the interval-wise estimation of Oko et al. (2023) and Tang–Yang (2024), carried out here with explicit ambient costs for classes of intrinsic size that get smaller as the noise grows (Eq. (8)). In comparison, the bound of Tang–Yang (2024) hides constants that may grow like eDe^D, as Yakovlev–Puchkin (2025, Section 3.1) point out, and the derivation of Zhang et al. (2026) uses only a global bound on the score risk, which does not give the smaller risks that the transfer needs on later intervals. The W1W_1 rate of Azangulov et al. (2024, arXiv v3, Corollary 3.2) is for their structured estimator (see W1 (d)). For their ambient class they bound the score risk linearly in DD (Theorem 3.3) and state no W1W_1 rate, and their stated model size (number of nonzero weights) has no term in DD, although the input and output layers are DD-dimensional (Appendix A.1).

Q1: Is there a matching lower bound for this class, and which parameters are held fixed?

Yes. We have added a matching lower bound, Proposition J.1.

(a) Statement. Over normalized densities in Hβ(M,B0)\mathcal H^\beta(\mathcal M,B_0) bounded below by pmin⁡p_{\min}, we have inf⁡P^nsup⁡pEpW1(Pp,P^n)≥cn−(β+1)/(2β+d)\inf_{\widehat P_n}\sup_p\mathbb E_pW_1(P_p,\widehat P_n)\ge cn^{-(\beta+1)/(2\beta+d)}, even if M\mathcal M is known. The proof uses Assouad's method (cf. Niles-Weed and Berthet (2022); Divol (2022)). So our sample exponent is minimax up to logarithms.

(b) What is held fixed. We fix DD, d>2d>2, β>0\beta>0, a compact smooth dd-manifold M⊂[0,1]D\mathcal M\subset[0,1]^D without boundary together with its density atlas, and B0≥1B_0\ge1, pmin⁡∈(0,1]p_{\min}\in(0,1] with pmin⁡<V−1<B0p_{\min}<V^{-1}<B_0, where V=vol⁡(M)V=\operatorname{vol}(\mathcal M). The constant cc depends only on these quantities.

Q3: Can the lower bound on the density be relaxed? What are the obstacles, and would intrinsic rates still hold?

This is a natural question. The present argument does not allow it, and we have added Appendix J.3 to explain why.

(a) Obstacle. Together with the reach condition, positivity gives the lower bound Qˉ≥cΓ−Cpmin⁡\bar Q\ge c\Gamma^{-C}p_{\min} for the normalized low-noise denominator, and it gives local mass bounds at moderate noise. Smoothing keeps pt>0p_t>0, but the lower bound need not be polynomial. For a smooth density with a flat zero, the denominator can be exponentially small in 1/σt1/\sigma_t.

(b) Rates without positivity. Chakraborty, Berthet and Bartlett (2026) obtain rates close to n−1/dn^{-1/d}, with no gain from smoothness. Keeping both the smoothness gain and intrinsic network sizes without positivity is open, and we see it as a natural next step.

Thank you again for the careful technical reading.

I thank the authors for the detailed response, which resolves most of my concerns. I agree that some dependence on DD is unavoidable and that the remaining factors are hard to improve for the general function class considered. The explicit treatment of the approximation error and the statistical guarantee clarifies the novelty, and the matching lower bound in Proposition J.1 complements the results well. I also agree that removing the positivity assumption while keeping both the smoothness gain and the intrinsic dimension is difficult. I have raised my scores for soundness and will maintain my overall positive rating.

Thank you for the careful follow-up and continued positive assessment. We appreciate your constructive feedback, which helped us clarify the contribution and strengthen the paper.

Reviewer_71SROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the sampling error for score-based diffusion models when the data distribution is supported on a smooth manifold. The assumptions are that the data density is β\beta-Holder smooth, the score is learned from a training dataset of size n, and the sampler solves the reverse-time diffusion process exactly without discretization error. Given these, the paper shows that the Wasserstein-1 error is proportional to ncn^c, where the exponent c depends only on the intrinsic dimention of the manifold, not the ambient dimension. This establishes that the sample size for learning diffusion models mainly depends on the intrinsic dimension when data is low-dimensional.

Critical strengths
  • The convergence of diffusion models for low-dimensional data is an important and interesting problem.
  • The main theoretical result is presented clearly in general.
Critical weaknesses
  • My major question is on the comparison with prior works, in particular, with Tang & Yang (2024). The discussion on Lines 77-82 states that prior works achieve "the same intrinsic exponent" as this paper. It's quite confusing, since it suggests that the main contribution of this paper for proving a "bound with the intrinsic sample-size exponent" is not new. Can the authors provide more explanation on the difference from prior works and state the contribution of this paper more precisely? How does this work improve on previous results?
Questions for authors

In addition to the question in Weaknesses, below are additional comments.

  • The presentation of technical results is a bit hard to follow. For example, In theorem 2.2 and Prop. 5.1, equations from the appendix, which have not been introduced beforehand, are referenced. This could hurt readability.
  • Can the authors explain why they use OβO_\beta to hide the dependence on β\beta? Can this dependence be written explicitly?
  • Can the authors explain why a projection step is necessary for the sampler? Without this step, will the Wasserstein distance be ill-defined? How will the guarantees change if removing this step?
3 comments
AuthorsOpenReview ↗

Response to reviewer 71SR

Thank you for the supportive review and the clear questions. Your comments led us to sharpen the comparison with prior work (Table 1) and to make Theorem 2.2 self-contained. Substantive changes are marked in blue in the updated manuscript. Below, εn=n−(β+1)/(2β+d)\varepsilon_n=n^{-(\beta+1)/(2\beta+d)} is the target rate, qq is the chart smoothness order of Eq. (4), and GG is a polynomial in the geometry and density bounds.

W1: Earlier work has the same sample exponent. What exactly is the contribution, and how does it improve on earlier results?

You are right that the sample exponent is not new, and we have revised the introduction to say so explicitly. The contribution is a W1W_1 guarantee for ambient ERM that remains valid when the ambient dimension DD is large, and the new approximation constructions that make it possible.

(a) Improvement in the result. The bound of Tang–Yang (2024) does not track DD: at the parameter choice of their main theorem it contains a binomial factor, described there as (log⁡n)D(\log n)^D, and its hidden constants may grow like eDe^D (Yakovlev–Puchkin (2025, Section 3.1)). The projection network of Zhang et al. (2026) has a constant that may grow exponentially in DD (Appendix A.2). Our bound is GC(D∨log⁡n)pβ,dεnG^C(D\vee\log n)^{\mathfrak p_{\beta,d}}\varepsilon_n with an explicit power pβ,d\mathfrak p_{\beta,d} (given in Q2 below), and it still goes to zero when DD grows like a small power of nn (Corollary H.4).

(b) New approximation results. Two results are not in these works.

  • A projection network (Theorem F.13). It approximates the nearest-point projection to accuracy εΠ\varepsilon_\Pi with width and sparsity GCD1+d/qεΠ−2d/qG^CD^{1+d/q}\varepsilon_\Pi^{-2d/q} up to logarithms, using the smoothness order qq of the charts, compared with O~(DεΠ−d)\widetilde O(D\varepsilon_\Pi^{-d}) for Cloninger–Klock (2021, Corollary 5). This gives the accuracy needed as the noise vanishes, while the score network stays at the intrinsic size nd/(2β+d)n^{d/(2\beta+d)}.
  • A low-noise score network in which the number of scalar coefficient functions to approximate does not depend on DD (Lemmas F.27–F.28). These functions depend only on the dd chart variables and on time, which is how an ambient network can adapt to the manifold.

With these networks, the standard interval-wise estimation argument gives score estimation bounds on every noise interval.

Q1: The main results refer to equations in the appendix that have not been introduced, which hurts readability.

Thank you for pointing this out. We have revised the paper accordingly.

  • Theorem 2.2 and Propositions 3.1 and 5.1 now refer only to displays in the main text.
  • Section 2.3 defines the parameters before the theorem (Eqs. (4)–(5)), and Section 4.1 defines the score classes.
  • The appendix states its results under more detailed conditions, and Appendix H shows that Eq. (6) implies them.

Q2: Why does OβO_\beta hide the dependence on β\beta, and can it be written explicitly?

  • (a) Why OβO_\beta. The exponent in the submission contained a constant that we had not computed.

  • (b) Explicit form. Yes. The exponent is now pβ,d=max⁡⌈6β⌉+8,⌈5d/4⌉+13\mathfrak p_{\beta,d}=\max\\{\lceil6\beta\rceil+8,\lceil5d/4\rceil+13\\} (Theorem 2.2), and the main text no longer uses OβO_\beta. So the exponent depends on β\beta and dd explicitly. Geometry and density enter through powers of G=CG(ΓB0pmin⁡−1)aGG=C_G(\Gamma B_0p_{\min}^{-1})^{a_G} (Eq. (4)), where Γ\Gamma bounds the geometry of the manifold, B0B_0 bounds the Hölder norm of the density and pmin⁡p_{\min} is the lower bound of the density. These powers depend only on β\beta, dd and the constant a0a_0 of Theorem 2.2.

Q3: Why is the projection step needed? Is W1W_1 ill-defined without it, and how do the guarantees change if it is removed?

  • (a) Why it is needed. The step clips each coordinate of the output to the cube C=[0,1]DC=[0,1]^D. It is not a projection onto the manifold. Since P0P_0 lies in CC and clipping is 1-Lipschitz, the step cannot increase the error, and the proof uses it to control the transport cost on rare events.

  • (b) Is W1W_1 ill-defined without it? No. W1W_1 is finite without the step, because the drift of the sampler is linear plus a bounded term.

  • (c) Guarantees without it. We have added a paragraph on this to Appendix H. The raw output YY has the same rate exactly when its expected distance to the cube, Edist⁡(Y,C)\mathbb E\operatorname{dist}(Y,C), does. We have not bounded this distance, so the guarantee is stated for the projected output.

Our Corollary H.11 also gives a finite-step version with the same rate up to logarithms, for scores trained on a grid-weighted DSM objective.

Thank you for the comments, which helped us clarify both the comparison and the main statement.

Thank the authors for answering my questions and clarifying the main contribution. I will keep my original score. I would encourage the authors to state their contribution, making the dependence on DD explicit, and its significance clearly in the abstract.

Thank you for your follow-up and continued positive assessment. We appreciate your suggestion and will revise the abstract accordingly.

Reviewer_AK59OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies statistical rates for diffusion models in the setting of density estimation (similar to Oko et al. 2023). The rates are in W1W_1 under the hypothesis that the data lies in a low dimensional manifold and that the density are β−\beta-Holder. The way this differs from the work of Azangulov et al. in that they attempt to get results that are adaptive to the geometry by plain ERM, whereas Azangulov et al. have a manifold learning preconditioning which allows them to get good rates. Using ambient score matching with sparse ReLU networks and early stopping, they obtain the rate Ddn−β+1d+2βD^d n^{-\frac{\beta+1}{d+2\beta}} up to log factors, without estimating the manifold. The same setting is pursued in Tang & Yang (2024) but the dependence on DD is non-explicit (potentially exponential?) so this is a significant improvement when d≪Dd \ll D. The key technical step is a low-noise approximation of the score via a Laplace expansion centered at the nearest-point projection, implemented by Gauss–Newton networks initialized from finitely many anchors in intrinsic coordinates.

Critical strengths
  • Analysis is end-to-end in that it includes the error of the sampler. Furthermore the rates presented are explicit (geometry and density constants are tracked rather than hidden)
  • The paper studies the low-noise regime in a technical original way. The score is centered at the nearest-point projection, implemented by finite-anchor Gauss--Newton networks in intrinsic coordinates, which makes the ambient-dimension cost explicit, unlike the related projection-centered analysis of Zhang et al. (2026)
  • The dependence on DD is explicit and the proofs seem complete and unambiguous
  • Honest scoping of contribution and fairly well presented related works section (see weakness 1)
Critical weaknesses
  • It is a bit tricky to trace the exact place where a contribution was made. In particular, it is clear that Tang & Yang (2024) and Zhang (2026) consider the same setting and get rates for the ERM without explicitly learning the manifold. The authors claim that the benefit of their work over Tang & Yang as well as Zhang is that their rates are explicit (i.e. that they hide some dependence on DD), but it is not clear if that is simply a matter of how they chose to present their work. I suggest that the authors provide a table with the rates of different works for easier comparison since the novelty of the paper rests on this discussion
  • A few experiments would have been useful, and the absence of a lower bound on the manifold makes it difficult to assess how tight the analysis is.
  • It is not clear if DdD^d is tight in any respect or captures a limitation of using the ERM instead of learning the geometry first?
  • It is currently quite difficult to understand which parts of the proof are genuinely novel, largely because of the length of the appendix (which contains and reproduces some known results). A self-contained formal statement of the main theorem in the main text, together with a clear separation of new and standard components, would make the paper much easier to verify and understand
Questions for authors
  • Is there a lower bound (in the minimax sense) that verifies how optimal your bound is?
  • Do the authors believe that the gap between the rates of Azangulov et al. and the ones here are fundamental or a limitation of their proof technique?
  • Would it be possible to present (even toy) experiments that demonstrate that the rate DdD^d is correct without a manifold learning step
  • Could this be used to understand architectural/methodological properties of diffusion models used in practice? E.g. Is the limitation of the ERM a result of the very generic architecture?
2 comments
AuthorsOpenReview ↗

Response to reviewer AK59 (Part 1)

Thank you for the positive review and the careful reading. We added the comparison table you suggested (Table 1) and made Theorem 2.2 self-contained. Substantive changes are marked in blue in the updated manuscript. Below, εn=n−(β+1)/(2β+d)\varepsilon_n=n^{-(\beta+1)/(2\beta+d)} is the target rate and pβ,d=max⁡⌈6β⌉+8,⌈5d/4⌉+13\mathfrak p_{\beta,d}=\max\\{\lceil6\beta\rceil+8,\lceil5d/4\rceil+13\\} is the power of DD in Theorem 2.2.

W1: Is the advantage over Tang–Yang (2024) and Zhang et al. (2026) only a matter of presentation?

We believe it is more than presentation. The difficulty is at low noise, where the projection onto the manifold must be approximated to an accuracy that increases as t→0t \to 0, at a cost that stays polynomial in DD. Table 1 lists the rates. In both papers, the dependence on DD enters through steps of their proof, (a) and (b), where it is not controlled and may be exponential. The W1W_1 derivation of Zhang et al. (2026) also has a proof gap, (c).

  • (a) The score approximation of Tang–Yang (2024) uses polynomials in the DD ambient variables. The number of monomials gives a binomial factor in their network sizes at intermediate and high noise, which they describe as (log⁡n)D(\log n)^D at the parameter choice of their main theorem (their Remark B.2). They note that a fixed choice of an auxiliary parameter makes this term polynomial in DD. The remaining constants are not tracked, and Yakovlev–Puchkin (2025, Section 3.1) point out that the coefficients of the expansion hide multinomial coefficients that may be of order eDe^D. In our low-noise expansion, the normal residual enters only through finitely many scalar products, so the number of scalar coefficient functions to approximate does not depend on DD (see Appendix A.3, Lemmas F.27–F.28).

  • (b) Zhang et al. (2026) approximate the projection as a function of all DD variables (their Lemma D.18), with a constant that may grow exponentially in DD. Our projection network approximates only functions of the dd chart variables, and its size is polynomial in DD (Theorem F.13).

  • (c) Zhang et al. (2026) pass from the score risk to W1W_1 with a bound on the total risk only. As we read it, this leaves a proof gap. The transfer result they cite (Lemma D.7 of Oko et al. (2023)) gives larger weights to later time intervals, so the target rate needs smaller risks there. If the total risk n−2β/(2β+d)n^{-2\beta/(2\beta+d)} sits on intervals with tt of order one, the transfer gives only n−β/(2β+d)n^{-\beta/(2\beta+d)} (Appendix A.2). This does not show that their rate is false. We prove the required bounds on every interval, using score classes that get smaller as the noise grows (Eq. (8), Section 5).

With these steps, our Theorem 2.2 gives a bound that is polynomial in DD.

W2 and Q1: Is there a minimax lower bound?

We have added a minimax lower bound, Proposition J.1. On a fixed manifold, every estimator has worst-case E,W1≥c,εn\mathbb E\\,W_1\ge c\\,\varepsilon_n over the positive β\beta-Hölder densities of our class, even if it knows the manifold. So our sample exponent is minimax up to logarithms. The lower bound says nothing about the dependence on DD, and we leave the optimal dependence on DD as future work.

W2 and Q3: Can toy experiments show the dependence on DD without a manifold learning step?

Yes, in part. We have added experiments in Appendix K. The target is a nonuniform 3-sphere embedded in D=8,32,128D=8,32,128, and the networks see only ambient points and time.

  • Ambient learning works without a manifold step. In all 27 runs, the held-out score risk is lower than that of a Gaussian fitted to the same data, by factors between 1.781.78 and 4.724.72 (Figure 2). In a follow-up at D=32D=32 and n=4096n=4096, the learned score also gives a lower empirical W1W_1 than the Gaussian score under the same sampler (Table 5).
  • On this target and at this training budget, the risk does not grow much with DD. From D=8D=8 to D=128D=128 the mean risk changes by factors of 1.251.25, 1.411.41 and 0.920.92 at n=1024,4096,16384n=1024,4096,16384, with three seeds per setting.

The experiments cannot confirm the power pβ,d\mathfrak p_{\beta,d} itself. It is a worst-case bound over the whole class, and one target at a fixed training budget cannot test it. This target is also favorable, because the sphere lies in a four-dimensional affine subspace. We read the experiments as a check that ambient learning works here, not as evidence about the exponent.

AuthorsOpenReview ↗

Response to reviewer AK59 (Part 2)

(Continued; notation as in part 1.)

W3 and Q2: Is the dependence on DD tight? Is it a cost of ERM without learning the geometry, and is the gap to Azangulov et al. (2024) fundamental?

  • (a) Tightness. Our factor Dpβ,dD^{\mathfrak p_{\beta,d}} is probably not tight. It is an upper bound from our proof and comes from two steps, the low-noise approximation error and the entropy of the score network classes (see Table 2, Lemma F.28). Any improvement would have to come from one of them. For instance, we bound scalar products of the normal residual with chart-derivative vectors by the Cauchy–Schwarz inequality, which costs a factor D∨log⁡n\sqrt{D\vee\log n} each, whereas the projection of Gaussian noise onto a fixed direction has size of order one. This is only a heuristic, because these directions depend on the projection point, and we have not carried out the sharper analysis.

  • (b) Cost of ambient ERM. Some dependence on DD is expected. Azangulov et al. (2024) choose anchor points from the training data that cover the manifold, build a local linear subspace at each anchor, and project onto it. These are fixed before training, so their networks do not have to learn the DD-dimensional maps. Standard practice has no such step: a single network is trained directly on the ambient data. This is the setting we study, and such an ambient network must represent these maps: it needs at least order DD nonzero weights to output a score in RD\mathbb R^D (see the example in Appendix A.1), and it must be accurate on a tube of radius about σtD\sigma_t\sqrt D, where the noise lies. This explains why DD appears in our bound. It is not a statistical lower bound for ambient ERM, and it does not explain the size of our power, which comes from worst-case control on this tube (see (a)).

  • (c) Gap to Azangulov et al. (2024). Part of it comes from differences in the estimator and the sampler, not from our proof; the size of our power pβ,d\mathfrak p_{\beta,d} is probably a limitation of our technique (see (a)). Their dimension-free rate (arXiv v3, Corollary 3.2) is for an estimator whose anchors and projections are fixed from the training data (see (b)), and it uses a consistency-preserving sampler; with ordinary early stopping, even the exact score has bias Ω(D,εn)\Omega(\sqrt D\\,\varepsilon_n) on manifolds padded with order DD zero coordinates (Appendix J.2), and their arXiv v2 attributes its D\sqrt D factor to the same effect. For ambient ERM they bound the score risk but state no W1W_1 rate (arXiv v3, Theorem 3.3), and their stated model size has no term in DD, although the input and output layers are DD-dimensional (Appendix A.1). Whether ambient ERM can reach a comparable dependence on DD is open.

W4: It is hard to see which parts of the proof are new, and the main theorem should be formal and self-contained.

  • (a) Self-contained main theorem. Thanks for the suggestion. Theorem 2.2 is now a formal statement, and Section 2.3 defines all its parameters before stating it.

  • (b) New and standard parts. We have added two paragraphs to Section 2.3, "Main technical difficulty" and "New approximation components". Appendix B begins with two lists, the standard tools and the results specific to this paper. The new results are the projection network (Theorem F.13) and the low-noise score construction (Lemmas F.27–F.28 and Corollary F.29).

  • (c) Length. We shortened the proof appendices. The standard tools are collected in Appendix I, which can be skipped. To check the proof, one can read Theorem F.13, then the results F.27–F.29, then Appendices G–H.

Q4: Is the limitation of ERM a result of the generic architecture, and what does the analysis say about practice?

  • (a) Generic architecture. We have not proved that it is. A generic ambient network must represent the DD-dimensional linear maps that a structured architecture fixes in advance (see W3 and Q2, part (b)). This is a cost of representation, not a statistical lower bound. The size of the power pβ,d\mathfrak p_{\beta,d} is a separate question. It comes from worst-case control in our proof, and we have no evidence that a generic architecture must pay it.

  • (b) Practice. The construction suggests two structural features: a network size that decreases as the noise grows, and a separate treatment at low noise of the normal residual and the tangential density terms. We have not tested whether building them in helps training.

Thank you again for the helpful suggestions.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of revisions

Blue in the updated manuscript marks new results, changed statements or assumptions, and revised proofs.

  • Introduction (AK59 W1, 71SR W1, tRFk W2). It now states that the intrinsic sample exponent is known and that our question is the dependence on DD. A new paragraph and the new Table 1 compare the three closest works, with details in Appendix A.
  • Section 2.3 (AK59 W4, 71SR Q1). Theorem 2.2 is self-contained, and two new paragraphs describe the main technical difficulty and the new approximation components.
  • Appendix B (AK59 W3–W4, tRFk W1). It separates the standard tools from the new results. New Tables 2 and 3 give the source of each power of DD and the size of each network.
  • Appendix J (AK59 W2–W3, tRFk W1, Q1, Q3). New: a matching minimax lower bound for the sample exponent (J.1), the early-stopping bias D εn\sqrt D\,\varepsilon_n on padded manifolds (J.2), and why our argument needs a positive density (J.3).
  • Appendix K (AK59 W2, Q3). New experiments in D=8,32,128D=8,32,128.
  • Proofs. Tracking DD step by step gave the explicit exponent, which is smaller than in the submission. The geometry constant now measures chart derivatives in Euclidean and operator norms, the low-noise expansion uses finitely many scalar products of the normal residual (Lemmas F.27–F.28).

Apart from these changes, the setting, the sample exponent and the construction are those of the submission, namely the finite-anchor Gauss–Newton projection, the two noise regimes, ambient ERM and the transfer to W1W_1.

Area_Chair_VhhVOpenReview ↗

Reviewer-author discussion kickoff

Dear Reviewers,

Thank you for your review. The authors have now provided their rebuttal. We would appreciate it if you could take some time to read their responses and engage in the discussion, particularly on whether your concerns have been adequately addressed.

Thank you for your time and contributions to the review process!

AuthorsOpenReview ↗

General Response

We thank all three reviewers for their positive and constructive reviews. Their comments helped us improve the paper, and we are grateful for the time they spent on it. As the author discussion draws to a close, we summarize the main concerns below. The detailed answers are in the individual responses.

  1. Novelty and comparison with prior work (all reviewers). The sample exponent (β+1)/(2β+d)(\beta+1)/(2\beta+d) is already known. What is new is the dependence on the ambient dimension DD. Earlier rates for ambient ERM do not track it, and it may be exponential. Up to logarithmic factors, Theorem 2.2 gives E W1≲GC(D∨log⁡n)pβ,dn−(β+1)/(2β+d)\mathbb E\,W_1\lesssim G^C(D\vee\log n)^{\mathfrak p_{\beta,d}}n^{-(\beta+1)/(2\beta+d)} with pβ,d=max⁡{⌈6β⌉+8,⌈5d/4⌉+13}\mathfrak p_{\beta,d}=\max\{\lceil6\beta\rceil+8,\lceil5d/4\rceil+13\}, where GG is polynomial in the geometry and density bounds. The new Table 1 shows the comparison.
  2. What is new beyond an explicit dependence on DD (AK59 W1, 71SR W1, tRFk W2). Two new approximation results. (i) A finite-anchor Gauss–Newton network for the nearest-point projection (Theorem F.13) that uses the smoothness order qq of the charts to reach accuracy εΠ\varepsilon_\Pi with size of order εΠ−2d/q\varepsilon_\Pi^{-2d/q}. A size of order εΠ−d\varepsilon_\Pi^{-d}, as in existing constructions (e.g., Cloninger–Klock (2021)), would be too large: at low noise projection errors are amplified by σt−2\sigma_t^{-2}, and only the smaller exponent lets the score be learned by a network of intrinsic size. (ii) An expansion of the low-noise score in which the normal residual enters only through finitely many scalar products. It shows that the coefficient functions an ambient network has to approximate depend only on the dd chart variables and on time, and this is how it can adapt to the manifold without estimating it.
  3. Why the rate is not dimension-free, as in Azangulov et al. (2024) (AK59 W3 and Q2, tRFk W1). Their estimator takes anchor points from the training data and builds a local linear projection at each anchor. These are fixed before training. In ambient ERM, where a single network is trained directly on the ambient data, the network must learn them, so some dependence on DD is expected. Our factor Dpβ,dD^{\mathfrak p_{\beta,d}} is probably not tight, and we leave the optimal dependence on DD as future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.