acceptodds
Under review as a conference paper at ICLR 2027

Non-Asymptotic Bounds for Bregman Score Matching with Mixed ReLU-ReQU Networks

Abstract

Score matching provides a tractable approach to learning unnormalized statistical models, while finite-sample guarantees for deep estimators trained with general Bregman objectives remain limited. We study deep Bregman score matching, which extends the classical quadratic formulation to losses that depend jointly on the learned score field and its input Jacobian. We analyze empirical risk minimization over mixed ReLU–ReQU networks, consisting of a ReLU backbone followed by a shallow terminal ReQU block. For distributions on the full Euclidean space with bounded, Sobolev-smooth score functions, we establish non-asymptotic bounds for excess Bregman risk and population mean-squared score error under explicit regularity and tail conditions. Experiments evaluate score recovery, activation design, and the choice of Bregman divergence. On mixture-distribution benchmarks, the proposed estimator achieves lower score errors than representative kernel- and Stein-based estimators across the tested sample sizes and dimensions. Under the same quadratic score-matching objective and matched parameter counts, the mixed architecture achieves lower score and Jacobian errors than pure ReLU and pure ReQU networks. Further experiments show that different choices of Bregman divergence can improve accuracy for different score-estimation tasks, such as recovering selected coordinates or small-magnitude score components.

Primary area:
learning theory
Keywords:
Bregman score matching, Non-asymptotic generalization bounds, Mixed ReLU–ReQU networks.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_V2CFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides statistical guarantees for Bregman score matching using mixed ReLU-ReQU neural networks, which extends classical quadratic score matching to more general convex generators. Under Sobolev-smooth score, the authors establish bounds for mean-squared score estimation error and excess Bregman risk, with an additional extension to unbounded scores for quadratic generators. Experiments compare neural architectures and existing score estimators, and investigate how different Bregman generators affect estimation errors.

Critical strengths
  1. Using mixed ReLU-ReQU networks, the paper extends the analysis of (1) DNN score matching beyond the quadratic loss to general Bregman losses, and (2) quadratic score matching by relaxing the boundedness requirement on the true score function.
  2. The experiments examine several aspects of the proposed approach, including comparisons with different activation architectures and the effects of Bregman generators on coordinate-specific and magnitude-specific estimation errors. This is valueble for a theory paper.
Critical weaknesses
  1. I found the paper difficult to follow. For example, the main mathematical problem of score matching should be formally stated earlier in the paper. The function class WW studied in the paper is buried at the end of the notation section without sufficient emphasis. The introduction consists largely of verbal descriptions and technical terminology, without clearly presenting the mathematical questions. The intuition and explanations of the theoretical results are briefly stated but not developed in a clear, logical flow that helps readers understand the authors' reasoning. I suggest moving more experimental details to the appendix and devoting more space to explaining the mathematical ideas and helping readers follow the overall argument.

  2. Mismatch between theory and experiments. Theorem 1 studies the estimator sθ=fθ∈Fns_\theta=f_\theta\in\mathcal F_n, where the score itself is represented by a mixed ReLU-ReQU network. However, Section 4.1 actually uses sθ=(∇ϕ)−1(vθ)s_\theta=(\nabla\phi)^{-1}(v_\theta), where vθv_\theta is constructed from the network output. For quadratic loss, sθ=vθs_\theta=v_\theta, while for hypentropy, sθ=βsinh⁡(vθ)s_\theta=\beta\sinh(v_\theta). While I understand that theory and experiments need not match exactly, I suggest adding experiments comparing different losses with the same score function class.

  3. The theoretical results in this paper rely on the ReLU-ReQU network structure, which, together with its approximation error bounds, mainly stands from Yang et al. (2025). I would like the authors to further clarify the necessity and theoretical advantages of using this structure in their score matching set up, as well as the main mathematical difficulties and new contributions beyond Yang's work and other existing DNN score matching results.

Questions for authors
  1. In the introduction comparing with Yakovlev et al. (2025), the paper emphasizes "without assuming low intrinsic dimension" as an advantage. However, from my understanding, an important theoretical advantage of DNN estimators over classical nonparametric estimators is their ability to overcome the curse of dimensionality under additional structural assumptions, such as low intrinsic dimension. Can the authors clarify whether their estimator can achieve such convergence rates under similar assumptions? If so, how can these rates be established? If not, what is the theoretical advantage of the proposed DNN estimator considered in this paper?

  2. Please refer to the Weaknesses.

Reviewer_jJWrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper analyses empirical risk minimisation for Bregman score matching over mixed ReLU-ReQU networks. It proves a sampling/approximation/tail bound for the population L2L^2 score error (Thm 1), derives the rate n−2m/(d+4m)n^{-2m/(d+4m)} (Cor 1) and an excess-risk version (Cor 2), and extends the analysis to finite Fisher information and Gaussian mixtures (Thm 2, Cor 3). Experiments cover the choice of divergence, activation design, comparisons with kernel and Stein estimators, and a WAE application.

Critical strengths
  • The chain of argument is correct as far as I could verify: integration by parts, the 2/λ2/\lambda conversion, the joint (output, Jacobian) cover with d+d2\sqrt{d+d^2} correctly absorbed into the constant, the W2L2W^2L^2 pseudo-dimension and the exponent identities.

  • Table 6 and Figure 1 are consistent and convincing: best in 9/9 configurations by 1.84x-14.0x, smallest p=1.2×10−7p = 1.2\times10^{-7}.

  • The limitations section is candid about the absence of lower bounds and of a Jacobian rate.

Critical weaknesses
  1. The loss is [1]; the network class, including the restriction L2≤clog⁡(eL)L_2 \le c \log(eL), is [3]; the rate and the proof template are [2] — the authors concede the rate in Remark 4. The only new mathematics is an elementary Lipschitz bound for the Hessian-coupled term (Lemma C.3). The one claimed technical novelty, a covering argument for Jacobian-dependent losses, is the declared contribution of Crafts, [4], which is not cited.

  2. By the paper's own (E.7) and (D.31), condition (13) implies Wn≥W0≈4.4×1039W_n \ge W_0 \approx 4.4\times10^{39} for d=8d = 8, m=2m = 2, so Corollary 1 holds only for n≳10317n \gtrsim 10^{317}. Every experiment uses width 128-512 and is therefore excluded by about 40 orders of magnitude; at n=2000n = 2000 the corollary's "rate-optimal" network has 3 neurons. The threshold is never quantified — only "there exists n0n_0". Separately, the bound is looser than the trivial bound d(M+H0)2d(M+H_0)^2 at the reported sample sizes, because the hidden constant carries AT4m/d≈1.8×1013A_T^{4m/d} \approx 1.8\times10^{13}.

  3. Tables 2-3 use a compactly supported density on [0,1]8[0,1]^8, violating Assumption 1, and a dual parameterisation that the authors note is outside Fn\mathcal{F}_n. Table 6 is covered only by Thm 2 / Cor 3, which assume quadratic loss, so the general-Bregman results have no empirical support. No experiment reports MM, BJB_J or ϱ\varrho, and none compares an empirical error-vs-nn slope with −2m/(d+4m)-2m/(d+4m).

  4. For the density in (24) the exact constant-zero-predictor error on the two prioritised coordinates is 4.9177, and every reported MSE1:2\mathrm{MSE}_{1:2} is 101.1-105.1% of it: those coordinates are never fitted. The 3% shift is below the design's 80%-power minimum detectable difference (0.186) and does not survive Holm correction (smallest adjusted p=0.23p = 0.23). Hypentropy improves the low-magnitude bin by 8.4% but degrades the global metric — the quantity the theory bounds — by 9.1%.

Minor issues

  • Table 4 is not a controlled ablation: lr=3×10−4\mathrm{lr} = 3\times10^{-4} for the mixed model versus 3×10−33\times10^{-3} for both baselines, and epochs are selected with the true score (the authors' own "oracle-assisted" description).
  • Table 7 contradicts Table 4 for the same 35,208-parameter architecture (0.0905 at n=2000n = 2000 versus 0.1197 at n=5000n = 5000), does not name its target distribution, and shows no benefit from increased capacity.
  • Table 5: two different methods print the identical value 126.72 at dz=32d_z = 32; "Ours" is 7.01 worse than NKEF2 at dz=128d_z = 128; only "Ours" receives the seven stabilisers.
  • Two technical slips: the log exponent in Prop. F.1 / Remark 5 should be (log⁡en)3/2/n(\log en)^{3/2}/\sqrt{n} rather than (log⁡en)/n(\log en)/\sqrt{n}, a factor 2 is missing in (F.6), and (15) is not the balance point of (14) once ϱn=Θ(log⁡n)\varrho_n = \Theta(\log n) (the correct log exponent is 2md/(d+4m)2md/(d+4m)).
Questions for authors

See weakness for details, please.

Reviewer_6oKjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies finite-sample guarantees for deep Bregman score matching, a generalization of classical quadratic score matching.

The classical approach estimates the score function, defined as the gradient of the log-density, without knowing the normalizing constant. Its quadratic loss treats all coordinates and magnitudes uniformly. Bregman score matching extends this by using a convex generator function whose Hessian weights score errors differently. This allows task-dependent priorities, such as emphasizing selected coordinates via anisotropic quadratic generators or emphasizing small-magnitude score components via a hypentropy generator, while retaining the same population target.

The loss is not directly observable because it involves the unknown true score. Under integration-by-parts conditions, it can be rewritten into an observable form that depends jointly on the learned score field and its input Jacobian, weighted by the Hessian of the generator.

The authors analyze empirical risk minimization over a mixed ReLU-ReQU network class. The architecture consists of a ReLU backbone followed by a shallow terminal ReQU block whose depth is logarithmic in total depth. This design is motivated by the need to approximate both a function and its derivatives simultaneously, since ReQU can represent products and polynomials. The class is further constrained by global envelopes on the output magnitude and Jacobian Frobenius norm. The estimator is defined as an exact minimizer of the empirical Bregman loss over this class, with optimization error assumed to be zero and a fixed measurable representative for the weak Jacobian.

The theory has two parts. First, under strong assumptions including everywhere positive continuously differentiable density, uniformly bounded Sobolev-smooth score on the entire Euclidean space, strong convexity and Lipschitz Hessian of the generator on a bounded cube, and a uniform moment generating function bound, the authors prove a non-asymptotic bound for population mean-squared score error. The bound decomposes into a sampling term governed by the pseudo-dimension of the mixed class, an approximation term that decays polynomially in a product of width and depth with a radius factor, and a tail probability outside a growing cube. Balancing these terms yields a convergence rate that decays polynomially in sample size with an exponent that depends on dimension and smoothness order, up to logarithmic factors, and a matching excess Bregman risk bound. Second, for the quadratic generator only, the analysis is extended to unbounded scores under finite Fisher information and local Sobolev regularity, with envelopes allowed to grow with sample size. For finite Gaussian mixtures, an explicit rate with the same exponent is obtained by controlling the local Sobolev norm growth and tail decay.

Empirically, the paper evaluates three aspects. A coordinate-priority and magnitude-priority diagnostic on a synthetic cosine density in eight dimensions shows that anisotropic and hypentropy generators can reduce error on prioritized coordinates or low-magnitude components at the cost of increased error elsewhere. A grid Gaussian mixture benchmark against kernel and Stein estimators in dimensions 8, 32, 64 and sample sizes 500 to 2000 reports lowest mean error for the mixed estimator in all configurations. A matched-parameter ablation comparing pure ReLU, pure ReQU, and mixed networks on a correlated Gaussian reports lower score and Jacobian errors and higher cosine similarity for the mixed architecture, but uses oracle epoch selection based on true validation score. Finally, a Wasserstein autoencoder application on MNIST evaluates latent score estimation via annealed importance sampling negative log-likelihood, showing competitive but not statistically significant differences.

Critical strengths
  1. First attempt to provide non-asymptotic excess risk and L2 score bounds for general Bregman losses that depend on both network output and Jacobian, not just quadratic loss.

  2. Clearly separates sampling complexity via pseudo-dimension and Rademacher chaining, approximation via simultaneous Sobolev approximation, and tail via MGF assumption. Extension to finite Fisher information and Gaussian mixtures provides some unbounded case.

  3. Anisotropic and hypentropy generators provide a principled way to discuss coordinate and magnitude priorities, with diagnostic experiments illustrating trade-offs.

Critical weaknesses
  1. Assumption 2 requires the true score to be in Sobolev space with uniform bound on all derivatives up to order m on the entire space. Together with everywhere positive C1 density and uniform MGF bound in a neighborhood of origin, this essentially forces light tails and globally bounded scores and derivatives. This rules out even simple Gaussian scores which are unbounded linear functions, except via the separate quadratic-only extension. The fixed envelope and Jacobian bounds are also assumed independent of n in the main theorem. The hidden constants depend on dimension, smoothness, and curvature and are not tracked, potentially exponential in dimension. No adaptation to low intrinsic dimension, unlike recent work cited.

  2. Corollary rate is the standard nonparametric regression rate for m-smooth functions in d dimensions, suffering severely from curse of dimensionality. Same exponent already obtained by prior work for quadratic score matching with RePU networks. The contribution is extending to mixed networks and general Bregman, but exponent does not improve, and logarithmic factors are larger. For high dimensions, exponent is essentially vacuous for practical n. Width choice grows sub-polynomially and is impractical. No minimax lower bound to show optimality for Bregman class.

  3. Estimator defined as exact minimizer with zero optimization error, measurable selector, and fixed bounded Jacobian representative using a guard indicator. No analysis of SGD or non-convex optimization. Practical networks cannot guarantee pointwise output and Jacobian bounds on entire space. Authors admit in conclusion that practical experiments do not certify global function-class constraints. The theoretical loss requires exact trace evaluation with Hessian weighting, which is expensive and not compared to sliced score matching that avoids trace.

  4. Strong convexity and Lipschitz Hessian on bounded cube are required. For hypentropy, lower curvature, upper curvature, and Hessian Lipschitz constants blow up as beta goes to zero. To emphasize small components, small beta is desired, but then loss envelope and Lipschitz constants blow up, inflating stochastic error constants. Paper admits generators share same exponent and constants involve curvature through risk conversion, without ranking Euclidean errors, and explicitly states neither experiment establishes uniformly superior generator. Thus benefit is tautological: reweighting changes error allocation.

  5. Experiments are all small-scale.

  6. Proofs rely heavily on pseudo-dimension bound and VC-type covering, chaining with explicit constants that depend on loss envelope and Lipschitz constants and are fragile. AI use statement says generative AI tools were used to assist in formulation of mathematical claims and proof development, including suggesting intermediate arguments and bounds used in establishing stochastic error bound, which raises concerns about correctness of intricate chaining steps.

  7. Width definition contains exponential in dimension factor, making it impractical. Main text does not convey intuition for why mixed architecture needs logarithmic terminal depth. Heavy appendix with many lemmas but limited discussion of how to choose output and Jacobian envelopes in practice.

Questions for authors
  1. Can you give a single non-trivial distribution on full Euclidean space with dimension greater than one that satisfies uniformly bounded Sobolev score with order at least two and finite MGF, other than compactly supported smooth density? How does this relate to Gaussian mixtures which violate it?

  2. What is the explicit dependence of capacity and main constants on dimension? Does dimension factor grow exponentially due to width formula? Can you provide a non-asymptotic numerical bound for dimension 32 with smoothness order 2 and sample size 2000 to illustrate vacuity?

  3. How are output and Jacobian envelopes enforced during training? The guard in appendix is a post-hoc truncation for analysis, not part of SGD. What happens to theory if guard is omitted in practice?

  4. For beta equal to 1 and typical score range, what are curvature bounds? How do loss envelope and Lipschitz constants scale as beta goes to zero? Does this explain why global error worsens despite low-magnitude improvement?

  5. Why does activation ablation use true score for epoch selection? Can you repeat with validation score-matching loss and report gap? Does learning rate difference explain mixed advantage?

  6. Exact trace requires many Jacobian-vector products scaling with dimension. How does training time scale with dimension 64 versus sliced score matching? Would results hold with Hutchinson estimator?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.