Non-Asymptotic Bounds for Bregman Score Matching with Mixed ReLU-ReQU Networks

Score matching provides a tractable approach to learning unnormalized statistical models, while finite-sample guarantees for deep estimators trained with general Bregman objectives remain limited. We study deep Bregman score matching, which extends the classical quadratic formulation to losses that depend jointly on the learned score field and its input Jacobian. We analyze empirical risk minimization over mixed ReLU–ReQU networks, consisting of a ReLU backbone followed by a shallow terminal ReQU block. For distributions on the full Euclidean space with bounded, Sobolev-smooth score functions, we establish non-asymptotic bounds for excess Bregman risk and population mean-squared score error under explicit regularity and tail conditions. Experiments evaluate score recovery, activation design, and the choice of Bregman divergence. On mixture-distribution benchmarks, the proposed estimator achieves lower score errors than representative kernel- and Stein-based estimators across the tested sample sizes and dimensions. Under the same quadratic score-matching objective and matched parameter counts, the mixed architecture achieves lower score and Jacobian errors than pure ReLU and pure ReQU networks. Further experiments show that different choices of Bregman divergence can improve accuracy for different score-estimation tasks, such as recovering selected coordinates or small-magnitude score components.