acceptodds
Under review as a conference paper at ICLR 2027

Div, Grad, Curl, and All That: Electrostatic Models for Score Matching

Abstract

We show how to estimate a generative model from samples by parameterizing its score function as the electric field of a static charge distribution. The score function is given by the gradient of the model's log-density, and as such, it must define a conservative vector field. By modeling the score as an electric field, we not only guarantee that it satisfies this curl-free constraint; we can also efficiently compute its divergence, via Maxwell's equations, from a local charge density. We parameterize the score as a superposition of electric fields from spherically symmetric charge distributions and uniformly charged slabs. We estimate the coefficients in these superpositions by minimizing a Fisher divergence, and notably, the optimal coefficients are given by the closed-form solution of a regularized least-squares problem. On one hand, these models have a natural connection to energy-based models and product-of-expert models, and they include existing kernel estimators as a special case; on the other hand, they can be learned without recourse to iterative gradient-based methods. We present the results of experiments on synthetic data sets as well as problems of higher dimensionality. Our results show that these charge-based models perform competitively with neural and kernel-based baselines, and they do so while using fewer parameters, learning more efficiently, and yielding more geometrically interpretable densities.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
score matching, electrostatic models, score-based generative modeling, score estimation, energy-based models, kernel exponential families, physics-inspired machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dz1pOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper represents score functions as superpositions of electrostatic fields generated by spherical and slab components. This construction provides conservative fields with analytic divergences. For fixed geometries and a fixed base field, coefficient estimation reduces to a regularized quadratic problem. Experiments cover synthetic score estimation and latent score estimation in Wasserstein autoencoders.

Critical strengths

A tractable construction of conservative score fields. The proposed components have analytic fields and divergences, and coefficient estimation reduces to a regularized quadratic problem when the geometries and base field are fixed. This makes the score-matching objective directly computable while preserving the gradient-field structure, providing a useful alternative to unconstrained vector-field models.

Promising results with compact mixed dictionaries. On the synthetic GMM benchmark, the mixed model achieves a score MSE of 0.088±0.0120.088 \pm 0.012 with approximately 21 selected components, outperforming the reported spherical-only and slab-only variants. This provides concrete evidence of the approach’s practical promise for structured score-estimation problems.

Critical weaknesses

The source of the empirical improvements is insufficiently isolated. Although the appendix includes useful ablations, some comparisons differ in base-field fitting, regularization, dictionary size, or validation criteria. Given the close connections to kernel exponential families and tied-weight ReLU models, stronger controlled comparisons are needed to establish the benefit of the proposed mixed construction beyond these implementation choices. This uncertainty limits the evidence for its distinct empirical contribution.

The efficiency evaluation does not capture the complete training pipeline. The reported timings exclude dictionary construction and selection, which can involve clustering, ICA, adaptive search, and repeated fitting. A fast coefficient solve for a fixed dictionary therefore does not establish an end-to-end computational advantage. Since computational tractability is a central motivation of the paper, complete timing measurements under comparable tuning budgets are important to substantiate its practical benefits.

Questions for authors

Comparability of the experimental settings. Are base-field fitting and validation criteria matched between the proposed method and the closest KEF and tied-weight ReLU baselines? Which existing controlled comparisons best isolate the benefit of the mixed dictionary from these other choices?

End-to-end computational cost. For one representative experiment, could you provide a runtime breakdown covering dictionary construction, adaptive selection, and coefficient/base-field fitting? Does the reported speed advantage persist when these stages are included and the baselines are timed under the same accounting rules?

Scope of the density interpretation. Are any constraints imposed to ensure that the fitted potential defines a normalizable density? If not, is the method intended primarily as a score estimator, and do any of the reported experiments require the fitted field to correspond to a normalizable density?

1 comment

Thank you for identifying the comparisons needed to clarify the contribution. We have revised the manuscript with matched geometry and base-field comparisons (Appendix G.1), full search-cost measurements (Appendices G.4 and G.7), and a clarification of normalizability (Appendix A.4). These additions are highlighted in green in the revised manuscript.

1. Comparability of the experimental settings.

“Are base-field fitting and validation criteria matched between the proposed method and the closest KEF and tied-weight ReLU baselines?”

Yes. We added frozen-CIFAR comparisons using the same representation, splits and validation criterion, with 1,024 and 2,048 total components: K spheres for Gaussian KEF, K slabs for ReLU models, and K/2 of each for mixed. Each is tested without a base, with a fixed Gaussian base, and with joint symmetric base refitting. Fixed geometries share the applicable regularization grids; learned tied/untied ReLUs share their training-setting search.

Mixed has the lowest mean test J in every base setting at both budgets. With 2,048 components and joint refitting, it obtains −1628.83 ± 6.26, versus −1495.32 ± 0.65 for Gaussian KEF, −1538.09 ± 6.20 for fixed slabs, and −1544.25 ± 6.03 for learned untied ReLU; lower is better.

Thus, matching component budgets and base treatment preserves the advantage (Appendix G.1). Under the same SM objective, neural SM obtains −1531.85 ± 3.05, also outperformed by the mixed geometry model (Appendix G.2).

2. End-to-end computational cost.

“For one representative experiment, could you provide a runtime breakdown covering dictionary construction, adaptive selection, and coefficient/base-field fitting? Does the reported speed advantage persist when these stages are included and the baselines are timed under the same accounting rules?”

Yes. Full red/white wine searches take 3.28/3.57 seconds, including geometry construction, cross-validation, and final refitting, versus 199.59/269.75 for SSM. Appendix G.7 separates geometry proposals, moments, coefficient/base solves, validation and overhead. Both exclude preprocessing and checkpointing required from GPU use. SSM has lower mean wine loss; mixed provides a substantially faster fit with competitive performance results.

On the d=30 Gaussian mixture, mixed is both faster and more accurate: MSE 4.85 in 4.8 seconds, versus 12.67 in 34.5 seconds for neural SSM, including geometry construction and search for mixed (Appendix G.4). Wine selection compares fixed dictionaries; Appendix G.8 separately measures greedy additions and refitting beyond validation stopping, conditional on selected settings.

3. Scope of the density interpretation.

“Are any constraints imposed to ensure that the fitted potential defines a normalizable density? If not, is the method intended primarily as a score estimator, and do any of the reported experiments require the fitted field to correspond to a normalizable density?”

Our reported fits are score estimators; we do not impose a global normalizability constraint, and our experiments do not require a normalizing constant. Conservativeness means that the score can be written as −∇V-\nabla V. To additionally define a probability density over all space, we need ∫e−V(x)dx<∞\int e^{-V(x)}dx < \infty. A conservative field alone does not guarantee this.

Appendix A.4 explains a sufficient condition. Intuitively, the linear base must pull inward strongly enough that positive charge extending to infinity does not overcome it. Gaussian spherical potentials are bounded, and finite-thickness slab potentials grow at most linearly, so a confining quadratic base dominates these contributions far from the data. Our fitted one-sided slab boundaries can instead contribute quadratic growth. A sufficient condition is

\qquad \varepsilon > 0,$$ where $\Omega$ is the symmetric linear base, $\theta_j$ is a slab coefficient, and $u_j$ its unit direction. This ensures $V(x) \geq \varepsilon \lVert x \rVert^2/2 - C\lVert x \rVert - C$ for some finite $C$, making the integral finite. Negative slab coefficients help confinement. Alternatively, a continuous potential restricted to a bounded box defines a normalizable density on that box. Neither restriction is imposed in the reported experiments. **Score MSE and J evaluate the score field directly and do not require a normalizing constant.** WAE generation and AIS use the model defined by the Gaussian latent prior and decoder, rather than normalizing the fitted score potential. The finite-step diffusion experiment likewise demonstrates sampling without claiming that its estimated fields define a normalizable static density.
Reviewer_Y1B9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an electrostatic formulation of score matching to resolve a fundamental tension in training score-based generative models: parameterizing the score as a gradient ensures conservativeness but makes the divergence expensive to compute. By drawing a direct analogy to electrostatic, where the score is an electric field and the probability density relates to charge. The authors show that one can instead parameterize the charge density. Because the divergence of the field equals the charge by Gauss's law, this approach guarantees a conservative score field while making the divergence trivially easy to evaluate. To make the model expressive, the authors construct a dictionary of analytic basis functions based on spherically symmetric charges (Gaussian kernels) and uniformly charged slabs (which act like one-sided ReLU MLPs). Because the score is a linear combination of these components, the Fisher divergence reduces to a simple quadratic least-squares problem, allowing for highly efficient optimization without relying on standard neural network backpropagation.

Critical strengths
  1. The text is well-written and easy-to-follow
  2. By representing the score as a linear combination of analytical fields, the training objective becomes a simple quadratic optimization problem that can be solved directly (or with standard least squares) rather than requiring expensive iterative backpropagation.
  3. The method elegantly solves the dilemma of standard neural networks by guaranteeing that the learned score is a true conservative field (a valid gradient of a scalar) while avoiding the computational cost of calculating the Laplacian/Hessian.
Critical weaknesses
  1. The method is tested in experiments only with MNIST 32x32
  2. The model relies on a fixed dictionary of basis functions (spherical or slab components) combined linearly, which may require a massive number of components to approximate highly complex, non-linear data distributions compared to deep, multi-layered neural networks.
  3. The predefined shapes of the charge distributions (like the infinite, uniform slabs or strictly spherical fields) might lack the flexibility to capture arbitrary local geometries or sharp boundaries in real-world high-dimensional data.
Questions for authors
  1. Could you provide more high-dimesnisonal experiments with 64x64 Celeba or imagenNet?
  2. How does the required number of spherical and slab components scale with the dimensionality of the data compared to the parameter count of a standard deep neural network?
  3. How does this model perform on highly complex, high-dimensional image datasets where the underlying data manifold is highly curved and multimodal, rather than simple toy distributions?
2 comments

Thank you very much for your helpful feedback. We have now added 64x64 CelebA generation with a WAE, a 512-dimensional CIFAR-10 score-estimation experiment, and capacity studies. We have revised Section 5 and added results in Appendices F and H, with capacity and geometry ablations in Appendix G. These additions are highlighted in red in the revised manuscript.

1. Image generation beyond MNIST

“Could you provide more high-dimensional experiments with 64x64 CelebA or ImageNet?”

We now generate 64×64 CelebA images with WAEs at latent dimensions 8, 32 and 64, alongside the original MNIST and CIFAR-10 WAE experiments. Visuals are at the end of the manuscript, in Appendix H. These demonstrate image generation through a learned decoder.

We also test score estimation directly in a shared frozen CIFAR-10 representation with d=512. Mixed geometries had the lowest mean test loss among all baselines: J = −1628.83 ± 6.26, versus −1403.79 ± 1.57 for neural SSM and −1437.60 ± 2.13 for full Gaussian KEF, where lower is better. This separates the high-dimensional score-estimation test from the low-latent dimensional WAE task (Section 5.5).

2. How many components are needed?

“How does the required number of spherical and slab components scale with the dimensionality of the data compared to the parameter count of a standard deep neural network?”

The number of components is selected from the data, rather than prescribed by dimension alone. It depends on the sample size, distribution and validation criterion. In our GMM dimension sweep, the selected number of slabs and balls increases from about 58 at d=5 to 480 at d=100, compared to 17,925 to 42,340 fitted neural parameters. Appendix G.6 plots the sphere/slab counts and validation-selected neural capacity across dimensions.

With fixed geometries, we fit K+d(d+1)/2+dK + d(d+1)/2 + d parameters: one coefficient per component plus the symmetric base Ω\Omega. Most fitted parameters in these high-dimensional experiments belong to the base, rather than the geometries. A two-hidden-layer neural model of width w uses w² + 2dw + 2w + d parameters, growing quadratically with width. On frozen CIFAR-10, mixed uses 133,888 fitted parameters—131,840 from the base—versus 6,296,064 for SSM, about 47× fewer. On the 512-dimensional CIFAR-10 experiment, the mixed geometry model has the lowest number of fit parameters compared to all the non-Gaussian baselines, and the best test loss. We also vary component counts and neural widths at d=5, 30 and 75 to show how capacity affects accuracy (Appendices G.3 and G.9).

3. Can simple geometries represent complex scores?

“How does this model perform on highly complex, high-dimensional image datasets where the underlying data manifold is highly curved and multimodal, rather than simple toy distributions?”

The frozen CIFAR-10 experiment tests the score estimators on complex image data in 512 dimensions. The results above show that combining local spheres and directional slabs can represent complex scores better than the other baselines, with fewer fitted parameters, in this high-dimensional image representation. To test what the combination contributes, we compare sphere-only, slab-only and mixed models at matched total component counts. Mixed has the lowest mean test loss under no base, a fixed Gaussian base and joint affine refitting, at both tested budgets (Appendix G.1).

The advantage also holds when the neural model is trained with the same SM objective: neural SM obtains −1531.85 ± 3.05, versus −1628.83 ± 6.26 for mixed (Appendix G.2). Together with the CelebA WAE experiments, these provide both high-dimensional score-estimation and image-generation results beyond the original synthetic examples from MNIST and CIFAR-10 WAEs.

Reviewer_Y1B9OpenReview ↗

Answer

Thank you, you address my question, but careful scalability is still main concern. You should take more scalable datasets and make ablation of your model to compare with SOTA methods.

Reviewer_WUdzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose an alternative approach to score estimation based on concepts from electrostatics. They claim to parameterize the charge density (although, in practice, each component is defined through its potential) as a weighted dictionary of one-sided slab boundaries and spherically symmetric charges with fixed geometric characteristics. The components are chosen from a fixed candidate pool by greedy forward selection, and their geometry is not learned. A linear base field is obtained by solving a Lyapunov equation, alternated with a closed-form solution for the coefficients θₖ (block coordinate descent), so no gradient-based optimizer is needed to fit the score model. The resulting model is lightweight and has an exact, analytic divergence, which allows Hyvärinen score matching to be used at low computational cost. The method is evaluated on 2D synthetic distributions, a synthetic dimension sweep up to d = 100, and latent score estimation in Wasserstein autoencoders on MNIST and CIFAR-10.

Critical strengths
  1. The paper is well written and easy to follow. While the idea of using a scalar output parametrization to get a conservative score is not novel, the translation of generative modelling concepts to electostatic language is definitely novel.
Critical weaknesses
  1. The paper or related literature in general has no evidence to show that the "unconservativeness" of the score function affects the sampling process or creates any artifact in the generated sample. Hence, major claims of the paper are limited to being lightweight and obtaining exact divergence cheaply.
  2. Consequently, the results in Table 2 and Table 3 show that improvements are incremental or mixed. Moreover, choices like what dictionary to use and the characteristics of the spherical or slab constructs used in the dictionary seem to be heuristically decided or require a greedy search (adding to the computation cost).
Questions for authors
  1. Is there any evidence (a sanity check) to prove that hard constraining the score to be conservative helps with the quality of generation or vice versa?
  2. Can the optimization time also be compared with other benchmarks?
1 comment

Thank you for your time and thoughtful review. Your questions motivated additional tests of conservativeness, computational costs, and geometry selection. Below, we explain what these tests show and how they clarify our paper’s contribution. We have updated the manuscript to reflect these clarifications; conservativeness and selection additions are highlighted in blue, and runtime/baseline ablations are in green.

1. Conservativeness and generation

“Is there any evidence (a sanity check) to prove that hard constraining the score to be conservative helps with the quality of generation or vice versa?”

Prior work provides a motivation for conservative score estimation: Zhou et al. (2020) construct curl-free kernel estimators, enforcing that the fitted field is a gradient, and thereby improving score estimation in their settings. For diffusion sampling, Horvat and Pfister (2024) show that conservativeness alone is not necessary for exact sampling, but demonstrate that the constraint is useful to estimate the true score function near data manifolds. We clarify this literature in Section 1.

In Appendix A.3, we further discuss how conservativeness can help estimation. For a standard Gaussian target and a conservative base field, removing the antisymmetric part of a score correction reduces population score error. This illustrates how the constraint can remove the curl components introduced by finite samples.

We then test this with the same data and geometry: the conservative fit reduces score MSE from 0.281 to 0.265, and penalizing curl brings the unrestricted fit towards it. The advantage is smaller as the amount of data increases (Appendix G.5). Separately, we perform a small generation experiment with half-moons diffusion; mixed geometries achieves a sliced Wasserstein distance of 0.068, compared to 0.183 for neural DSM, and 0.372 for neural SSM (lower is better, Appendix F.3). This demonstrates a small generation task in which the conservative mixed geometry model performs better than non-conservative alternatives, however the conservative-specific experiment in Appendix G.5 isolates its benefit.

2. Exploration of speed advantage

“Can the optimization time also be compared with other benchmarks?”

We performed additional experiments on the UCI red/white wines dataset. Full searches took 3.28/3.57 seconds on average, versus 199.59/269.75 for neural SSM. SSM has lower mean wine loss, so this is a speed-accuracy tradeoff. On the 30-dimensional Gaussian mixture, the mixed geometry model is both faster and more accurate throughout the whole pipeline: MSE 4.85 in 4.8 seconds, versus 12.67 in 34.5 seconds for neural SSM. These search-computation timings, full experiments, and results are represented in the updated Appendix G.4 and Appendix G.7.

3. What does mixing geometries contribute?

“Improvements are incremental or mixed,... choices like what dictionary to use and the characteristics of the spherical or slab constructs used in the dictionary seem heuristically decided or require a [computationally expensive] greedy search”

The question is whether combining geometries helps beyond simply adding more components. To test this, we compare models with the same total component count on a higher-dimensional d=512 frozen CIFAR-10 dataset under no base, a fixed base, and a jointly refit base. Mixing geometries beats matched-parameter sphere- or slab-only models in a direct comparison, with J=-1628.83, versus -1495.32 for Gaussian KEF, and -1538.09 for fixed slabs, where lower is better (Appendix G.1). On this high-dimensional CIFAR-10 experiment, the mixed geometries also outperforms every baseline; we have updated our main experiments section to reflect this new experiment.

Selection also makes a measurable difference: at the d=30 GMM, greedy selection reduces MSE from 11.29 to 4.93 compared with random subsets. We now additionally show accuracy against search cost in Appendices G.8-G.9.

Comments on the paper

1 comment

We thank the AC and the reviewers for their time and thoughtful feedback. Below, we briefly summarize the main additions to the revised manuscript:

  • Expanded, competitive evaluations: We added additional high-dimensional real data: a score estimation task in 512 dimensions using a frozen CIFAR-10 representation. Our mixed geometry model achieves the lowest mean test loss among all evaluated baselines, with fewer fitted parameters than all non-Gaussian baselines. We also added red and white wine score-estimation experiments, with competitive performance using substantially fewer fitted parameters, 64×64 CelebA WAE generation, and a half-moons diffusion sampling experiment to address the requested generation sanity check.
  • Computation cost: New measurements include geometry construction, model selection, and coefficient/base-field fitting. On red and white wine, the complete geometry construction, candidate fitting, validation, and final refitting searches take 3.28 and 3.57 seconds in total, compared to 199.59 and 269.75 seconds for SSM, respectively. Our model achieves slightly higher mean loss than SSM on the low-dimensional wine datasets, although it offers competitive accuracy with significantly lower computational cost and parameter counts. On the 30-dimensional synthetic GMM, our model is both faster and more accurate than SSM.
  • Detailed, controlled ablations: On the 512-dimensional CIFAR-10 representation, mixed geometries outperform sphere-only and slab-only models at both a matched-component budget and across different base-field integration options. We include additional comparisons with neural models trained using different objectives to confirm the advantage the mixed geometry model holds. We also include ablations of how selected component counts and neural parameter counts vary with dimension, and test geometry selection and the accuracy–computation tradeoff.
  • Clarified conservativeness and normalizability: We also clarified the role of conservativeness and normalizability in score estimators. We clarified the circumstances in which the estimator would have a normalizable density, and confirmed that our score-matching experiments did not require normalization of the estimated potential. However, we did add a sufficient condition in order for the fitted potential to define a normalizable density. Separately, we expanded the discussion of conservativeness with a literature discussion and an added controlled experiment showing improved score estimation from enforcing this conservative structure, with the advantage shrinking as the sample size increases.

Following our additions, Reviewer Dz1p wrote: “Based on this response, my main concerns are substantially alleviated, and my assessment is more positive.” All of these results and clarifications are already included in the revised manuscript and will be retained. We appreciate the constructive feedback and remain happy to address any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.