acceptodds
Under review as a conference paper at ICLR 2027

Riemannian Energy Matching for Fixed-Density Transport

Abstract

An energy-based model defines a target density, but efficient sampling also depends on how particles move through it. Energy Matching uses the same scalar function to define both the density and the transport field. Improving transport by changing the energy therefore also changes the distribution being sampled. We ask whether learning the geometry of transport can improve sampling while keeping the energy fixed. We introduce Riemannian Energy Matching (\REM), which learns a symmetric positive-definite mobility from optimal-transport velocities. This mobility redirects the frozen energy gradient, while a unit-determinant constraint rules out uniform time rescaling. We characterize the velocities this model can represent and derive an objective for fitting them along transport paths. A divergence correction ensures that the resulting diffusion preserves the original Boltzmann density in continuous time. On a curved multimodal target, full mobility captures motion that diagonal models miss. Removing the correction from the same fitted mobility substantially increases sampling error. A high-dimensional study shows that geometry can improve field fitting without changing the energy-defined density. For image generation, we use learned geometry during deterministic transport and return to Euclidean dynamics during stochastic refinement. This phase-gated strategy improves generation quality under matched solver budgets. \REM provides a framework for separating errors in density learning from limitations in transport geometry, and for testing whether a better fitted field leads to better samples.

Primary area:
generative models
Keywords:
energy-based models, optimal transport, riemannian Langevin dynamics, mobility learning, generative sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6QLPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Traditional Energy Matching (EM) learns both the target density and the transport field. However, changing the transport velocity also changes the target density. This may affect training when the velocity field has not been learned accurately. To address this issue, the authors freeze the energy function, keeping the target density fixed, and introduce a symmetric positive-definite (SPD) mobility matrix to modify only the transport velocity. The proposed method focuses on modifying the flow-like objective of EM. The experiments first validate the effectiveness of the method on a 64-dimensional Gaussian distribution. The authors then evaluate the method on CIFAR-10 and ImageNet32, achieving better results than EM under matched solver budgets.

Critical strengths
  1. Well-motivated problem formulation. The paper identifies an important limitation of Energy Matching (EM), where the same scalar energy function determines both the target density and the transport dynamics. Consequently, modifying the energy to improve transport may also alter the learned density. The proposed separation of density modeling and transport geometry provides a conceptually appealing direction for improving energy-based generative models.

  2. Solid theoretical analysis. The paper provides a characterization of the velocity fields representable by determinant-one SPD mobility matrices and establishes conditions under which the corrected stochastic dynamics preserve the original Boltzmann distribution. These results offer a clear mathematical foundation for the proposed framework and help distinguish transport flexibility from density preservation.

  3. Carefully designed controlled experiments. The synthetic experiments demonstrate the benefits of full SPD mobility on curved multimodal distributions. In particular, REM substantially improves velocity fitting and mode passage compared with Euclidean and diagonal alternatives. The ablation on the divergence correction is also informative, highlighting its importance for preserving the target distribution under state-dependent mobility.

Critical weaknesses
  1. Limited practical improvements over Energy Matching. Although the motivation is compelling, the improvements on real-world image generation benchmarks are relatively modest. On CIFAR-10, REM reduces FID from 3.462 to 3.297, while on ImageNet32, the improvement is from 6.742 to 6.555. These gains are consistent under the main evaluation protocol but remain small relative to the additional modeling and training complexity. Furthermore, the improvements are not robust across sampling budgets: REM performs worse than EM at 400 energy-gradient evaluations. The paper would benefit from a more comprehensive analysis demonstrating when and why the proposed approach provides meaningful practical advantages.

  2. Insufficient evidence that the main geometric mechanism is effective in high-dimensional applications. A central motivation of REM is that full SPD mobility can capture coordinate interactions that Euclidean or diagonal transport cannot represent. However, the strongest evidence for this advantage comes from low-dimensional synthetic experiments. In the main image generation experiments, REM employs diagonal mobility, and the rank-four extension does not improve performance over the diagonal alternative on CIFAR-10 (FID 3.299 versus 3.297). The higher-dimensional synthetic results also show substantially smaller sampling improvements than the two-dimensional experiments. Consequently, the practical importance of learning non-diagonal transport geometry remains insufficiently established.

  3. Additional complexity without a sufficiently demonstrated efficiency–performance trade-off. REM requires training an additional mobility network for 150k steps on each pretrained image energy. While the inference overhead of the phase-gated image sampler is relatively small, the paper does not provide a comprehensive end-to-end comparison that accounts for additional training, inference cost, and generation quality. Moreover, the corrected full-mobility sampler is substantially more expensive than Euclidean sampling in the synthetic experiments. A comparison based on matched wall-clock budgets or total computational costs would be more informative than relying primarily on matched solver steps or energy-gradient evaluations.

  4. A gap between the theoretical guarantees and the main empirical improvements. One of the central theoretical contributions is the preservation of the original Boltzmann distribution through divergence-corrected Riemannian dynamics. However, the primary image generation experiments disable learned mobility during stochastic refinement and use it only in the deterministic transport phase. Thus, these results primarily demonstrate improvements in finite-time transport rather than the practical benefits of the proposed invariant-density stochastic dynamics. Although an additional experiment considers ungated sampling, its evidence is limited, and the corrected ungated sampler performs worse than the phase-gated alternative. Stronger evaluations of high-dimensional stochastic sampling and convergence to the target distribution are needed to connect the theoretical framework with its main empirical claims.

  5. Limited experimental scope compared with the original Energy Matching framework. The original EM paper demonstrates applications beyond unconditional image generation, including protein inverse design, image inpainting, and Hessian-based local intrinsic dimension estimation. In contrast, REM mainly evaluates synthetic distributions and low-resolution image generation. While Hessian-based LID estimation is not necessarily expected to improve when the energy function is frozen, the absence of downstream conditional generation or inverse-design experiments leaves the broader applicability of REM unclear. In particular, evaluating protein inverse design or another energy-guided sampling task would provide valuable evidence that the learned transport geometry improves practical sampling efficiency or solution quality beyond standard image generation benchmarks.

  6. The motivating limitation of EM is not directly validated. The paper argues that EM suffers from coupling density learning and transport optimization, but it does not directly establish how severely this coupling affects the optimization process or final generation quality. Figure 1 provides an intuitive illustration of different transport trajectories rather than direct evidence of optimization conflicts in EM. Additional experiments examining the interaction between transport and density objectives, or comparing geometry adaptation with simpler transport preconditioning and continued energy optimization, would strengthen the causal justification for REM's improvements.

Questions for authors
  1. Can REM also improve the results of EM on the other two tasks, Protein Inverse Design and Hessian-based LID Estimation?

  2. Table 11 shows that REM does not consistently outperform EM across different sampling budgets. In particular, REM performs worse than EM at 400 energy-gradient evaluations. Could the authors explain why the performance gains are sensitive to the sampling budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6QLP (1/2)

Thank you for the careful review. We address your six weaknesses in order, and the two questions within them. Shared results are in the general response. All changes are in blue in the revised PDF, with new experiments in Section 4.5 and Appendix E.

1. Practical gain and budget sensitivity (Q2). We agree the gain is modest, and the new equal-search control makes it smaller still. When both samplers receive the same 12-candidate time/noise-onset search on disjoint validation streams, REM keeps 3/3 paired wins on fresh 50k streams at both 400 and 650 NFE. The mean gain is −0.041 and −0.026 FID (Table 4).

This also answers why the earlier result depended on budget. At 400 NFE, the original schedule (fixed cutoff τ_g = 1, not retuned) was a poor operating point. The equal search selects a shorter terminal time and an earlier onset/cutoff (T = 2.75, 0.5), and there the earlier loss becomes a small gain. The sensitivity therefore comes from the interaction between the learned geometry and the time/noise schedule. It is not a budget-independent loss. The original reversal stays in the paper, and the revised abstract and conclusion state the smaller tuned gain.

2. Non-diagonal geometry in high dimension. We agree. Rank-four image mobility does not improve on diagonal mobility, and the 64D gain is a field-fit result. The new lawful controls in 16D give SPD the lowest field error but no consistent stochastic sampling advantage. The revised discussion separates these representation findings from the finite-time diagonal image result we actually demonstrate.

3. End-to-end cost. We now report the complete ledger (Appendix E.6):

  • Mobility training: 11.7–12.9 H100-hours per fit.
  • Phase-gated inference: adds only 0.4–0.9% to 50k sampling time.
  • Active stochastic mobility: costs about 44 ms vs. 30 ms per image step with one divergence probe.

The most direct total-compute comparison is the continuation control. Spending the same extra training time on the original EM objective, instead of training a mobility, worsens FID from 3.484 to 4.309, while REM reaches 3.315 under the same fixed schedule.

In synthetic equal-wall-time tests, identity has the lowest estimator error. The revision therefore limits efficiency claims to matched-step synthetic gains and finite-time image gains, and makes no wall-clock sampling claim.

AuthorsOpenReview ↗

Response to Reviewer 6QLP (2/2)

4. Gap between the guarantee and the image results. We agree, and the revised abstract and results now describe the image gains as finite-time preconditioning of the deterministic phase. To probe the stochastic phase directly, we started refinement from identical saved boundary states. Corrected Heun refinement with learned mobility then lowers 10k FID from 6.581 to 6.163, and every Heun condition improves on identity, at about 1.5× the stochastic cost (Appendix E.5). The earlier ungated screen (FID 9.018 vs. 7.259) differs from this boundary experiment in its prefix and activation range, so we do not attribute its gap to one cause. This is still a one-fit, finite-time diagnostic, not a convergence proof. The invariant-law mechanism itself is tested in the controlled synthetic experiments.

5. Experimental scope (Q1).

  • Protein inverse design. We added an independent AAV Medium/Hard confirmation: 3 fits × 5 fresh paired streams at locked operating points, 512 candidates per run, and at most 128 unique sequences selected with the official procedure. REM − EM on the selected sets:
TaskFitnessDiversityNovelty
Medium−0.0022+0.0710.0
Hard−0.0119+0.233−0.7

REM increases diversity on both tasks at a small fitness cost, and it is slower. This is a trade-off, not a uniform improvement, and the evaluation uses a learned oracle (Appendix E.7).

  • Hessian-based LID. As you note, freezing V leaves its Hessian unchanged at any fixed input, so REM cannot change this estimator by design (Appendix E.8).
  • Inpainting. EM's inpainting uses CelebA 64×64 with a separate, larger architecture whose weights are not publicly released. The public checkpoints cover CIFAR-10, ImageNet32 and AAV. Inpainting, like the 64×64 question from vLp5, would require training a new energy, so we state it as a limitation.

6. Is the motivating coupling real? We added the comparison you suggest: continuing to optimize the energy versus adapting geometry. Continuation restores the official full checkpoint, optimizer, scheduler and EMA, and runs the original transport + contrastive objective for one mobility fit's training time. It does not improve transport: relative OT velocity MSE rises from 0.1301 to 0.1365, while REM lowers it to 0.1193. FID also worsens, from 3.484 to 4.309 (Appendix E.2). The simpler preconditioning baseline you mention is the constant SPD mobility (FID 3.420), which recovers part but not all of the gain.

Together with the analytic 64D result, where refitting the energy reduces field error but moves the density by KL 18.3, this is consistent with the coupling being a practical concern. It does not prove that a better-tuned continuation could never succeed, and we state that limit.

We hope these additions address your concerns, and we would be glad to discuss further.

Thank you for the detailed response and additional experiments. The authors have addressed most of my concerns, and I appreciate the effort to provide more comprehensive comparisons. However, the new results also highlight the limited practical gains of REM. These results leave me concerned about whether the practical benefits of REM justify its additional complexity. Overall, I acknowledge the theoretical contributions and the improvements made in the revision, but I remain concerned about the practical significance of the reported performance gains.

AuthorsOpenReview ↗

Response to Reviewer 6QLP

Thank you for acknowledging the revisions and additional controls. We understand your remaining concern about practical significance. We view the contribution as a controlled study of transport adaptation at a fixed energy, including evidence that improved velocity fitting need not translate into better sampling efficiency. The revised paper makes the modest tuned image gains, added training cost, and mixed sampling outcomes explicit. We appreciate your careful assessment.

Reviewer_wmcUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The paper studies Energy Matching (EM), in which a single scalar energy V defines both a Boltzmann density p ∝ exp(−V/ε) and a transport field −∇V fitted to optimal-transport (OT) velocities. Because the two roles share one function, improving transport by updating V also changes the modeled density.

The authors propose Riemannian Energy Matching (REM). The energy is frozen, and a position-dependent symmetric positive-definite (SPD) mobility G_ϕ(x) with unit determinant is learned so that −G_ϕ∇V fits OT velocities through a path-integrated projection loss. Full SPD mobility is used in 2D, diagonal mobility for images, and a diagonal-plus-rank-four parameterization in higher dimensions.

On a sheared 2D Gaussian mixture, full REM greatly reduces OT field error (relative MSE 17.8 vs. 1068 for identity), speeds mode passage (75 vs. 215 steps), and lowers MMD (7.5×10⁻⁴ vs. 5.41×10⁻³). Removing only the divergence term from the same fitted mobility raises MMD to 5.09×10⁻². On a correlated 64D Gaussian, rank-four REM reduces field error by 70.3% relative to a parameter-matched diagonal model, with only a small sampling gap. On frozen CIFAR-10 and ImageNet32 EM energies, a diagonal mobility used only during the deterministic transport phase ("phase gating") improves FID from 3.462 to 3.297 and from 6.742 to 6.555 at the main solver budgets.

Critical strengths
  1. A clean, well-posed question. Asking whether transport geometry can be improved while the energy, and hence the density, stays fixed is a useful way to separate density-learning error from transport error in energy-based generative models. The 64D experiment illustrates the point: continued energy fitting attains a smaller field residual but incurs an analytic KL of 18.3 to the original density, whereas mobility models retain zero KL.

  2. Controlled experimental design. Samplers share initial states, budgets, temperatures, and reference draws, and seeds are the analysis units, with paired results shown per seed. The known-geometry target (supervising with the analytic field −G_c∇V_c) isolates geometric recovery from OT-fitting noise: full REM recovers the metric with relative error 0.00167, against 0.748 for diagonal mobility. The divergence ablation reuses the identical fitted mobility, and step refinement with fixed analytic geometry rules out discretization: the uncorrected MMD stays near 0.018 across all four step sizes, against 1.00×10⁻³ corrected at Δt = .001. Together, these cleanly separate fitting, correction, and numerical error.

  3. Representability construction. The determinant-one mobility in Eq. 5 maps the energy gradient exactly to the target velocity and has unit determinant. The paper also states the descent obstruction clearly and measures it (21.7% of held-out OT chords violate the descent condition).

  4. Low overhead at image scale. The 58k-parameter image mobility adds about 2.0% time and 2.9% memory at 650 energy-gradient evaluations. Code is provided.

Critical weaknesses

1. The image gains do not use the paper's central mechanism. The fixed-density guarantee (Prop. 3.2) concerns the corrected stochastic diffusion. On images, however, the learned mobility is active only during the deterministic phase, where the flow −G∇V carries no density-preservation guarantee, and the stochastic refinement uses G = I, which is standard EM.

When the corrected Riemannian diffusion is kept active during stochastic refinement (Table 22), FID is worse than phase gating: 9.018 vs. 7.259 (one fit, one seed, 10k samples). The image result is therefore better described as learned diagonal preconditioning of EM's deterministic ODE phase. That is a narrower claim than the "fixed-density transport" framing of the title and abstract.

2. The theoretical contributions are largely classical. Prop. 3.2 and Cor. 3.3 restate the invariance and reversibility of position-dependent preconditioned Langevin dynamics. Prop. 3.1 is the standard gradient-like vector field characterization, which the paper cites (Borodzik et al., 2016), here with an explicit determinant-one formula. Presenting these as "representation and sampling guarantees" overstates the theoretical novelty.

3. The key density-preserving alternative is missing. Ma et al.'s complete recipe allows the drift −(G + Q)∇V, plus the corresponding correction, with Q skew-symmetric. Any skew Q also preserves exp(−V/ε), introduces rotational motion, and is known to accelerate mixing through irreversible dynamics, which the paper cites. REM uses only the SPD part.

The "free residual" control in Table 1 is an unconstrained additive field that breaks the target law by construction, so it is not the appropriate comparison. A learned skew-symmetric perturbation, or the full SPD-plus-skew recipe, preserves the density and can absorb the rotational components that motivate the comparison with PEM.

4. The stochastic sampler depends on unidentified components of G. OT supervision identifies only the product G∇V, which gives d constraints, while G has d(d+1)/2 − 1 degrees of freedom. The authors acknowledge this. But the noise covariance 2εG and the divergence correction depend on the full matrix, including directions fixed only by the regularizer λ_geo. It is therefore unclear why fitting OT velocities should yield a better stochastic sampler, and the evidence is mixed: i) a clear gain in 2D ii) no consistent advantage in 16D despite a 20.7% lower field error iii) a small gap in 64D (MMD 0.0135 vs. 0.0141) iv) worse results on images when the stochastic phase uses the learned geometry.

5. Sampling efficiency is not shown per unit compute. In the 2D mixture, ESS rises from 660 to 1351, but 8,000 steps take 51.45 s for corrected full REM vs. 3.28 s for identity, roughly 15× slower. Per second, corrected REM is less efficient. The paper limits its claims to matched-step gains, but the abstract's "improve sampling" reads more broadly. The cost of divergence correction at image scale is not measured.

6. The image improvements are small, budget-dependent, and lack solver-tuning controls. The gains are about 0.17 FID on CIFAR-10 and 0.19 on ImageNet32. They hold at 100 and 650 energy-gradient evaluations, are inconsistent at 200 (2/3 wins), and reverse at 400 (0/3 wins). At 400 evaluations, a different cutoff (τ_g = 0.5) would have been preferable (Table 21).

Constant SPD mobility already reaches 3.420, so the gain attributable to position dependence is roughly 0.12 FID. No schedule-tuned or step-size-tuned identity baseline is reported, although retuning EM's time discretization could plausibly produce comparable gains. KID is not reported for ImageNet32.

7. The controlled targets align closely with the model class. In the curved 2D target, the true mobility is an exact determinant-one SPD matrix quadratic in x, so it lies in REM's hypothesis class. The headline illustration (Figure 2) supervises REM with this analytic field. With ordinary OT supervision, a clear sampling advantage appears only in the 2D setting, which limits how far the controlled results generalize.

Questions for authors

Please address the weaknesses listed above, in particular the following.

  1. Since the learned mobility is active only during the deterministic phase on images, can you clarify which results actually rely on the fixed-density guarantee? Would you reframe the image contribution accordingly?
  2. Can you add a skew-symmetric (complete-recipe) baseline, −(G + Q)∇V with Q antisymmetric, which also preserves exp(−V/ε)?
  3. Can you report a schedule-tuned or step-size-tuned identity EM baseline, and the cost of the divergence correction per unit compute?
  4. It would strengthen the paper to compare against, or at least discuss, the setups of recent flow-matching and energy-optimization methods that learn geometry or non-conservative transport. Examples include GE-FM (https://openreview.net/pdf?id=Wb9wwUF7aV), metric flow matching (Kapuśniak et al., 2024), Projected Energy Matching (Barco et al., 2026), and Flux Matching (Pao-Huang et al., 2026).

I am looking forward to revising my score based on the author's response and help the authors in enhancing the paper to reach acceptance.

3 comments
AuthorsOpenReview ↗

Response to Reviewer wmcU (1/2)

Thank you for the detailed review and for separating representation, invariant-law sampling and finite-time image generation. We agree these need separate claims, and we have run the controls you asked for. Shared image results are in the general response.

1. What relies on density preservation? The invariant-law statement applies to the corrected stochastic diffusion under its regularity and boundary assumptions. The phase-gated image sampler uses learned mobility only during deterministic transport. Its Euclidean refinement does not make the preceding ODE density-preserving. We agree with your description: the image result is learned diagonal preconditioning of EM's deterministic phase. The revised abstract, results and discussion now say this. The fixed-density guarantee is tested by the controlled corrected-diffusion experiments, not by image FID. "Fixed density" in the title refers to the frozen energy, which all image runs keep, but we are happy to adjust the title if you find it overstates the image claim.

You are right that the ungated corrected sampler in Table 22 is worse than phase gating (9.018 vs. 7.259). That comparison changes where geometry is active and runs inside the historical schedule. We have not isolated the cause of the gap. To test the stochastic phase on its own, we started refinement from identical saved boundary states, produced by an identity deterministic prefix, over a registered 20-condition grid. The grid crosses Euler–Itô vs. diagonal Heun, 50/100 steps, mobility strength 0.25/1, and 1/4 divergence probes, plus identity controls. All conditions completed. At 100 Heun steps with one probe, full-strength corrected refinement gives FID 6.163 vs. 6.581 for identity, and every Heun condition improves on identity. Some Euler conditions do not improve (e.g., 50 steps at strength 1: +0.04). This is a one-fit, one-stream, 10k-sample diagnostic, and corrected refinement costs more (≈1.5× with one probe). It does not replace the 50k protocol or demonstrate stationary sampling. The complete grid is in Appendix E.5 (Table 31).

2. Theoretical novelty. We agree that position-dependent Langevin invariance and reversibility are classical, and that the gradient-like representability characterization has prior foundations. The revised abstract, introduction and contributions now present these as established foundations. Our contributions are the explicit determinant-one construction, the bounded model classes, the frozen-energy learning formulation and the controlled evidence. We do not claim a new general invariance principle or infer improved mixing from it.

3. Lawful skew and complete-recipe controls. We added three controls: constant skew, learned state-dependent skew, and joint SPD+skew. All use the same frozen energy and the same ordinary minibatch-OT supervision. For A⊤=−AA^\top=-A,

dX=[−(G+A)∇V+ε Div(G+A)]dt+2ε B dW,BB⊤=G.dX=\big[-(G+A)\nabla V+\varepsilon\,\mathrm{Div}(G+A)\big]dt+\sqrt{2\varepsilon}\,B\,dW,\qquad BB^\top=G.

The skew divergence is included, and AA does not enter the noise covariance. Float64 audits confirm the row divergence (error <6e−11) and the noise covariance. On a known Gaussian, omitting Div A\mathrm{Div}\,A or reversing its sign produces a large bias that persists at smaller steps, while the exact correction does not. We evaluated five training seeds in 2D and 16D and retained every registered cell, including four coarse-step 2D burn-in failures.

Held-out relative OT velocity MSE (mean ± SD, 5 seeds):

Arm2D, curv. 02D, curv. 1.216D
identity / const. skew64.88 ± 1.81100.6 ± 1133.565 ± 0.034
state skew64.82 ± 1.81100.4 ± 1133.561 ± 0.034
SPD (REM)14.58 ± 0.4419.46 ± 2.21.644 ± 0.017
joint SPD+skew15.52 ± 0.5124.49 ± 1.51.668 ± 0.015

The bounded 16D skew model uses four fixed planes, so these numbers do not rank unrestricted classes. Sampling does not show a matching general SPD advantage. At the finest registered 16D step, MMD² is 0.00409 (state skew), 0.00473 (SPD) and 0.00479 (joint), and the paired intervals include zero. In equal-wall-time (3 s) stationary-start estimator tests, identity or the cheaper skew dynamics often give lower MSE. Skew drift is tangent to the energy level sets, so it cannot change the descent component of the velocity. In practice, it barely moves the OT residual (64.82 vs. 64.88 in 2D), which is dominated by that component. All of these results are in Appendix E.3 (Tables 27–29) and Section 4.5, and the discussion now cites these lawful controls as the main density-preserving comparison.

AuthorsOpenReview ↗

Response to Reviewer wmcU (2/2)

4. Unidentified components of the mobility. We agree that in dimensions d≥3d\ge 3, fitting G∇VG\nabla V does not identify the full determinant-one SPD matrix. The noise and divergence terms depend on the unidentified completion, so a lower velocity residual is not by itself evidence of a better stochastic sampler. We ran a five-seed 16D regularizer study over {10−5,10−4,10−3}\{10^{-5},10^{-4},10^{-3}\} and two step sizes, retaining all 30 registered cells (0 failures). Discrepancies change little over this local range, which does not establish identifiability or global mixing. The new skew/control chains also fail a stringent multi-chain convergence check at the registered durations (no group reaches R^≤1.01\hat R\le1.01 on all seeds). Their short-chain ESS values are therefore descriptive only. Appendix E.4 and the Discussion now state this unresolved sampler-design problem explicitly. The 2D case differs: at nonzero gradient, an exact full action with determinant one is pointwise unique.

5. Efficiency per compute. We agree that the historical matched-step improvement does not establish a wall-clock benefit, and the new equal-wall-time tests do not support a general efficiency claim. A synchronized H100 image profile at batch 64 gives the following per-step costs:

Identity Euler stepCorrected, 1 probeCorrected, 4 probes
Total step (ms)≈30≈44≈83
Divergence part (ms)—≈14≈53

These stochastic costs are separate from the small overhead of phase gating, which raises the 50k sampling time by only 0.4–0.9%. Training one image mobility fit takes ≈11.7–12.9 H100-hours, including a logged restart. Appendix E.6 (Table 33) reports fit training, validation search, sampling and metrics separately rather than treating inference overhead as the total cost.

6. Tuned identity baseline and budget sensitivity. The equal-search-budget CIFAR comparison is in the general response. REM keeps 3/3 paired wins at both 400 and 650 NFE, but after tuning its mean FID advantage is only 0.041 and 0.026. The revised abstract, Section 4.5 and conclusion foreground this smaller gain. Appendix E.1 reports all paired differences, and the original 400-NFE reversal stays in the paper. The results show an interaction with the time/noise schedule. At 400 NFE, the equal search independently selects onset/cutoff 0.5, consistent with your reading of Table 21. These results do not identify a unique cause of the earlier reversal or establish budget-independent gains. We also agree that under the original schedule constant SPD already reaches 3.420, so position dependence accounts for about 0.12 of the 0.165 gain. We did not rerun constant SPD under the new tuning. We also added fresh ImageNet32 KID computed from per-image features: the mean paired difference is −0.00145 at the historical 500-NFE setting.

7. Controlled model class and related work. The analytic-field illustration is a recovery test within the model class, not evidence of broad superiority. The new lawful controls use ordinary minibatch-OT supervision, and their mixed sampling results reinforce the distinction between field approximation and sampler performance. The paper keeps the limitation that practical benefits of non-diagonal image geometry are unestablished. On related work:

  • Metric Flow Matching learns a metric that shapes interpolation paths and a time-dependent flow.
  • GE-FM learns a time-dependent metric induced by kinetic plus learned potential energy, with Christoffel-adjusted flow on non-Euclidean manifolds.
  • Flux Matching matches probability-flux divergence and can select nonconservative stationary dynamics.
  • PEM v1 absorbs rotation in an auxiliary residual while updating the energy.

REM differs from all four: it keeps an ambient scalar energy, and hence its Boltzmann law, fixed and learns only a reversible SPD mobility on its gradient. Related Work makes this comparison and ties PEM to v1. We will add the GE-FM citation in the final version.

We hope these additions address your concerns, and we would be glad to clarify anything further.

Reviewer_wmcUOpenReview ↗

Final response

Thank you for the thorough response and the new controls. The revised framing resolves my concerns about overclaiming: the image result is now described as deterministic-phase preconditioning, the invariance and representability results are presented as established foundations, and the efficiency claim is limited appropriately. The skew-symmetric and complete-recipe controls are exactly the comparison I asked for.

However, the new results narrow the demonstrated benefit further. After tuning the identity baseline, the CIFAR-10 gain is 0.041 FID at 400 NFE and 0.026 at 650 NFE, and constant SPD was not rerun under the new tuning, so the contribution of position-dependent mobility under tuning is unknown. In sampling, state-dependent skew attains the lowest 16D MMD with paired intervals including zero, identity or skew dynamics are often better at equal wall time, the new chains do not pass the multi-chain convergence check, and the identifiability of the stochastic sampler remains unresolved. The corrected-refinement improvement comes from a single fit, stream, and 10k-sample screen.

The paper now accurately states its scope, which I appreciate. But its demonstrated contribution is mainly improved OT field fitting under a frozen energy, which has not yet translated into sampling or generation gains of practical size. I therefore maintain my score, and encourage the authors to pursue the equal-wall-time sampling and tuned image comparisons further in a future version.

Reviewer_vLp5OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper Riemannian Energy Matching (REM) addresses a key limitation of energy-based models. The authors characterize the representable velocities and derive a divergence-corrected diffusion that provably preserves the original Boltzmann density in continuous time. Experiments show that full mobility captures motion diagonal models miss on curved multimodal targets, that removing the divergence correction substantially increases sampling error, and that rank-four mobility cuts held-out velocity error by 70.3% in 64 dimensions at matched parameter count. For image generation, a phase-gated strategy using learned geometry during deterministic transport and Euclidean dynamics during stochastic refinement improves FID in all nine paired evaluations per dataset on CIFAR-10 and ImageNet32.

Critical strengths
  1. the text is well written and easy-fo-follow
  2. the interesting idea lies at the heart of the method, that I have never seen before
Critical weaknesses
  1. This method is propelled with images 32x32. It seems that ti is slighlty insufficient to detect robustness of the method.
  2. There is no comparison with other energy-based models with OT as Mokrov et al "Energy-Guided Optimal Transport" (ICLR 2024)
  3. Problems with parametrization: When the target velocity becomes nearly perpendicular to the energy gradient, a key denominator in the construction approaches zero, causing the matrix to "blow up" and making training highly unstable. Also, it seems, that there is a problem with degenerate points. At critical points of the energy (where the gradient is zero), the invertible mobility cannot produce any movement. This requires delicate and complex handling to maintain smoothness across these regions.
Questions for authors
  1. Does this method work for more high-dimensional data as 64x64 and more to bettter detect improvements?
  2. Could you compare your work with Mokrov et al "Energy-Guided Optimal Transport" (ICLR 2024) in image experiments?
  3. The text states that a bounded network provides a restricted approximation. How do you enforce the determinant-one and SPD constraints within a neural network architecture? Do you use a specific parameterization (e.g., matrix exponential, Cholesky decomposition) to guarantee these properties during optimization?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vLp5 (1/1)

Thank you for pointing out the resolution limitation and for asking for a more explicit description of the parameterization. Shared new results are in the general response. We address each point below.

Higher resolution. Our image experiments cover CIFAR-10 and ImageNet32 only. We agree they do not establish robustness at 64×64 or above. The revision states this limitation explicitly (Discussion; Appendix E.8). The publicly released Energy Matching checkpoints cover CIFAR-10, ImageNet32 and AAV. A native 64×64 study would therefore require training a new energy model, which we could not do properly within the rebuttal period. The new experiments address solver tuning, corrected refinement and computational cost on the existing backbones. We do not present them as higher-resolution evidence.

Comparison with Mokrov et al. (Energy-guided Entropic Neural OT, ICLR 2024). That method learns an entropic transport plan between source and target distributions, with conditional density ∝exp⁡((f(y)−c(x,y))/ε)\propto \exp((f(y)-c(x,y))/\varepsilon). Its high-resolution AFHQ experiments perform unpaired translation in a pretrained StyleGAN latent space. REM instead freezes a scalar energy and learns a mobility acting on its gradient. Both methods connect energies and transport, but they learn different objects and use different image protocols. A numerical comparison would require a common task, representation and compute budget. Its translation FID is not directly comparable to our unconditional-generation FID. We have not run a full EOT image reproduction and do not imply otherwise. The revision cites this work in Related Work and compares the two settings in Appendix E.8.

SPD and determinant constraints. Both constraints are enforced by construction rather than by a penalty. For images,

ai(x)=αtanh⁡hi(x),si(x)=ai(x)−1d∑jaj(x),G(x)=diag(esi(x)).a_i(x)=\alpha\tanh h_i(x),\qquad s_i(x)=a_i(x)-\tfrac1d\textstyle\sum_j a_j(x),\qquad G(x)=\mathrm{diag}(e^{s_i(x)}).

Hence GG is SPD and det⁡G=exp⁡(∑isi)=1\det G=\exp(\sum_i s_i)=1 in exact arithmetic. With α=1.5\alpha=1.5, the condition number is at most e2α≈20.09e^{2\alpha}\approx 20.09. To answer your question directly: we use matrix-exponential (log-space) parameterizations, not Cholesky factors. The full-matrix model computes G=exp⁡(S)G=\exp(S) for a bounded, symmetric, trace-free SS, which again gives SPD and det⁡G=etrS=1\det G=e^{\mathrm{tr}S}=1. The structured model bounds its low-rank correction so that its congruence factor stays nonsingular. The revision states these bounds explicitly (Appendix B.1, E.6) and notes in Section 3.2 that the network never evaluates the inverse construction.

Nearly orthogonal velocities and critical points. You correctly identify a limitation of the pointwise inverse construction. Let g=∇Vg=\nabla V and v=−Ggv=-Gg. When g≠0g\neq 0, strict descent g⊤v<0g^\top v<0 is necessary, and the constructive inverse can become unbounded as the descent margin vanishes. This inverse appears in the existence proof, not in the network's forward pass. The trained bounded network never divides by this margin. Instead, it incurs approximation error on velocities outside its bounded class. At g=0g=0 the deterministic velocity is exactly zero, so nonzero targets cannot be fitted there. Positive-temperature diffusion can still move through noise and the correction drift. That does not imply rapid escape or an arbitrary smooth deterministic extension.

New held-out diagnostics use 24,576 OT pairs per dataset and three mobility fits:

DatasetIdentity rel. MSEREM rel. MSEMax. condition numberSaturated coord. fraction
CIFAR-100.12960.1189–0.119015.98–16.910.36–0.70%
ImageNet320.17560.1563–0.156418.58–18.760.06–0.36%

The learned mobilities remain finite and well inside the bound, with little saturation. These finite checks do not remove the representational limitation. Appendix E.6 reports this table, and we state these restrictions rather than claim universal stability.

Reviewer_vLp5OpenReview ↗

Answer

Thank you for answers, I raise my score,

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their careful reading. We have uploaded a revised PDF. All changes are marked in blue, and all new experiments are collected in the new Appendix E. The main text is now 10 pages, within the revision limit, and includes a new Section 4.5 summarizing the review controls. A shared concern is the distance between the invariant-density guarantee and the image results. We agree it should be stated more precisely. The guarantee applies to the corrected continuous-time diffusion. The image experiments instead use learned preconditioning during deterministic transport, followed by Euclidean stochastic refinement. They establish a finite-time generation result, not faster mixing under a learned Riemannian image diffusion. The revision makes this scope explicit in the abstract, results and discussion. We also agree that the invariance of position-dependent Langevin dynamics is classical. Our contribution is supervised transport adaptation over a frozen energy, together with the empirical consequences and limits of that intervention.

New image results. All image results use fresh 50k-sample evaluations. Each comparison has three paired runs: three independently learned mobility fits, one fresh sampling stream per fit, and the same frozen identity (EM) model on the paired streams. This design does not separate fit variation from stream variation. Δ\Delta = method − identity, so lower is better. KID is unscaled.

DatasetProtocol / NFEEM FIDREM FIDPaired ΔFID (SD)Paired ΔKID
CIFAR-10Equal search / 4003.4313.390−0.041 (0.005)−6.7e−5
CIFAR-10Equal search / 6503.4103.384−0.026 (0.024)−7.3e−5
ImageNet32Historical / 5006.7626.530−0.232 (0.094)−1.45e−3

Every row has 3/3 paired FID wins. In the CIFAR search, each arm gets the same 12 terminal-time × noise-onset candidates. Each arm's top two candidates are confirmed on a second disjoint 10k stream, with selection by mean KID and FID as the tie-break. Configurations are locked before final evaluation on fresh streams. ImageNet32 keeps its original 500-NFE schedule and is not schedule-tuned. Sampling time ratios (REM/EM) are 1.004–1.009.

Matched extra training of the energy (CIFAR-10, fixed original 650-NFE configuration).

EMREMContinued EM
FID (3 fresh paired runs)3.4843.3154.309
KID0.0009310.0007500.001362
Rel. OT velocity MSE0.13010.11930.1365

Continuation restores the official full checkpoint, optimizer, scheduler and EMA. It trains with the original transport + contrastive objective for the same logged H100 time as one mobility fit (12.67–12.85 h). The sampler configuration is held fixed, and we did not tune a separate sampler for continuation.

Other new controls (details in individual responses).

  • Lawful complete-recipe skew and joint SPD+skew baselines in 2D and 16D, with 5 seeds and every registered cell retained (wmcU).
  • Refinement from identical saved deterministic boundary states over a 20-condition grid (wmcU).
  • Synchronized per-component cost profile and a full training-cost ledger (wmcU).
  • Regularizer-sensitivity study for the unidentified mobility components, 30 cells (wmcU).
  • Held-out mobility stability diagnostics on 24,576 OT pairs per dataset (vLp5).
  • Independent AAV confirmation, 3 fits × 5 fresh streams (6QLP).
  • Fresh ImageNet32 KID from per-image features (wmcU, 6QLP).

Changes in the revision (blue). The abstract, contributions and conclusion now report the smaller tuned gain and state that the invariance result is classical. Section 4.5 (Table 4) adds the tuned and continuation results, and Section 4.4 keeps the original 400-NFE reversal. The discussion separates field-fit, sampler-correctness and wall-clock claims. Appendix E.1–E.8 contain every new table and the complete grids, and the resolution limitation (CIFAR-10 and ImageNet32 only) is stated explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.