- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Traditional Energy Matching (EM) learns both the target density and the transport field. However, changing the transport velocity also changes the target density. This may affect training when the velocity field has not been learned accurately. To address this issue, the authors freeze the energy function, keeping the target density fixed, and introduce a symmetric positive-definite (SPD) mobility matrix to modify only the transport velocity. The proposed method focuses on modifying the flow-like objective of EM. The experiments first validate the effectiveness of the method on a 64-dimensional Gaussian distribution. The authors then evaluate the method on CIFAR-10 and ImageNet32, achieving better results than EM under matched solver budgets.
Critical strengths
-
Well-motivated problem formulation. The paper identifies an important limitation of Energy Matching (EM), where the same scalar energy function determines both the target density and the transport dynamics. Consequently, modifying the energy to improve transport may also alter the learned density. The proposed separation of density modeling and transport geometry provides a conceptually appealing direction for improving energy-based generative models.
-
Solid theoretical analysis. The paper provides a characterization of the velocity fields representable by determinant-one SPD mobility matrices and establishes conditions under which the corrected stochastic dynamics preserve the original Boltzmann distribution. These results offer a clear mathematical foundation for the proposed framework and help distinguish transport flexibility from density preservation.
-
Carefully designed controlled experiments. The synthetic experiments demonstrate the benefits of full SPD mobility on curved multimodal distributions. In particular, REM substantially improves velocity fitting and mode passage compared with Euclidean and diagonal alternatives. The ablation on the divergence correction is also informative, highlighting its importance for preserving the target distribution under state-dependent mobility.
Critical weaknesses
-
Limited practical improvements over Energy Matching. Although the motivation is compelling, the improvements on real-world image generation benchmarks are relatively modest. On CIFAR-10, REM reduces FID from 3.462 to 3.297, while on ImageNet32, the improvement is from 6.742 to 6.555. These gains are consistent under the main evaluation protocol but remain small relative to the additional modeling and training complexity. Furthermore, the improvements are not robust across sampling budgets: REM performs worse than EM at 400 energy-gradient evaluations. The paper would benefit from a more comprehensive analysis demonstrating when and why the proposed approach provides meaningful practical advantages.
-
Insufficient evidence that the main geometric mechanism is effective in high-dimensional applications. A central motivation of REM is that full SPD mobility can capture coordinate interactions that Euclidean or diagonal transport cannot represent. However, the strongest evidence for this advantage comes from low-dimensional synthetic experiments. In the main image generation experiments, REM employs diagonal mobility, and the rank-four extension does not improve performance over the diagonal alternative on CIFAR-10 (FID 3.299 versus 3.297). The higher-dimensional synthetic results also show substantially smaller sampling improvements than the two-dimensional experiments. Consequently, the practical importance of learning non-diagonal transport geometry remains insufficiently established.
-
Additional complexity without a sufficiently demonstrated efficiency–performance trade-off. REM requires training an additional mobility network for 150k steps on each pretrained image energy. While the inference overhead of the phase-gated image sampler is relatively small, the paper does not provide a comprehensive end-to-end comparison that accounts for additional training, inference cost, and generation quality. Moreover, the corrected full-mobility sampler is substantially more expensive than Euclidean sampling in the synthetic experiments. A comparison based on matched wall-clock budgets or total computational costs would be more informative than relying primarily on matched solver steps or energy-gradient evaluations.
-
A gap between the theoretical guarantees and the main empirical improvements. One of the central theoretical contributions is the preservation of the original Boltzmann distribution through divergence-corrected Riemannian dynamics. However, the primary image generation experiments disable learned mobility during stochastic refinement and use it only in the deterministic transport phase. Thus, these results primarily demonstrate improvements in finite-time transport rather than the practical benefits of the proposed invariant-density stochastic dynamics. Although an additional experiment considers ungated sampling, its evidence is limited, and the corrected ungated sampler performs worse than the phase-gated alternative. Stronger evaluations of high-dimensional stochastic sampling and convergence to the target distribution are needed to connect the theoretical framework with its main empirical claims.
-
Limited experimental scope compared with the original Energy Matching framework. The original EM paper demonstrates applications beyond unconditional image generation, including protein inverse design, image inpainting, and Hessian-based local intrinsic dimension estimation. In contrast, REM mainly evaluates synthetic distributions and low-resolution image generation. While Hessian-based LID estimation is not necessarily expected to improve when the energy function is frozen, the absence of downstream conditional generation or inverse-design experiments leaves the broader applicability of REM unclear. In particular, evaluating protein inverse design or another energy-guided sampling task would provide valuable evidence that the learned transport geometry improves practical sampling efficiency or solution quality beyond standard image generation benchmarks.
-
The motivating limitation of EM is not directly validated. The paper argues that EM suffers from coupling density learning and transport optimization, but it does not directly establish how severely this coupling affects the optimization process or final generation quality. Figure 1 provides an intuitive illustration of different transport trajectories rather than direct evidence of optimization conflicts in EM. Additional experiments examining the interaction between transport and density objectives, or comparing geometry adaptation with simpler transport preconditioning and continued energy optimization, would strengthen the causal justification for REM's improvements.
Questions for authors
-
Can REM also improve the results of EM on the other two tasks, Protein Inverse Design and Hessian-based LID Estimation?
-
Table 11 shows that REM does not consistently outperform EM across different sampling budgets. In particular, REM performs worse than EM at 400 energy-gradient evaluations. Could the authors explain why the performance gains are sensitive to the sampling budget?
4 comments
Response to Reviewer 6QLP (2/2)
4. Gap between the guarantee and the image results. We agree, and the revised abstract and results now describe the image gains as finite-time preconditioning of the deterministic phase. To probe the stochastic phase directly, we started refinement from identical saved boundary states. Corrected Heun refinement with learned mobility then lowers 10k FID from 6.581 to 6.163, and every Heun condition improves on identity, at about 1.5× the stochastic cost (Appendix E.5). The earlier ungated screen (FID 9.018 vs. 7.259) differs from this boundary experiment in its prefix and activation range, so we do not attribute its gap to one cause. This is still a one-fit, finite-time diagnostic, not a convergence proof. The invariant-law mechanism itself is tested in the controlled synthetic experiments.
5. Experimental scope (Q1).
- Protein inverse design. We added an independent AAV Medium/Hard confirmation: 3 fits × 5 fresh paired streams at locked operating points, 512 candidates per run, and at most 128 unique sequences selected with the official procedure. REM − EM on the selected sets:
| Task | Fitness | Diversity | Novelty |
|---|---|---|---|
| Medium | −0.0022 | +0.071 | 0.0 |
| Hard | −0.0119 | +0.233 | −0.7 |
REM increases diversity on both tasks at a small fitness cost, and it is slower. This is a trade-off, not a uniform improvement, and the evaluation uses a learned oracle (Appendix E.7).
- Hessian-based LID. As you note, freezing V leaves its Hessian unchanged at any fixed input, so REM cannot change this estimator by design (Appendix E.8).
- Inpainting. EM's inpainting uses CelebA 64×64 with a separate, larger architecture whose weights are not publicly released. The public checkpoints cover CIFAR-10, ImageNet32 and AAV. Inpainting, like the 64×64 question from vLp5, would require training a new energy, so we state it as a limitation.
6. Is the motivating coupling real? We added the comparison you suggest: continuing to optimize the energy versus adapting geometry. Continuation restores the official full checkpoint, optimizer, scheduler and EMA, and runs the original transport + contrastive objective for one mobility fit's training time. It does not improve transport: relative OT velocity MSE rises from 0.1301 to 0.1365, while REM lowers it to 0.1193. FID also worsens, from 3.484 to 4.309 (Appendix E.2). The simpler preconditioning baseline you mention is the constant SPD mobility (FID 3.420), which recovers part but not all of the gain.
Together with the analytic 64D result, where refitting the energy reduces field error but moves the density by KL 18.3, this is consistent with the coupling being a practical concern. It does not prove that a better-tuned continuation could never succeed, and we state that limit.
We hope these additions address your concerns, and we would be glad to discuss further.
Thank you for the detailed response and additional experiments. The authors have addressed most of my concerns, and I appreciate the effort to provide more comprehensive comparisons. However, the new results also highlight the limited practical gains of REM. These results leave me concerned about whether the practical benefits of REM justify its additional complexity. Overall, I acknowledge the theoretical contributions and the improvements made in the revision, but I remain concerned about the practical significance of the reported performance gains.
Response to Reviewer 6QLP
Thank you for acknowledging the revisions and additional controls. We understand your remaining concern about practical significance. We view the contribution as a controlled study of transport adaptation at a fixed energy, including evidence that improved velocity fitting need not translate into better sampling efficiency. The revised paper makes the modest tuned image gains, added training cost, and mixed sampling outcomes explicit. We appreciate your careful assessment.
Response to Reviewer 6QLP (1/2)
Thank you for the careful review. We address your six weaknesses in order, and the two questions within them. Shared results are in the general response. All changes are in blue in the revised PDF, with new experiments in Section 4.5 and Appendix E.
1. Practical gain and budget sensitivity (Q2). We agree the gain is modest, and the new equal-search control makes it smaller still. When both samplers receive the same 12-candidate time/noise-onset search on disjoint validation streams, REM keeps 3/3 paired wins on fresh 50k streams at both 400 and 650 NFE. The mean gain is −0.041 and −0.026 FID (Table 4).
This also answers why the earlier result depended on budget. At 400 NFE, the original schedule (fixed cutoff τ_g = 1, not retuned) was a poor operating point. The equal search selects a shorter terminal time and an earlier onset/cutoff (T = 2.75, 0.5), and there the earlier loss becomes a small gain. The sensitivity therefore comes from the interaction between the learned geometry and the time/noise schedule. It is not a budget-independent loss. The original reversal stays in the paper, and the revised abstract and conclusion state the smaller tuned gain.
2. Non-diagonal geometry in high dimension. We agree. Rank-four image mobility does not improve on diagonal mobility, and the 64D gain is a field-fit result. The new lawful controls in 16D give SPD the lowest field error but no consistent stochastic sampling advantage. The revised discussion separates these representation findings from the finite-time diagonal image result we actually demonstrate.
3. End-to-end cost. We now report the complete ledger (Appendix E.6):
The most direct total-compute comparison is the continuation control. Spending the same extra training time on the original EM objective, instead of training a mobility, worsens FID from 3.484 to 4.309, while REM reaches 3.315 under the same fixed schedule.
In synthetic equal-wall-time tests, identity has the lowest estimator error. The revision therefore limits efficiency claims to matched-step synthetic gains and finite-time image gains, and makes no wall-clock sampling claim.