- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents a constrained-optimization solver built on diffusion models, driven by two gradient flows: the Wasserstein gradient flow and the Fisher-Rao gradient flow. Given a fixed diffusion model, the authors compute the update direction for the score field;. The paper also introduces a Dual Network to adaptively adjust penalty for constraints during training. Finally, this paper conduct experiments on several nonconvex problems and ablation studies on gradient flows.
Critical strengths
- The Wasserstein gradient flow and the Fisher-Rao gradient flow are applied to a diffusion-based constrained-optimization solver at the same time. This is a direction I had thought about and explored before, and the authors have completed it here.
- The writing is clear. The paper involves a lot of concepts, but they are strung together in a sensible order; the propositions come with proofs, and the implementation methods are efficient.
Critical weaknesses
- The experiments are not enough. As a constrained-optimization solver, the paper is missing some classical optimization problems. The Wasserstein gradient flow can also be disturbed by unbounded gradients. DiOpt [1] gives one example, concave quadratic optimization: its objective goes to negative infinity at infinity, and this may mislead methods that rely on reward gradients.
- The paper mentions a Self-Distillation step. Its role is to keep the velocity field canonical, or in other words, close to the correct score field. But the paper provides no experiment showing that this Self-Distillation is necessary. There are works in other fields that follow the same Fisher-Rao gradient flow paradigm, for example DiffusionNFT [2]; that paper has no Self-Distillation, and yet it still gets decent results in image post-training. (One can show that, when roughly one gradient descent step is used, the two-branch scheme in that paper is equivalent to first constructing a target direction from the gradient flow and then fitting it.)
[1] Ding, Shutong, et al. "Diffusion-Based Learning Framework for Constrained Nonconvex Optimization with Weighted Bootstrapped Refinement." Forty-Third International Conference on Machine Learning, 2026.
[2] Zheng, Kaiwen, et al. "DiffusionNFT: Online Diffusion Reinforcement with Forward Process." International Conference on Learning Representations, 2026.
Questions for authors
- If possible, could the authors run an experiment on concave quadratic optimization? If the equality constraints are indeed hard to satisfy, they could start with a problem that has only inequality constraints, such as box constraints, so that the original problem keeps a finite optimum.
- I would like the authors to add experimental evidence on whether Self-Distillation is necessary. For example, would training collapse without it?
If the authors are willing to address these two points, I would consider raising my score further.
4 comments
Thank you for your patient answers to my questions. However, there is still one remaining minor issue that I would like to discuss with you. I have conducted self-distillation experiments myself, and the results showed that discretization itself changes the output distribution (for example, few-step Euler discretization compresses the variance). In this case, repeated self-distillation may instead corrupt the original distribution and lead to instability. This seems to be a kind of inconsistency between theory and practice. I wonder whether you have encountered this problem, and how you addressed it. If you are willing to answer this question, I will raise my score.
Thank you for the insightful follow-up question. We also encountered this practical problem when implementing our algorithm.
In the literature, prior works also discussed that the iterative training on generated samples can lead to distribution corruption (e.g., variance shrinkage) and unstable training (due to the approximation error from learning with finite samples [1, 2] and the discretization error in diffusion sampling [3]).
Empirically, we find that this problem can be mitigated in a few ways. Under the same total self-distillation budget, it works better to run self-distillation as a separate periodic stage with enough steps in each stage and enough primal-dual training between stages, than to perform shorter self-distillation more frequently. Using enough steps makes the fitted distribution more accurate, while limiting the frequency allows the solution improvement to make sufficient progress between self-distillation stages. Another technique that can be helpful in some settings is adding Gaussian noise to the samples after diffusion sampling to compensate for the variance shrinkage. In our main experiments, we did not add this noise, since the variance shrinkage is not severe with the periodic self-distillation design and the method performs well without it.
We thank the reviewer again for the insightful and detailed question.
[1] Alemohammad, Sina, et al. "Self-consuming generative models go mad." International Conference on Learning Representations. Vol. 2024. 2024.
[2] Bertrand, Quentin, et al. "On the stability of iterative retraining of generative models on their own data." International Conference on Learning Representations. Vol. 2024. 2024.
[3] Sabour, Amirmojtaba, Sanja Fidler, and Karsten Kreis. "Align your steps: Optimizing sampling schedules in diffusion models." arXiv preprint arXiv:2404.14507 (2024).
I appreciate your patient reply, which helped clear up a question I had been struggling with for some time. On that basis, I personally consider this manuscript worthy of acceptance, and I have raised my score accordingly.
We sincerely thank the reviewer for the recognition of our contributions and for the constructive comments. We address each point below. We have updated the manuscript and marked the modified parts in blue.
We thank the reviewer for this suggestion. We conducted additional experiments on the concave quadratic optimization problem
\min_{x\in \mathbb{R}^d} \quad & \frac{1}{2}x^\top Qx + p^\top x,\\\\ \text{s.t.} \quad & Ax-b \leq 0,\\\\ & -1 \leq x_i \leq 1,\quad i=1,\dots,d, \end{aligned}$$ where $Q$ is a diagonal matrix and satisfies $Q \preceq 0$. We use $d=100$ and the problem parameter is the vector $b$. The results are reported in the table below. | **Method** | **Obj.** ↓ | **Viol.** ↓ | |------------|:----------:|:-----------:| | IPOPT | -58.228 | 0.000 | | MS-IPOPT | -58.620 | 0.000 | | PDL\$^{*}\$ | N/A | N/A | | DiOpt | -50.469 | 0.000 | | **DiFlowOpt** | -51.163 | 0.000 | | **DiFlowOpt (Wasserstein flow only)**$^{\dagger}$ | -51.339 | 0.001 | $^{*}$PDL diverges during training. $^{\dagger}$Diverges in the late stage of training and test results are reported at the iteration with the best validation performance. PDL diverges during training, while both DiOpt and DiFlowOpt (WFR flow) converge to feasible solutions. The diffusion-based methods do not show advantage compared to the classical solver in this setting, since the objective is quadratic and does not have a rugged landscape, making the multimodal exploration from the diffusion model less necessary. We also evaluate DiFlowOpt with the Wasserstein flow only. At its best iteration it reaches an objective comparable to the WFR flow version, but training is unstable and eventually diverges. Making the Wasserstein flow version more stable would require extra tuning. From our empirical observations, whether training diverges on this problem depends on the specific experimental settings, such as the learning rate of the dual variables, i.e., whether the constraint term can grow fast enough to pull the solutions back before the gradient of the objective term becomes too large. PDL moves a single solution point based only on the gradient at that point, and is more likely to reach the region that leads to divergence quickly before the dual variables can catch up. We have added this experiment to Appendix D.8. > On the self-distillation We thank the reviewer for the insightful question and for drawing the connection with DiffusionNFT. We agree that DiffusionNFT can be viewed as following the Fisher-Rao flow, as also discussed in [1]. Empirically, we find that without the self-distillation stage, our method still achieves decent performance, but the objective is noticeably worse as the training becomes unstable after the first few iterations (Appendix D.1 and the table below). | Self-distillation | Obj. ↓ | Viol. ↓ | |-------------------|:---------------:|:-------------:| | w/ (default) | 10.533 ± 0.885 | 0.000 ± 0.000 | | w/o | 77.183 ± 25.512 | 0.000 ± 0.000 | One recent work [2] discusses that in the DiffusionNFT loss, the velocity field can also become noncanonical, which introduces a "forward-reverse consistency force". It also shows that the two-branch training cancels that term partially while the EMA-based off-policy training further stabilizes training with restoration. In contrast, our method is trained on-policy without EMA, so the self-distillation stage plays this stabilizing role and is necessary for the training. We believe that how to stabilize training in iterative reward-based diffusion training/fine-tuning is an important question, and we plan to conduct more ablation studies on both our method and DiffusionNFT to investigate it further. Thank you again for raising this point. We hope these additional results address the reviewer's concerns, and we are happy to provide further clarification if needed. [1] https://www.cs.utexas.edu/~lqiang/notes/taylor_diffusion_nft.pdf [2] Wan, Zhenglin, et al. "DiffusionART: Efficient, Stable and Scalable Reinforcement of Diffusion Models." arXiv preprint arXiv:2610.12572 (2026).