acceptodds
Under review as a conference paper at ICLR 2027

Guiding Diffusion Samplers with Black-Box Optimization Solvers

Abstract

From robot motion planning to scientific inverse problems, generative tasks often require optimizing target objectives while adhering to a complex data-driven prior. Diffusion models excel as such priors, but guiding them to optimize targets or obey constraints remains challenging, even with powerful optimization solvers available. When integrated into diffusion sampling, these solvers often distort the sampling trajectory with their hard projections. We propose Proximal Diffusion Guidance (ProxDG), a training-free method that cleanly integrates solvers as soft guidance during diffusion sampling. Specifically, ProxDG guides samples using the gradient of a time-varying Moreau envelope. This envelope smooths the original objective without shifting its optima, providing well-behaved guidance even under hard constraints. Importantly, computing this guidance reduces to a proximal subproblem that admits powerful, problem-specific solvers. With this proximal scheme, ProxDG samples along a continuous probability path to the target posterior, supporting both stochastic and deterministic sampling with exact constraint enforcement at the final step. We demonstrate in both synthetic and real-world experiments that ProxDG effectively leverages off-the-shelf solvers to significantly outperform training-free baselines.

Primary area:
generative models
Keywords:
Proximal methods, Diffusion models, Generative models, Constrained sampling, Black-box solvers, Posterior sampling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XSd8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ProxDG, a Training-Free Guidance (TFG) method that integrates Gradient Guidance and Solver-Based Guidance. The core of the paper is to compute a guidance direction for each xtx_t based on the Proximal Operator and the Moreau Envelope, and inject it into the denoising process. In the practical algorithm, for different tasks, dedicated solvers for the corresponding Proximal Operator can be designed to improve efficiency. In addition, the authors introduce an Annealing Schedule for the guidance term and provide extensive theoretical analysis of ProxDG. Finally, the authors conduct experiments on PDE Solution Generation, Scientific Inverse Problems, and Robot Motion Planning to verify the performance of the algorithm.

Critical strengths
  1. The algorithm is simple and easy to understand, and it combines Gradient Guidance and Solver-Based Guidance into a single algorithmic framework.
  2. The theory is sufficient, and sufficient theoretical analysis is provided for the theoretical properties of the algorithm.
Critical weaknesses
  1. The computational cost is high. Because of the Proximal Operator, a minimization problem has to be solved at every step. For certain problems, closed-form solutions may exist, but for general problems, this means that every step requires an expensive solve.
  2. The organization of the appendix is not appropriate. Since the appendix contains a large amount of theoretical analysis, it would be more reader-friendly to organize it better: for example, providing a table of contents for the appendix, and explaining the order in which the appendix is presented, the main theoretical results in each part, and the proof tools.
Questions for authors
  1. Since this paper presents a unified framework, I would like to know how DSG [1] should be understood within this framework.

[1] Yang, L., Ding, S., Cai, Y., Yu, J., Wang, J. & Shi, Y. (2024). "Guidance with Spherical Gaussian Constraint for Conditional Diffusion." Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, 235:56071–56095.

1 comment

I thank the authors for patiently answering my questions and for carefully revising the paper. Overall, the paper has become noticeably more readable. In addition, the authors' discussion of the computational cost has partly convinced me. Finally, the authors' discussion of DSG exceeded my expectations. I initially thought that DSG would be a special case of ProxDG, specifically analogous to imposing a spherical constraint, since DSG itself appears to derive a spherical constraint based on a Gaussian distribution and projects the distribution onto a sphere. The authors' discussion, however, has revealed another possibility. For these reasons, I choose to maintain a positive evaluation of the paper.

Reviewer_jfsnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Proximal Diffusion Guidance (ProxDG), a training-free approach for incorporating black-box optimization solvers into diffusion sampling. The key idea is to replace the original task objective used in gradient guidance with its Moreau envelope, whose gradient can be computed from a proximal solution without differentiating through the solver. Experiments on PDE-constrained generation, scientific inverse problems, and robotic motion planning demonstrate improvements over existing guidance methods.

Critical strengths
  1. The proposed method is simple and effective, and well-supported by the theoretical foundation. Basically, I believe this is an elegant solution.

  2. Solid empirical evidence, showing that the proposed method takes a consistent lead over existing methods in various domains.

Critical weaknesses
  1. I do not see a clear weaknesses as I am not so familiar with the tasks targeted by this work. But the reliance on the external solver could be a potential concern.
Questions for authors
  1. What is the computational cost for using a solver, compared with prior solver-free methods?

  2. How sensitive is the model performance to the solution quality (like not solving the problem to the optimum when the exact solving is expensive)?

Reviewer_9YpqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes Proximal Diffusion Guidance (ProxDG), a method to integrate black box optimization solvers into diffusion guidance. This work achieves this using smoothing operators such as the Moreau envelope. At each step, it solves a proximal problem centered at the denoiser’s clean estimate, then backpropagates the proximal residual through the denoiser. Unlike methods that directly substitute the proximal solution into sampling, ProxDG uses it to construct a guidance gradient. Experiments are comprehensive and cover constrained PDE generation, scientific inverse problems (black hole imaging, full-waveform inversion), MRI and robot motion planning.

Critical strengths
  1. Wide coverage of inverse problems from PDE solving, black hole imaging, full-waveform inversion, motion planning to MRI.
  2. Integrating task-specific solvers without needing to back propagate through their internal workings.
  3. The paper is generally well-written and easy to follow.
  4. The method that is proposed subsumes previous posterior sampling techniques.
Critical weaknesses
  1. While the paper has many strengths (shows very good experimental results, is well-written), upon several readings I still found it hard to find a distinguishing feature that separates this paper from prior works. That is, this paper seems to be a combination of well-known techniques that include optimization with smoothing operators such as the Moreau envelope, gradient based guidance that has a manifold projection property. Kindly provide more context in positioning the paper better against related work. In my opinion, the three contributions mentioned are not sufficient in clarifying the novel contributions of this work.

  2. Fig. 1 is focused on showing how smoothing along with annealing can help reach the MAP solution, I feel the paper is more aimed towards true posterior sampling. While the figure is useful for illustrating the effect of smoothing, it does not align well with the rest of the text. Could the authors try and modify this figure to reflect more of what the rest of the paper is focused on?

  3. The method is likely very computationally expensive (acknowledged by the authors in Sec 6). Are there particular settings or domains/problems where this may be permissible or desired? If so, could the authors kindly add a section to discuss these scenarios and trade-offs?

Questions for authors

Main question: Appendix D establishes the convergence of the proposed smoothed density path to the target posterior, but in Appendix E, the authors acknowledges that the guided dynamics need not follow this path (Notes). What controls the resulting gap in Appendix G to avoid bias? Does this gap vanish as the smoothing parameter and integration step size approach zero, assuming exact scores and proximal solves? If not, should the posterior-convergence claim be restricted to the ideal density path rather than the implemented sampling algorithm described in the paper?

Please see other questions in the Weakness section.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Paper Revisions

The reviewers provided valuable feedback, which we have incorporated into the revised submission. The main changes are highlighted in blue. Section, table, and figure numbers below refer to the revised submission.

Changes in response to reviewers

  • Positioning and contributions (9Ypq). The contributions in Section 1 make the same three points as before, but now also position ProxDG against related work. Section 2 briefly compares ProxDG against prior gradient and proximal guidance methods. Appendix B.1 extends this comparison to other plug-and-play samplers, proximal sampling, and guidance theory. Appendices J.2.2, J.3.2, J.4.2, and J.5.3 compare each of our solvers with existing solvers for its domain. The bibliography grows accordingly.

  • Figure 1 (9Ypq). Figure 1 now illustrates posterior sampling rather than convergence to the MAP.

  • Computational cost (9Ypq, jfsn, XSd8). Sections 5.2 and 5.3 report and analyze solver costs more rigorously. Our rewritten discussion section (Section 6) covers when stronger solves pay off. Table 20 now times all eight motion-planning methods on all six environments, whereas the original table covered four methods on two environments.

  • Theorem 4.4 (9Ypq). The informal statement now separates the sampler's error from the convergence of the ideal path. The formal results are unchanged.

  • DSG (XSd8). Figure 26 adds a black hole imaging experiment that combines ProxDG with DSG.

Experiments

  • Few-step PDE sampling (Table 11). We added PCFM and PDM baselines at 4x fewer sampling steps. We also corrected the ProxDG rows. In the original submission, we had unknowingly tuned them with the final projection turned off. The revised rows following the tuning procedure of Appendix J.1.2, and ProxDG now attains the best MMSE on all four 1D tasks.

  • Motion planning (Tables 3 and 19, Figure 29). We added the SafeDiffuser results on Spheres3D and Warehouse, which the original submission omitted because of their long run time. Figure 29 replaces the solver-strength tables. It adds a fourth environment and the PDM and CPS baselines, and it retunes the guidance hyperparameters of every method at each solver budget.

Corrections

  • Section 4.3 now states that the guided SDE is unbiased only if both ft=ft∗f_t = f_t^\ast and w=1w = 1.

  • Section 1 now defers the effect of the final projection on the sampled distribution to Theorem 4.4, which states it precisely.

Changes that are not highlighted. The highlighted copy marks only the main changes to the presentation and results. It does not mark the small edits we made throughout the paper to improve clarity, correct grammar and typos, unify notation, and fit the new material in 10 pages.

Thanks to all three reviewers for engaging with us during the discussion period. We appreciate that they found ProxDG to be a simple and elegant method with strong theoretical support, which unifies gradient and solver-based guidance. We also appreciate that they recognized its consistent lead over existing methods across a wide range of domains.

We highlight three specific ways we engaged with reviewers and strengthened our paper:

  1. Novelty: We clarified our contributions in Sections 1 and 2 and new Appendix B.1. To our knowledge, ProxDG is the first guidance method to take the envelope of a general nonsmooth objective or hard constraint as its guidance function. This yields a continuous probability path to the posterior and well-posed SDE and ODE dynamics, even for nonconvex constraints of zero prior mass. 9Ypq wrote, "I now see where your contribution lies," and raised their score.

  2. Solver cost: New cost analyses (Sections 5.2, 5.3, 6) show that the solver acts as a dial between cost and accuracy. On FWI, ProxDG samples 3× faster than the fastest baseline (DPS) and leads on every metric. jfsn confirmed that they have no remaining concerns.

  3. Guidance Tools: Unlike prior proximal methods, ProxDG works with guidance tools like DSG, so we tested their combination on black hole imaging. We found that ProxDG + DSG more than triples ProxDG's Blur PSNR margin over the best baseline at every observation ratio, and more than doubles its PSNR margin at the 10% and 100% ratios (Figure 26); XSd8 said these results "exceeded my expectations." As DSG illustrates, existing guidance techniques plug easily into ProxDG.

We also revised Figure 1 and Theorem 4.4, reorganized the theory appendices, and ran additional baselines in our few-step PDE, motion planning, and solver-strength experiments.

If our responses have resolved the reviewers' concerns, we would be grateful if they would raise their scores accordingly. We are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.