- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper develops a new method for reward tilting, that is, sampling from a tilted distribution defined by a pretrained diffusion model and a reward model. The method builds on the idea of the Stein Transport (Nüksen, 2024), which concerns learning an additional drift correction term to a reward-titled SDE so that no additional importance weighting is needed. Learning this transport requires solving a linear system related to a regularized RKHS objective at each sampling step, which scales with the number of particles. However, the particle system is only an approximation and there is a residual term that in principle requires importance corrections while the paper deliberately ignores it (more-or-less justified when the RKHS norm regularization strength is small). The paper also proposes a control-variate variant. The empirical study includes GMM posterior sampling, image inpainting, and protein generation from distance measurements. The proposed method is leading on the GMM tasks on several distance metrics; On image and protein tasks, the trends are mixed for the distance metrics, while the proposed method seems to have better diversity.
Critical strengths
-
The paper provides a new perspective for diffusion reward tilting, and the method is very well motivated to alleviate the issue of importance weight collapse.
-
The derivation is mostly easy to follow (assuming background on Stein Transport)
-
The proposed method seems computationally tractable, and provides a nice alternative to Drift Lite where a basis needs to be specified.
Critical weaknesses
(i) the novelty and I wonder how much is it different from just applying the original Stein Transport to this diffusion model reward tilting setting. If that's the case, the authors should make it more clear in the writing. The paper also misses a background in Stein Transport which the current method is heavily developed on.
(ii) practical relevance: the proposed method does not uniformly improve over baselines across metrics; diversity seems to be the only highlight. But note that the paper does not really use importance sampling so that itself is less surprising.
(iii) No detailed computational cost comparison to other methods
(iv) No detailed discussion on the choice of kernel and its impact.
Questions for authors
-
Can the author discuss the choice of the kernel for different tasks? What is a good criteria of the kernel?
-
Is computing the kernel for high-dimensional data and across many particles a computational concern?
-
What is a good guiding principle to choose N, the number of particles? What would you expect the performance scaling with N?
-
Have you considered accounting for the neglected residual with the importance weights and how would the resulting algorithm perform?
-
Another note on practical relevance: Why limit comparison to methods that are designed to be consistent? For GMM posterior sampling task, it makes sense. But for image inpainting or protein generation, the metrics listed do not necessarily align with consistency. Would the authors consider some other baselines on these tasks, in particular, the one without important sampling or SMC, and how would the proposed method perform against them?
-
How many mixture components are there in the GMM experiment?
-
In GMM experiment Fig 1, the qualitative difference across methods is not clear to me. Can the authors summarize here and provide some insights?
2 comments
I would like to thank the authors for their detailed reply. Most of my concerns are resolved and I'll increase my score to 4.
We thank the reviewer for their detailed feedback, and address the concerns and questions raised below.
[W1] Connection to Stein transport:
Stein transport samples the posterior with a deterministic particle flow from the tractable prior π along the tempering path πt∝πexp(tR), where R is the log-likelihood. In all their numerical evaluations, π is an isotropic Gaussian whose score ∇logπ is known exactly. Applying this method directly to a generative prior q1, would suggest a deterministic particle flow along πt∝q1exp(tR), relying on ∇logq1 from the generative model, which is both approximate and ill-behaved in the data space.
Motivated by the empirical success of guidance methods, IPG instead transports particles along pt∝qtexp(Rt) induced by the latent generative flow. The dynamics are primarily driven by the generative drift and the Langevin guidance (neither is part of Stein transport), and the corrective drift only accounts for the remaining reweighting term. What IPG shares with Stein transport is that the PDE to be solved (eq. 11) has the same form, and we approximate its solution by minimizing a regularized objective in an RKHS, following their approach.
We have expanded our earlier remark under Section 3.1 to emphasize these connections.
[W2, Q5] Practical significance:
Our aim is to sample from the reward-tilted prior which is of relevance in many scientific applications, and a method to do so without resampling avoids particle collapse in high dimensions. In an image restoration problem, if a user only needs a single perceptually realistic point estimate, we do not claim IPG is the optimal choice. We include the image experiments to visually evaluate the behavior of different samplers designed with the same goal as ours on high dimensional, complex target distributions.
[W3, Q2] Computational cost:
In kernel-based methods for sampling, such as Stein transport and SVGD, the score is typically cheap, and the cost of the kernel evaluations O(N2d) and the linear solve O(N3) are often discussed. However, with generative priors, the guidance term involving the prior score and the reward gradient requires a forward and backward pass through the generative model, which dominates the computational and memory cost. Unlike DriftLite, IPG assembles the Gram matrix by reusing these expensive terms and the additional terms k(Xi,Xj), ∇Xjk(Xi,Xj), and ∇Xi⋅∇Xjk(Xi,Xj) add negligible overhead. This is visible from the run times reported in the image and protein experiments which show that IPG is nearly as fast as FKC (122.9s vs 122.1s per image, and 126s vs 125.7s per protein).
[W4, Q1] Choosing the kernel:
In all our experiments, we use a simple RBF kernel with the bandwidth adaptively chosen at each time step using the median heuristic. It is twice differentiable with the gradients known in closed-form and are cheap to compute. More specialized choices such as kernels learned offline at different noise levels [1], or applying the kernel to an intermediate representation of the generative model itself k(Φ(Xi),Φ(Xj) could readily be used with IPG. While such more advanced kernels could possibly improve the performance, they would also increase the cost of evaluating the kernel and its gradients. Since IPG is generic to the choice of kernel we did not pursue an in-depth empirical evaluation of different kernels in this paper.
[Q3] Choosing the number of particles N:
We recommend using the largest N that fits in memory. As N increases, the number of basis functions available to represent ut increases and the estimated drift approach the optimal corrective drift within the RKHS. That being said, our empirical results show that even with a modest N (N=16 in most of our experiments) the interactions provided by IPG improves performance.
[Q4] Residual weights:
In all our original experiments, tracking the weights for IPG and IPG-CV results in ESS/N≥0.999. Consequently, systematic resampling at the final time step returns all N particles with high probability. So ignoring the residual weights is a practical implementation choice. We have updated the paper to emphasize this point. See also the reply to Reviewer rRhd.
[Q6, Q7] GMM experiment:
In our original experiment, the prior GMM has 10 equally weighted components, and the posterior is concentrated on one component. In Figure 1, the differences between the methods are not visually striking from the two random slices of the 256 dimensional posterior, however, we hope they are clear from Table 1. We have now also included an illustrative 2D GMM experiment in Appendix E, which is described in our response to Reviewer rRhd.
References:
[1] Galashov et. al., Deep MMD Gradient Flow without adversarial training, ICLR, 2025.