acceptodds
Under review as a conference paper at ICLR 2027

Interacting Particle Guidance for Sampling Reward-tilted Generative Priors

10 views

Abstract

Inference-time steering adapts pretrained diffusion and flow-based models to new tasks, e.g., to generate samples from a conditional distribution or samples with desired properties, without retraining. This can be formalized as sampling from a reward-tilted generative prior. As exact sampling from this distribution is intractable, guidance-based methods rely on approximations producing biased samples, and sequential Monte Carlo (SMC) methods correct for this bias using importance weights. However, while exact in the large particle limit, SMC suffers from weight degeneracy and particle collapse in practice. We propose interacting particle guidance (IPG), which replaces reweighting with transport. The particles interact through an additional drift, derived from the Feynman–Kac PDE to cancel the reweighting term, and remain unweighted. Choosing the drift in a reproducing kernel Hilbert space yields a closed-form solution that is cheap to compute, with negligible overhead compared to SMC. We demonstrate the method on Gaussian mixtures with known posteriors, and on high-dimensional image inpainting and protein structure inference tasks.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Inference-time steering, interacting particles, reward-tilting

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_BzrAOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper develops a new method for reward tilting, that is, sampling from a tilted distribution defined by a pretrained diffusion model and a reward model. The method builds on the idea of the Stein Transport (Nüksen, 2024), which concerns learning an additional drift correction term to a reward-titled SDE so that no additional importance weighting is needed. Learning this transport requires solving a linear system related to a regularized RKHS objective at each sampling step, which scales with the number of particles. However, the particle system is only an approximation and there is a residual term that in principle requires importance corrections while the paper deliberately ignores it (more-or-less justified when the RKHS norm regularization strength is small). The paper also proposes a control-variate variant. The empirical study includes GMM posterior sampling, image inpainting, and protein generation from distance measurements. The proposed method is leading on the GMM tasks on several distance metrics; On image and protein tasks, the trends are mixed for the distance metrics, while the proposed method seems to have better diversity.

Critical strengths
  • The paper provides a new perspective for diffusion reward tilting, and the method is very well motivated to alleviate the issue of importance weight collapse.

  • The derivation is mostly easy to follow (assuming background on Stein Transport)

  • The proposed method seems computationally tractable, and provides a nice alternative to Drift Lite where a basis needs to be specified.

Critical weaknesses

(i) the novelty and I wonder how much is it different from just applying the original Stein Transport to this diffusion model reward tilting setting. If that's the case, the authors should make it more clear in the writing. The paper also misses a background in Stein Transport which the current method is heavily developed on.

(ii) practical relevance: the proposed method does not uniformly improve over baselines across metrics; diversity seems to be the only highlight. But note that the paper does not really use importance sampling so that itself is less surprising.

(iii) No detailed computational cost comparison to other methods

(iv) No detailed discussion on the choice of kernel and its impact.

Questions for authors
  • Can the author discuss the choice of the kernel for different tasks? What is a good criteria of the kernel?

  • Is computing the kernel for high-dimensional data and across many particles a computational concern?

  • What is a good guiding principle to choose N, the number of particles? What would you expect the performance scaling with N?

  • Have you considered accounting for the neglected residual with the importance weights and how would the resulting algorithm perform?

  • Another note on practical relevance: Why limit comparison to methods that are designed to be consistent? For GMM posterior sampling task, it makes sense. But for image inpainting or protein generation, the metrics listed do not necessarily align with consistency. Would the authors consider some other baselines on these tasks, in particular, the one without important sampling or SMC, and how would the proposed method perform against them?

  • How many mixture components are there in the GMM experiment?

  • In GMM experiment Fig 1, the qualitative difference across methods is not clear to me. Can the authors summarize here and provide some insights?

2 comments

We thank the reviewer for their detailed feedback, and address the concerns and questions raised below.

[W1] Connection to Stein transport:

Stein transport samples the posterior with a deterministic particle flow from the tractable prior π\pi along the tempering path πt∝πexp⁡(tR)\pi_t \propto \pi \exp(tR), where RR is the log-likelihood. In all their numerical evaluations, π\pi is an isotropic Gaussian whose score ∇log⁡π\nabla\log \pi is known exactly. Applying this method directly to a generative prior q1q_1, would suggest a deterministic particle flow along πt∝q1exp⁡(tR)\pi_t \propto q_1 \exp(tR), relying on ∇log⁡q1\nabla \log q_1 from the generative model, which is both approximate and ill-behaved in the data space.

Motivated by the empirical success of guidance methods, IPG instead transports particles along pt∝qtexp⁡(Rt)p_t \propto q_t \exp(R_t) induced by the latent generative flow. The dynamics are primarily driven by the generative drift and the Langevin guidance (neither is part of Stein transport), and the corrective drift only accounts for the remaining reweighting term. What IPG shares with Stein transport is that the PDE to be solved (eq. 11) has the same form, and we approximate its solution by minimizing a regularized objective in an RKHS, following their approach.

We have expanded our earlier remark under Section 3.1 to emphasize these connections.

[W2, Q5] Practical significance:

Our aim is to sample from the reward-tilted prior which is of relevance in many scientific applications, and a method to do so without resampling avoids particle collapse in high dimensions. In an image restoration problem, if a user only needs a single perceptually realistic point estimate, we do not claim IPG is the optimal choice. We include the image experiments to visually evaluate the behavior of different samplers designed with the same goal as ours on high dimensional, complex target distributions.

[W3, Q2] Computational cost:

In kernel-based methods for sampling, such as Stein transport and SVGD, the score is typically cheap, and the cost of the kernel evaluations O(N2d)O(N^2d) and the linear solve O(N3)O(N^3) are often discussed. However, with generative priors, the guidance term involving the prior score and the reward gradient requires a forward and backward pass through the generative model, which dominates the computational and memory cost. Unlike DriftLite, IPG assembles the Gram matrix by reusing these expensive terms and the additional terms k(Xi,Xj)k(X^i, X^j), ∇Xjk(Xi,Xj)\nabla_{X^j}k(X^i, X^j), and ∇Xi⋅∇Xjk(Xi,Xj)\nabla_{X^i} \cdot \nabla_{X^j}k(X^i, X^j) add negligible overhead. This is visible from the run times reported in the image and protein experiments which show that IPG is nearly as fast as FKC (122.9122.9s vs 122.1122.1s per image, and 126126s vs 125.7125.7s per protein).

[W4, Q1] Choosing the kernel:

In all our experiments, we use a simple RBF kernel with the bandwidth adaptively chosen at each time step using the median heuristic. It is twice differentiable with the gradients known in closed-form and are cheap to compute. More specialized choices such as kernels learned offline at different noise levels [1], or applying the kernel to an intermediate representation of the generative model itself k(Φ(Xi),Φ(Xj)k(\Phi(X^i), \Phi(X^j) could readily be used with IPG. While such more advanced kernels could possibly improve the performance, they would also increase the cost of evaluating the kernel and its gradients. Since IPG is generic to the choice of kernel we did not pursue an in-depth empirical evaluation of different kernels in this paper.

[Q3] Choosing the number of particles NN:

We recommend using the largest NN that fits in memory. As NN increases, the number of basis functions available to represent utu_t increases and the estimated drift approach the optimal corrective drift within the RKHS. That being said, our empirical results show that even with a modest NN (N=16N=16 in most of our experiments) the interactions provided by IPG improves performance.

[Q4] Residual weights:

In all our original experiments, tracking the weights for IPG and IPG-CV results in ESS/N≥0.999N \geq 0.999. Consequently, systematic resampling at the final time step returns all NN particles with high probability. So ignoring the residual weights is a practical implementation choice. We have updated the paper to emphasize this point. See also the reply to Reviewer rRhd.

[Q6, Q7] GMM experiment:

In our original experiment, the prior GMM has 10 equally weighted components, and the posterior is concentrated on one component. In Figure 1, the differences between the methods are not visually striking from the two random slices of the 256 dimensional posterior, however, we hope they are clear from Table 1. We have now also included an illustrative 2D GMM experiment in Appendix E, which is described in our response to Reviewer rRhd.

References:

[1] Galashov et. al., Deep MMD Gradient Flow without adversarial training, ICLR, 2025.

I would like to thank the authors for their detailed reply. Most of my concerns are resolved and I'll increase my score to 4.

Reviewer_isdbOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose a sampling algorithm to tilt a generated distribution pθ(x)p_\theta(x) effectively (without collapse), particularly for use cases where the reward function is differentiable and the compute is typical (GPU memory), preventing too many particles. They do so by using a drift correction with a known closed form instead of resampling as used in prior methods. They show quantitative results for two different versions of the algorithm IGP and IGP-CV (based on how E[g] is treated), proving the samplers' performance in the case of a toy GMM distribution, an inpainting task, and a protein structure inference task. Qualitatively, they have given a comparison of generated samples in all three tasks to prove the primary gain of the sampler—diversity.

Critical strengths
  1. The paper is overall theoretically well motivated. The reasoning behind why particle collapse occurs in low N settings because of resampling makes the solution of using drift correction pretty intuitive. It is also clear from the Feynman-Kac formulation how in the ideal case scenario the drift correction will be able to perfectly compensate for the absence of reweighting. Including IGP-CV, where 𝔼pt[gt]𝔼_{pₜ}[gₜ] is considered as a free parameter, is also commendable.
  2. It is clear from the quantitative and qualitative results on the real-world tasks that the IGP and IGP-CV do solve the particle collapse problem by drastically increasing the diversity while providing competitive results in the quality metrics.
Critical weaknesses
  1. It seems counterintuitive to use N=256 in the case of GMM experiments, as the primary claim of the rest of the paper is about the effectiveness of the sampler in low N (=16) settings. Consequently, it also reduces the usefulness of including Figure 1, as qualitatively all the algorithms seem to cover the posterior.
Questions for authors
  1. What is the intent behind using N=256 in the case of GMM instead of N=16?
  2. Is it possible to quantitatively show the extent of interaction between particles in the case of images or proteins, just to be sure interaction does matter (as it's in the title)?
  3. Don't you feel the paper shares its name with another paper that also deals with a different use case of steering [1], which hasn't been cited/ resolved in the related works section?

[1] Particle Guidance: Non-I.I.D. Diverse Sampling with Diffusion Models

1 comment

We thank the reviewer for their positive assessment and thoughtful feedback. We address the questions raised below.

[Q1] Choice of NN in the GMM experiment:

The GMM experiment provides an exact posterior to estimate distributional metrics such as MMD and SWD. We chose N=256N=256 to have a reasonably low variance estimate of these metrics, while still keeping NN small relative to the dimensions, as in the case of generative priors.

Based on your feedback, we present an alternative evaluation using N=16N=16, targeting a fixed GMM posterior. To reduce the variance in the estimated metrics, they are instead averaged over 20 random seeds, with MMD and SWD computed against 4096 samples from the true posterior. As a reference, we include an ``exact'' row that report the metrics for 16 posterior samples averaged over the seeds.

MethodResamplingESS/NN ↑\uparrowMean error ↓\downarrowMMD ↓\downarrowSWD ↓\downarrow
Exact––2.215±0.1202.215 \pm 0.1200.002±0.0020.002 \pm 0.0020.227±0.0050.227 \pm 0.005
FKCEvery step1.000±0.0001.000 \pm 0.0006.236±0.8296.236 \pm 0.8290.252±0.0460.252 \pm 0.0460.460±0.0530.460 \pm 0.053
DriftLiteEvery step1.000±0.0001.000 \pm 0.0004.512±0.6274.512 \pm 0.6270.136±0.0470.136 \pm 0.0470.352±0.0370.352 \pm 0.037
IPGNone1.000±0.0001.000 \pm 0.0002.360±0.1502.360 \pm 0.1500.004±0.0020.004 \pm 0.0020.234±0.0090.234 \pm 0.009
IPG-CVNone1.000±0.0001.000 \pm 0.0002.277±0.1062.277 \pm 0.1060.009±0.0280.009 \pm 0.0280.230±0.0060.230 \pm 0.006

IPG and IPG-CV remain closer to the reference than FKC and DriftLite across all metrics.

[Q2] Particle interactions:

To assess the role of the particle interaction for the image and protein experiments, we split each particle's corrective drift utu_t into self-interaction and cross-interaction terms, and show that the median ratio of their norms stays close to one for all tt in Appendix H. This shows that both self-interaction and cross-interaction contribute actively to the method. The RBF kernel with an adaptive bandwidth chosen using the median heuristic at each time step promotes this. We also note that for N=1N=1, the corrective drifts vanish, as the centered gt=0\mathrm{g}_t = 0 (eq. 13 and 15), and so ϕ=0\phi = 0.

[Q3] Connection to Corso et al.:

Thank you for pointing out this reference. While the names are similar and both methods use interacting particles, they address different problems. IPG targets the reward-tilted prior p(x)∝q(x)exp⁡(R(x))p(x)\propto q(x)\exp(R(x)), and do so by augmenting the standard guided flow with an interacting corrective drift to account for the reweighting term. Particle guidance [1] instead aims to generate NN diverse samples from the generative prior, formalized as targeting the joint pN(xˉ)∝Φ(xˉ)Πi=1Nq(xi)p^N(\bar x) \propto \Phi(\bar x)\Pi_{i=1}^N q(x_i) where xˉ=(x1,…,xN)\bar x = (x_1,\dots, x_N) and Φ(xˉ)\Phi(\bar x) is a repulsive potential, and samples approximately by adding the log-potential gradient as guidance. Hence, in their setting the particle interaction comes only through the manually chosen distributional potential, not through the mechanism by which the tilted prior is sampled.

Interestingly, Howard & Nüsken [2] propose an FKC-like approach with reweighting and resampling to exactly sample generative priors tilted by such distributional rewards. However, they note that resampling may not be well suited to promote diversity. IPG can be adapted to this setting to account for the reweighting term through transport instead.

We now include a brief remark on particle guidance in Section 4.1.

References:

[1] Corso et. al., Particle guidance: non-iid diverse sampling with diffusion models, ICLR, 2024.

[2] Howard & Nüsken, A mean-field framework for inference-time distributional control of diffusion models, SPIGM Workshop, ICML, 2026.

Reviewer_rRhdOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to sample from reward-tilted distributions using pretrained diffusion and flow models. Guidance can introduce bias, while reweighting and resampling can leave the particles representing very similar samples. The proposed method adds a corrective drift intended to replace reweighting by moving the particles toward the target distribution. This drift is derived from the Feynman-Kac formulation and estimated in a reproducing kernel Hilbert space, with a closed-form solution. Remaining weight updates are discarded in practice. The paper evaluates IPG and its control-variate variant on Gaussian mixtures with known posteriors, image inpainting, and protein backbone inference. It reports better posterior approximation in the synthetic setting, along with improved diversity and several better best-of-N results in the application tasks.

Critical strengths
  1. I find the method well motivated and computationally practical. It addresses particle collapse through a corrective drift that can be computed without an inner training loop. The Gaussian-mixture experiment provides direct evidence of better posterior approximation, while the reported application runtimes suggest little additional cost over FKC.

  2. The evaluation connects the synthetic results to two different applications. The image results show visible diversity where the tested baselines largely collapse. In the protein task, the method better preserves the expected variation in orientation and finds a designable structure in more test cases. These are useful gains when only a small set of candidate samples can be generated.

Critical weaknesses

The contribution of the corrective drift could be established more directly. The paper does not compare against the same guided dynamics with this drift removed and the particles still returned without weights. The baselines labelled "None" do not provide this control, because their final samples are resampled according to the weights. Without this comparison, it is difficult to separate the benefit of the proposed correction from that of avoiding reweighting and resampling. The justification for discarding the residual weights is also incomplete. Nearly uniform weights are encouraging, but do not by themselves show that the unweighted particles follow the target distribution.

Questions for authors
  1. Could you report the Gaussian-mixture results without the corrective drift, while keeping the other sampling settings unchanged and returning unweighted samples? This would help clarify how much of the improvement comes from the proposed correction.

  2. Could you clarify whether the Gaussian-mixture posteriors retain substantial probability mass in multiple components or are dominated by a single component? This would help interpret the scope of the synthetic evaluation, given the limitation concerning transport across separated modes.

2 comments

We thank the reviewer for their constructive comments. We address the questions raised, and include a new experiment to illustrate the effect of the corrective drift (Appendix E).

[W] Residual weights:

In all our original experiments, we track the log-weight increments dωti=−λϕi dtd\omega_t^i = -\lambda\phi_i\, dt for diagnostics, which for IPG and IPG-CV results in ESS/N≥0.999N \geq 0.999. Consequently, systematic resampling at the final time step returns all NN particles with high probability. So ignoring the weights is a practical implementation choice which did not have any visible effect in our experiments. However, we acknowledge that obtaining an ESS/N≥0.999N \geq 0.999 need not hold in every setting, since the weights depend not only on λ\lambda but also on ϕ\phi. We have made minor revisions in Sections 1-3 to clarify this, and report the tracked ESS/NN for IPG in Table 1.

As the corrective drift is fitted to cancel the log-weight increments, we agree that a high ESS is expected by construction and by itself should not be read as a measure of accuracy. We have revised the discussion in Section 5.1 to highlight this.

[Q1, Q2] GMM experiment:

For the 256-dimensional GMM priors with 10 equally weighted components, under 128-dimensional observations, the posteriors are concentrated on one component. For these cases, a heuristic guidance without the corrective drift, reweighting or resampling (i.e., similar to "None" in the original submission, but simply ignoring the weights), and with just the Langevin terms performs nearly as well as IPG with slightly higher mean error and SWD.

MethodMean errorMMDSWD
Heuristic0.874±0.0460.874 \pm 0.0460.012±0.0010.012 \pm 0.0010.094±0.0010.094 \pm 0.001
IPG0.841±0.0510.841 \pm 0.0510.012±0.0020.012 \pm 0.0020.093±0.0020.093 \pm 0.002

However, in general, heuristic guidance does not follow the target ptp_t, as the PDE for its marginals is missing the term pt(gt−Eptgt)p_t(g_t - \mathbb{E}_{p_t}g_t). In high dimensions, correcting for this term with reweighting and resampling leads to particle collapse, which a transport-based correction avoids. To illustrate the effect of the corrective drift, we include an experiment, adapted from Potaptchik et al. [1], where the prior GMM has 3 equally weighted components and the posterior component weights are 0.5720.572, 0.3660.366, and 0.0620.062. With the same settings as before, the metrics averaged over 20 random seeds are:

MethodESS/NN ↑\uparrowMean error ↓\downarrowMMD ↓\downarrowSWD ↓\downarrow
Heuristic–1.635±0.2331.635 \pm 0.2330.207±0.0320.207 \pm 0.0321.462±0.1851.462 \pm 0.185
IPG0.968±0.0050.968 \pm 0.0050.269±0.2220.269 \pm 0.2220.029±0.0380.029 \pm 0.0380.433±0.2210.433 \pm 0.221
IPG-CV0.962±0.0050.962 \pm 0.0050.276±0.1710.276 \pm 0.1710.039±0.0400.039 \pm 0.0400.460±0.1850.460 \pm 0.185
IPG (σt=0\sigma_t=0)0.977±0.0200.977 \pm 0.0201.351±1.3731.351 \pm 1.3730.289±0.2250.289 \pm 0.2251.226±0.9311.226 \pm 0.931
IPG-CV (σt=0\sigma_t=0)0.953±0.0070.953 \pm 0.0070.408±0.2340.408 \pm 0.2340.054±0.0620.054 \pm 0.0620.454±0.2100.454 \pm 0.210

Here, tracking the weights for IPG shows that ESS/NN is high but not close to one, so we resample the final particles. The heuristic guidance performs significantly worse and including the corrective drift leads to clear improvements. We also evaluate deterministic flows with only the generative drift and corrective drift, which performs reasonable well with the control variate version, but still worse than IPG with Langevin terms. Furthermore, the approach without the Langevin terms would need much larger NN to scale with dimensions.

References

[1] Potaptchik et. al., Meta flow maps enable scalable reward alignment, ICML, 2026.

Thank you for the additional experiments. The authors have substantially addressed my concerns throughout the rebuttal, and I will raise my rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.