acceptodds
Under review as a conference paper at ICLR 2027

Inference-Time Alignment of Diffusion Models via Trust-Region Iterative Twisted Sequential Monte Carlo

Abstract

We study inference-time alignment of diffusion models: steering a fixed pretrained model toward high-reward outputs by changing its sampling procedure. Existing Sequential Monte Carlo (SMC) methods often rely on the base model to propose samples, then use reward information to reweight and resample them. When high-reward outputs are rare, this can lead to weight degeneracy and require many particles. Twisted SMC addresses this problem by modifying proposals using look-ahead information, but learning effective twisting functions can be unstable in high-dimensional diffusion models. We propose Trust-Region Iterative Twisted Sequential Monte Carlo (TRI-TSMC), which learns twisting functions through two steps. First, it constructs an intermediate trajectory distribution that moves toward the reward-tilted target while staying within a KL trust region around the current proposal. This step has a closed-form solution based on tempered importance weights. Second, it fits the twisting function to this distribution by weighted maximum likelihood. Theoretically, we characterize the optimal twist through a soft value function and show that it yields a zero-variance sampler. We prove that the exact trust-region update follows an escort path that reduces residual importance-weight variance, and that the population fitting objective is a forward-KL projection. Experiments on discrete diffusion text generation and text-to-image generation show improved primary alignment objectives under matched total sampled-trajectory budgets.

Primary area:
generative models
Keywords:
diffusion model alignment, sequential monte carlo, inference-time alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yfxKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inference-time alignment for diffusion-based generative models and introduces TRI-TSMC, a trust-region iterative SMC algorithm. Different from standard SMC algorithms (say, FK-steering), the authors design a sequence of intermediately aligned distributions through the trust region, which enables more stable distributional updates. Theoretical analyses are provided to justify such a stability, and experiments on MDLM and T2I show advantages over standard SMC.

Critical strengths
  1. A new algorithm that performs better than standard SMC
  2. Theory that justifies variance reduction (to some extent)
Critical weaknesses
  1. The paper seems to be too focused on standard SMC (particularly FK-steering) while missing discussion/comparison with some related works after that. As far as I know, this includes the SP-gSMC and the DOIT algorithms, both are recently proposed inference-time algorithms. While SVDD is mentioned, it is done too slightly.

  2. I am not certain that the current experiment fully supports the advantage claimed. See below. 2.1 In the main text, only the average performance is reported. How does this justify the central claim on the better pathwise stability for the proposed algorithm (and, after all, how is this related to the variance reduction from theory)? The same critique holds for the sensitivity analysis in Appendix D.7. 2.2 Both tables include only KK, but, from Algorithm 1, it seems that the total computation for the algorithm is I⋅KI \cdot K.

Questions for authors
  1. I'd like the authors to clarify the position of this paper within related works.
  2. See the weakness section above. Besides, in Appendix D.6 only the case with and without trust region is compared. Is it possible to address the intermediate values as well?
  3. Typo: On the fourth line, "DDPPO" should be DPPO.
4 comments

Response to Reviewer yfxK

We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.


Q1: Clarification about the position of this paper within related works

A1: We thank the reviewer for pointing out these relevant works. These methods are closely related to TRI-TSMC through reward-guided sampling. Our specific focus is how to learn and iteratively update the twisting function from a limited number of trajectories.

  • SP-gSMC [1] uses an approximate value function to guide sampling and analyzes how its approximation error affects sampling accuracy and complexity. This addresses a complementary question: how the quality of a given guide affects sampling, while TRI-TSMC focuses on learning and updating the guide.
  • DOIT [2] shares our goal of modifying diffusion sampling toward a reward-tilted target. It estimates Doob corrections at each denoising step through simulation or a practical approximation. TRI-TSMC instead learns a parameterized twisting function from completed trajectories and updates it across sampling rounds.
  • SVDD [3] shares the soft-value interpretation underlying our twisting function. It estimates soft values through regression or a posterior mean approximation, then uses them to select among candidate next states. TRI-TSMC uses weighted maximum likelihood to fit the twisting function and changes the proposal used in subsequent sampling rounds.

The key contribution of TRI-TSMC is to separate each update into constructing a KL-constrained intermediate trajectory distribution and fitting the twisting function to that distribution. The trust region controls the update size and tempers the fitting weights, allowing more trajectories to contribute to learning. Our theory characterizes the exact distribution update, while Appendix D.6 demonstrates reduced weight concentration during twist fitting. We have expanded the Related Work discussion to make these connections and differences explicit.


Q2: Pathwise stability and its connection to the variance-reduction theory

A2: Our stability analysis focuses on how importance weights are distributed across complete generation trajectories. Theorem 4.5 shows that the normalized variance of the residual importance weights decreases along the exact intermediate distributions as they move toward the target.

Appendix D.6 provides complementary evidence from the twist-fitting step. Under the same sampling and optimization setup, trust-region tempering reduces the fitting-weight CV² from 14.378 to 0.446 and increases ESS/K from 0.0666 to 0.692. The largest normalized weight decreases from 0.973 to 0.171. With 16 trajectories, the effective sample size therefore increases from approximately one to eleven, allowing more trajectories to contribute to the update. PPL also improves from 33.390 to 31.694.

Together, the theory explains the reduction in residual-weight variance under the exact distribution update, while Table 6 demonstrates reduced weight concentration during twist fitting. These weight diagnostics provide evidence of stability beyond average generation performance.


Q3: In Appendix D.6 only the case with and without trust region is compared. Is it possible to address the intermediate values as well?

A3: We provide a trust-region radius sweep in Appendix D.7, Table 7. In the GPT-2 mixed-reward setting, we vary ϵ\epsilon over (0.10,0.20,0.50)(0.10, 0.20, 0.50) while keeping the remaining settings fixed.

Trust-region radius ϵ\epsilonPPL ↓
0.1030.582
0.2029.781
0.5031.067

A smaller ϵ\epsilon imposes a tighter constraint on each update, while a larger ϵ\epsilon allows a larger change in the proposal. The intermediate value ϵ=0.20\epsilon=0.20 achieves the lowest PPL among these settings.

Together with the controlled ablation in Appendix D.6, these results directly support our claims that trust-region tempering reduces weight concentration during twist fitting and improves alignment performance. Appendix D.6 demonstrates the benefit of introducing the trust region, while Table 7 shows how its strength affects the resulting performance.


Q4: Clarification about total trajectory budget and computational cost

A4: KK denotes the number of particles per sampling round, and the total number of sampled trajectories is IKIK, where II is the number of sampling rounds. Our experiments use the following budgets:

TaskParticles per roundSampling roundsTotal trajectories
Text generation16348
Text-to-image generation428

These totals match the corresponding 48-trajectory and 8-trajectory baselines. For images, the budget includes four initial trajectories used to learn the twisting function and four trajectories sampled after the update.

We also provide end-to-end runtime measurements, including twist optimization, in Appendix D.8. Table 8 reports runtime at the same 48-trajectory budget. Table 9 compares the direct-PiP_i variant with the baselines under the PPL-only reward at approximately matched runtime and peak memory. In that comparison, TRI-TSMC uses 48 trajectories and takes 21.22 seconds per output, while FK-Steering uses 108 trajectories and takes 20.59 seconds.


Q5: Correction of “DDPPO”

A5: Thank you for pointing out the typo. The reference is DDPO—Denoising Diffusion Policy Optimization by Black et al. [4]. We have corrected the acronym accordingly.


We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?

References:

[1] Zhu and Lu. On the Power of (Approximate) Reward Models for Inference-Time Scaling: Sequential Monte Carlo and Beyond. ICML 2026.

[2] Zhu et al. Training-Free Adaptation of Diffusion Models via Doob’s h-Transform. ICML 2026.

[3] Li et al. Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding. 2024.

[4] Black et al. Training Diffusion Models with Reinforcement Learning. ICLR 2024.

My concerns have been partly addressed. While I do think the current connection between theory and experiment can be strengthened, there is merit in the current version. I have raised my score.

Thank you for taking the time to reconsider our work. Your thoughtful feedback has helped us improve the quality of the manuscript.

Reviewer_rN5fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies inference-time alignment of diffusion models. The authors argue that SMC-based steering methods suffer from weight degeneracy when high-reward outputs are rare. Twisted SMC (TSMC) can address this in principle, but learning the twisting functions is unstable in high dimensions.

They propose TRI-TSMC, a method that improves TSMC by learning the twist function iteratively. Each iteration runs TSMC with the current twist and then forms an intermediate target that moves toward the reward-tilted target while staying within a KL trust region around the current proposal. The twist network is then fit to this target by weighted maximum likelihood.

The paper also proves that the log of the optimal twist is a soft value function, which gives a zero-variance sampler. It further shows that the exact trust-region update reduces the variance of the importance weights, and that the fitting step is a forward-KL projection.

Experiments on text generation (MDLM) and text-to-image generation (SD v1.5 and SDXL) show that, with the same total number of sampled trajectories, TRI-TSMC achieves the best score on the main target metric (PPL for text, IR for images). However, it has lower diversity in text, and lower GenEval and similar HPS scores in images.

Critical strengths

The method improves TSMC with a really simple method. The trust-region design only tempers the importance weights, and the temperature is chosen automatically by a one-dimensional dual problem. The method needs only black-box reward evaluations, does not fine-tune the base model, and works for both discrete and continuous diffusion models. This helps stabilize the twist learning and enable TSMC for diffusion alignment, and this is easy to adopt.

Critical weaknesses
  1. The gains appear mainly on the rewards being optimized. The metrics (PPL, CoLA, and IR) are the same rewards that TRI-TSMC optimizes during sampling. In Table 1, TRI-TSMC uses a PPL + CoLA reward, while FK-Steering uses only the PPL reward, and the reward used by BoN is not stated. So the large CoLA gain over the baselines is expected. When FK-Steering uses the same mixed reward (Table 4), it reaches PPL 31.1 and CoLA 0.71, close to TRI-TSMC’s 29.8 and 0.73. This comparison is also not fully matched, since FK-Steering uses β = 0.40 but TRI-TSMC uses 0.55. However, Table 5 reports a PPL decrease from 36.702 to 31.545 under the same PPL-only reward and 48-trajectory budget, showing an improvement independent of the additional CoLA reward. For images, BoN, FK-Steering, and TRI-TSMC all use IR, and TRI-TSMC’s gain over BoN and FK-Steering with K=8 is small (0.979 vs. 0.945 and 0.930). On the secondary metrics, which are not optimized, GenEval is lower than both K=8 baselines and HPS is similar. In addition, the results are averaged over 3 seeds but reported without error bars, so it is hard to tell whether these small gaps are meaningful. The results do not show consistent improvements across quality metrics beyond the optimized rewards.

  2. Diversity drops. TRI-TSMC has the lowest Dist-n among all methods in text. When FK-Steering uses the same mixed reward (Table 4), it reaches similar PPL and CoLA but keeps more diversity. With the Qwen2.5 reward (Figure 4), the diversity gap is larger. For images, no diversity metric is reported, and the qualitative examples in Figures 3 and 6 look very similar to each other. The authors also note diversity as the main limitation of the method (Appendix D.9).

Questions for authors
  1. Which reward does BoN use in the text experiments? Could you compare BoN and FK-Steering using the same mixed reward and β = 0.55 as TRI-TSMC, and match the potential between FK-Steering and TRI-TSMC? A clear gain in this setting would address my main concern.

  2. Since the main metrics are the same rewards used during sampling, could you evaluate with metrics that are not optimized? (maybe human judgment?)

  3. Could you report error bars over the 3 seeds for Tables 1, 2, and 13? Some gaps are small, so it is hard to tell whether they are meaningful.

  4. Could you report a quantitative diversity metric for images? Like CLIP-I distance of images generated from the same prompt?

7 comments

Response to Reviewer rN5f

We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.


Q1: Which reward does BoN use in the text experiments? Could you compare BoN and FK-Steering using the same mixed reward and β=0.55\beta = 0.55 as TRI-TSMC, and match the potential between FK-Steering and TRI-TSMC? A clear gain in this setting would address my main concern.

A1: BoN in the original text experiments uses the single PPL reward for final candidate selection, consistent with the main FK-Steering baseline. We have explicitly state this in our revised experimental setup.

We provide additional comparisons using the same mixed reward rPPL+βrCoLAr_{\mathrm{PPL}}+\beta r_{\mathrm{CoLA}} for all three methods, with β=(0.4,0.55)\beta = (0.4,0.55). Results are reported as mean ± standard deviation across three seeds.

β\betaMethodPPL ↓\downarrowCoLA ↑\uparrow
0.4BoN35.634±0.96535.634 \pm 0.9650.588±0.0480.588 \pm 0.048
0.4FK-Steering32.769±0.83232.769 \pm 0.8320.719±0.0110.719 \pm 0.011
0.4TRI-TSMC (ours)27.518±0.80027.518 \pm 0.8000.706±0.0220.706 \pm 0.022
0.55BoN35.525±0.83835.525 \pm 0.8380.642±0.0280.642 \pm 0.028
0.55FK-Steering33.395±0.50533.395 \pm 0.5050.750±0.0230.750 \pm 0.023
0.55TRI-TSMC (ours)28.407±1.19528.407 \pm 1.1950.748±0.0260.748 \pm 0.026

TRI-TSMC achieves the lowest PPL at both tested β\beta values, with comparable CoLA to FK-Steering and higher CoLA than BoN. These results demonstrate its advantage under the same reward, independently of the additional CoLA reward used in the original comparison.


Q2: Since the main metrics are the same rewards used during sampling, could you evaluate with metrics that are not optimized?

A2: In inference-time alignment, the specified reward defines the target distribution. Evaluating this reward therefore directly measures how effectively a method achieves the alignment objective. Our comparisons use the same reward across methods, as shown in Q1.

We also evaluate quality beyond the optimized reward. Following the evaluation protocol of FK-Steering, our text-to-image experiments use ImageReward as the primary alignment metric and HPS and GenEval as independent metrics. Neither HPS nor GenEval is used for steering or final sample selection. HPS predicts human preference, while GenEval checks whether the objects, attributes, and relationships specified in the prompt are correctly generated.

Compared with the base model, TRI-TSMC improves HPS from 0.246 to 0.270 and GenEval from 0.428 to 0.601 on SD v1.5. The same pattern holds on SDXL: HPS increases from 0.282 to 0.300 and GenEval from 0.551 to 0.660. These results show that the improvements over the base generator extend beyond ImageReward to independently evaluated image quality and prompt fidelity.

Under the same eight-trajectory budget, TRI-TSMC achieves the highest ImageReward on both generators, with HPS scores close to those of BoN and FK-Steering.


Q3: Could you report error bars over the 3 seeds for Tables 1, 2, and 13? Some gaps are small, so it is hard to tell whether they are meaningful.

A3: We provide mean ± standard deviation across the three seeds for all metrics in Tables 1, 2, and 13. Each seed is evaluated over the full prompt set, and the standard deviation is computed across the three resulting scores.

MethodKKPPL ↓\downarrowCoLA ↑\uparrowDist-1 ↑\uparrowDist-2 ↑\uparrowDist-3 ↑\uparrow
Base1104.427±2.005104.427 \pm 2.0050.223±0.0410.223 \pm 0.0410.594±0.0050.594 \pm 0.0050.919±0.0010.919 \pm 0.0010.941±0.0010.941 \pm 0.001
BoN1651.301±0.63751.301 \pm 0.6370.347±0.0240.347 \pm 0.0240.552±0.0020.552 \pm 0.0020.879±0.0030.879 \pm 0.0030.922±0.0020.922 \pm 0.002
BoN4844.109±2.19944.109 \pm 2.1990.371±0.0450.371 \pm 0.0450.532±0.0100.532 \pm 0.0100.852±0.0160.852 \pm 0.0160.904±0.0110.904 \pm 0.011
FK-Steering1643.050±1.24843.050 \pm 1.2480.390±0.0120.390 \pm 0.0120.528±0.0050.528 \pm 0.0050.849±0.0030.849 \pm 0.0030.905±0.0040.905 \pm 0.004
FK-Steering4836.702±1.48136.702 \pm 1.4810.393±0.0420.393 \pm 0.0420.520±0.0050.520 \pm 0.0050.831±0.0060.831 \pm 0.0060.888±0.0060.888 \pm 0.006
TRI-TSMC (ours)1629.781±1.68029.781 \pm 1.6800.727±0.0130.727 \pm 0.0130.488±0.0090.488 \pm 0.0090.774±0.0120.774 \pm 0.0120.833±0.0080.833 \pm 0.008
MethodKIR(*)GenEvalHPS
Base10.233 ± 0.0090.428 ± 0.0010.246 ± 0.002
DPO10.359 ± 0.0490.441 ± 0.0140.256 ± 0.001
Best-of-N40.750 ± 0.0890.599 ± 0.0080.267 ± 0.003
Best-of-N80.945 ± 0.0320.670 ± 0.0130.276 ± 0.001
FK-Steering40.718 ± 0.0310.553 ± 0.0100.261 ± 0.001
FK-Steering80.930 ± 0.0340.630 ± 0.0100.270 ± 0.002
TRI-TSMC40.979 ± 0.0080.601 ± 0.0090.270 ± 0.002
BaseMethodKIR(*)GenEvalHPS
SDXLBase10.752 ± 0.0310.551 ± 0.0030.282 ± 0.002
SDXLDPO10.913 ± 0.0280.586 ± 0.0040.299 ± 0.001
SDXLBest-of-N41.094 ± 0.0070.732 ± 0.0100.296 ± 0.001
SDXLBest-of-N81.195 ± 0.0190.811 ± 0.0100.301 ± 0.000
SDXLFK-Steering41.076 ± 0.0160.665 ± 0.0210.292 ± 0.001
SDXLFK-Steering81.238 ± 0.0260.723 ± 0.0140.298 ± 0.000
SDXLTRI-TSMC41.275 ± 0.0160.660 ± 0.0050.300 ± 0.001

The text results show substantial improvements in PPL and CoLA relative to the observed variation across seeds. For images, TRI-TSMC achieves the highest mean ImageReward on both SD v1.5 and SDXL, with smaller standard deviations than the eight-trajectory BoN and FK-Steering baselines. Thus, the image results show both higher average reward and lower variation across seeds.


Q4: Could you report a quantitative diversity metric for images? Like CLIP-I distance of images generated from the same prompt?

A4: We provide a quantitative comparison of image diversity using the mean pairwise cosine distance between normalized CLIP image embeddings.

For each prompt, we collect the same number of final images from independent runs of each method. Each run follows the method's original sampling and final selection procedure. We first average the distances between image pairs within each prompt, then average across prompts. A higher score indicates greater diversity among images generated from the same prompt.

We report this metric alongside ImageReward to show both output diversity and performance on the alignment objective.

MethodKImageRewardCLIP
Base10.233 ± 0.0090.174 ± 0.009
Best-of-N80.945 ± 0.0320.156 ± 0.009
FK-Steering80.930 ± 0.0340.158 ± 0.009
TRI-TSMC40.979 ± 0.0080.160 ± 0.009

Q5: Text diversity and the quality–diversity trade-off

A5: Table 7 shows how the number of sampling rounds affects PPL and diversity. With 16 trajectories per round, increasing the number of rounds from two to three reduces PPL from 34.344 to 29.781, while Dist-1/2/3 changes from 0.522/0.826/0.882 to 0.488/0.774/0.833. Using fewer rounds preserves more diverse continuations, while additional rounds further reduce PPL.

Appendix D.9 also provides a qualitative analysis of generated text in Tables 10 and 11. In the Qwen-based examples, some outputs from single-reward FK-Steering contain broken word fragments or repeated year-like strings, whereas TRI-TSMC with the mixed reward produces more complete and readable sentences. These examples illustrate why Dist-n should be considered together with fluency and grammatical acceptability when evaluating the generated text.

The matched-reward comparison in Q1 also includes Dist-n, allowing us to compare this trade-off.


We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?

Reviewer_9NaHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper trains the twist in twisted SMC during inference to estimate the expected reward. It relies on tempering the SMC weights to keep the update in a KL trust region around the proposal. 

Experiments use MDLM for text and stable diffusion for images. Reward is reported higher than Best-of-N and FK Steering.

Critical strengths
  • The method is simple and gradient-free, so it works with black-box rewards, and it is shown on both discrete and continuous diffusion.

  • The trust region does what it is supposed to do. Without it one trajectory gets 97% of the weight, with it the ESS goes from about 1 to 11 out of 16.

  • The appendix is thorough and honest (single-reward, mixed-reward, runtime-matched and sensitivity studies), and the diversity limitation is acknowledged.

Critical weaknesses

The paper seems to address the limitation that proposals come from the base distribution as a limitation, however Wu et al. (TDS), Skreta et al. (Feynman-Kac Correctors) and Lee et al. (Debiasing guidance for discrete diffusion with sequential monte carlo) use a proposal from a guided process. The paper does not refer to Skreta et al. and Lee et al., as well no twist-learning baseline. It also does not cite  Zhao et al. (2024, "Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo") who already learn twists for SMC on language models.

  • The text results in Table 1 and Figure 1 mix two effects: the reward and the method. TRI-TSMC uses the reward PPL + CoLA, but FK-Steering uses PPL only. With the mixed reward, FK-Steering gets almost the same CoLA (Table 4). With Qwen, FK-Steering with the mixed reward has a better CoLA and Dist-n, but a worse PPL.

  • With the same reward and the same number of trajectories, TRI-TSMC is better but uses approximately 1.8x the runtime. At the same runtime, FK-Steering matches TRI-TSMC though with higher CoLA and Dist-n. It would be interesting to see the result at matched reward and compute, and error bars

  • Results aren't as convincing for images however.

Questions for authors
  • How does TRI-TSMC compare to SMC with a guided proposal (TDS, Feynman-Kac Correctors) and to a twist-learning baseline (e.g. iAPF, TPPF or contrastive twist learning)?
  • Can you report the mixed-reward comparison at matched runtime, with the same β for both methods and with error bars? A sweep over β would show if TRI-TSMC is better than FK-Steering on the whole PPL / CoLA trade-off.
5 comments

Response to Reviewer 9NaH

We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.


Q1: Discussion of related work and our contributions

A1: We thank the reviewer for pointing out these relevant references. Our discussion of base-model proposals concerns the FK-Steering configuration used in our experiments. The main challenge we address is how to learn and update an effective twisting function from a limited number of trajectories.

TDS [1], Feynman–Kac Correctors [2], and Lee et al. [3] incorporate guidance into sampling, while Zhao et al. [4] learn twisting functions for language models. Guided proposals help move sampling toward the target, but approximate guidance can still leave a substantial mismatch between the proposal and the reward-tilted target. Consequently, importance weights can remain concentrated on a few trajectories. When these weights are used to fit a twisting function, those trajectories can dominate the update.

TRI-TSMC addresses this difficulty by controlling each update. It first constructs an intermediate trajectory distribution that moves toward the target while staying within a KL trust region around the current proposal. It then fits the twisting function to this distribution by weighted maximum likelihood. The trust region limits the change in the trajectory distribution and tempers the fitting weights, allowing more trajectories to contribute to learning.

Our theoretical analysis characterizes the exact distribution update, and Appendix D.6 shows that trust-region tempering reduces weight concentration during twist fitting. The additional comparisons in Q2 further show that TRI-TSMC achieves lower PPL than both TDS and TPPF-RE across the tested reward settings. These results support the effectiveness of our approach against both guided-proposal SMC and an existing twist-learning method. We have clarified this motivation and expanded the related-work discussion in the revision.


Q2: How does TRI-TSMC compare to SMC with a guided proposal (TDS, Feynman-Kac Correctors) and to a twist-learning baseline (e.g. iAPF, TPPF or contrastive twist learning)?

A2: We provide comparisons with TDS [1], an SMC method with a guided proposal, and TPPF-RE [5], a twist-learning baseline. All methods use the same base model and reward rPPL+βrCoLAr_{\text{PPL}}+\beta r_{\text{CoLA}}. We evaluate β∈(0,0.4,0.55)\beta\in (0,0.4,0.55) to cover both PPL-only and mixed-reward settings.

β\betaMethodPPL ↓\downarrowCoLA ↑\uparrowDist-1/2/3 ↑\uparrow
0TDS [1]34.7450.3830.489/0.805/0.880
0TPPF-RE [5]46.2710.3430.544/0.866/0.912
0TRI-TSMC (ours)31.5450.4000.505/0.826/0.892
0.4TDS33.3090.7100.498/0.808/0.880
0.4TPPF-RE44.7130.5230.548/0.869/0.914
0.4TRI-TSMC (ours)27.5180.7060.508/0.814/0.875
0.55TDS32.3450.7710.502/0.808/0.878
0.55TPPF-RE44.3970.5810.550/0.871/0.915
0.55TRI-TSMC (ours)28.4070.7480.515/0.817/0.879

TRI-TSMC consistently achieves the lowest PPL across all three reward settings, with comparable CoLA and diversity to TDS. It also achieves higher CoLA than TPPF-RE, demonstrating its effectiveness against both guided-proposal SMC and an existing twist-learning baseline.


Q3: Separating the effects of reward design and proposal adaptation

A3: Appendix D.5 compares FK-Steering and TRI-TSMC using the same PPL-only reward and a total budget of 48 sampled trajectories.

MethodTotal trajectoriesPPL ↓CoLA ↑Dist-1/2/3 ↑
FK-Steering4836.7020.3930.520/0.831/0.888
TRI-TSMC4831.5450.4000.505/0.826/0.892

TRI-TSMC reduces PPL by 14.1% with comparable CoLA and diversity, demonstrating the benefit of proposal adaptation independently of the additional CoLA reward. Q4 further provides comparisons using the same mixed reward and β\beta for both methods.


Q4: Additional experiments on mixed-reward comparison with the same β\beta for both methods and with error bars

A4: We provide comparisons using the same mixed reward and β\beta. Results are reported as mean ± standard deviation across three seeds.

β\betaMethodPPL ↓\downarrowCoLA ↑\uparrowDist-1 ↑\uparrowDist-2 ↑\uparrowDist-3 ↑\uparrow
0.4BoN35.634±0.96535.634 \pm 0.9650.588±0.0480.588 \pm 0.0480.5318±0.00160.5318 \pm 0.00160.8562±0.00410.8562 \pm 0.00410.9103±0.00680.9103 \pm 0.0068
0.4FK-Steering32.769±0.83232.769 \pm 0.8320.719±0.0110.719 \pm 0.0110.5290±0.00060.5290 \pm 0.00060.8270±0.00140.8270 \pm 0.00140.8814±0.00110.8814 \pm 0.0011
0.4TRI-TSMC (ours)27.518±0.80027.518 \pm 0.8000.706±0.0220.706 \pm 0.0220.5084±0.00360.5084 \pm 0.00360.8142±0.00270.8142 \pm 0.00270.8749±0.00040.8749 \pm 0.0004
0.55BoN35.525±0.83835.525 \pm 0.8380.642±0.0280.642 \pm 0.0280.5316±0.00470.5316 \pm 0.00470.8578±0.00390.8578 \pm 0.00390.9110±0.00680.9110 \pm 0.0068
0.55FK-Steering33.395±0.50533.395 \pm 0.5050.750±0.0230.750 \pm 0.0230.5269±0.00450.5269 \pm 0.00450.8220±0.00530.8220 \pm 0.00530.8764±0.00390.8764 \pm 0.0039
0.55TRI-TSMC (ours)28.407±1.19528.407 \pm 1.1950.748±0.0260.748 \pm 0.0260.5146±0.00320.5146 \pm 0.00320.8167±0.00470.8167 \pm 0.00470.8788±0.00310.8788 \pm 0.0031

TRI-TSMC achieves lower PPL at both tested β\beta values while maintaining comparable CoLA.


Q5: Effectiveness of TRI-TSMC on text-to-image generation

A5: The text-to-image generation experiments test whether the learned twisting function improves generation according to the specified ImageReward objective. Under the same total budget of eight sampled trajectories, TRI-TSMC achieves the highest ImageReward on both SD v1.5 and SDXL:

Base modelBoNFK-SteeringTRI-TSMC
SD v1.50.9450.9300.979
SDXL1.1951.2381.275

TRI-TSMC uses four initial trajectories to learn the twist and four trajectories from the updated proposal. Thus, one twist update improves the target reward on both generators without increasing the total trajectory count.

We also report HPS and GenEval to assess other aspects of image quality. HPS measures human preference, while GenEval checks whether the objects, attributes, and relationships specified in the prompt are correctly generated. TRI-TSMC obtains HPS scores close to the baselines, while the K=8 baselines achieve higher GenEval scores. These results show a trade-off between improving ImageReward and satisfying the specific requirements tested by GenEval. We will make this interpretation explicit in the experimental discussion.


We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?

References:

[1] Wu et al. Practical and Asymptotically Exact Conditional Sampling in Diffusion Models. NeurIPS 2023.

[2] Skreta et al. Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts. ICML 2025.

[3] Lee et al. Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo. 2025.

[4] Zhao et al. Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo. ICML 2024.

[5] Lu et al. Guidance for twisted particle filter: a continuous-time perspective. 2024

Dear Reviewer 9NaH,

Thank you again for your constructive feedback. In our response, we clarified how TRI-TSMC relates to prior work and provided the comparisons you suggested with TDS and TPPF-RE. We also compared TRI-TSMC with FK-Steering using the same mixed reward at two values of β\beta, with error bars across three seeds.

As the discussion period is approaching its end, we would appreciate it if you could let us know whether our response has addressed your concerns or whether there is anything else we can clarify.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.