- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies inference-time alignment for diffusion-based generative models and introduces TRI-TSMC, a trust-region iterative SMC algorithm. Different from standard SMC algorithms (say, FK-steering), the authors design a sequence of intermediately aligned distributions through the trust region, which enables more stable distributional updates. Theoretical analyses are provided to justify such a stability, and experiments on MDLM and T2I show advantages over standard SMC.
Critical strengths
- A new algorithm that performs better than standard SMC
- Theory that justifies variance reduction (to some extent)
Critical weaknesses
-
The paper seems to be too focused on standard SMC (particularly FK-steering) while missing discussion/comparison with some related works after that. As far as I know, this includes the SP-gSMC and the DOIT algorithms, both are recently proposed inference-time algorithms. While SVDD is mentioned, it is done too slightly.
-
I am not certain that the current experiment fully supports the advantage claimed. See below. 2.1 In the main text, only the average performance is reported. How does this justify the central claim on the better pathwise stability for the proposed algorithm (and, after all, how is this related to the variance reduction from theory)? The same critique holds for the sensitivity analysis in Appendix D.7. 2.2 Both tables include only , but, from Algorithm 1, it seems that the total computation for the algorithm is .
Questions for authors
- I'd like the authors to clarify the position of this paper within related works.
- See the weakness section above. Besides, in Appendix D.6 only the case with and without trust region is compared. Is it possible to address the intermediate values as well?
- Typo: On the fourth line, "DDPPO" should be DPPO.
4 comments
Q4: Clarification about total trajectory budget and computational cost
A4: denotes the number of particles per sampling round, and the total number of sampled trajectories is , where is the number of sampling rounds. Our experiments use the following budgets:
| Task | Particles per round | Sampling rounds | Total trajectories |
|---|---|---|---|
| Text generation | 16 | 3 | 48 |
| Text-to-image generation | 4 | 2 | 8 |
These totals match the corresponding 48-trajectory and 8-trajectory baselines. For images, the budget includes four initial trajectories used to learn the twisting function and four trajectories sampled after the update.
We also provide end-to-end runtime measurements, including twist optimization, in Appendix D.8. Table 8 reports runtime at the same 48-trajectory budget. Table 9 compares the direct- variant with the baselines under the PPL-only reward at approximately matched runtime and peak memory. In that comparison, TRI-TSMC uses 48 trajectories and takes 21.22 seconds per output, while FK-Steering uses 108 trajectories and takes 20.59 seconds.
Q5: Correction of “DDPPO”
A5: Thank you for pointing out the typo. The reference is DDPO—Denoising Diffusion Policy Optimization by Black et al. [4]. We have corrected the acronym accordingly.
We hope we have addressed all of your questions. If you have any further questions, we would be happy to answer them and if you don’t, would you kindly consider increasing your score?
References:
[1] Zhu and Lu. On the Power of (Approximate) Reward Models for Inference-Time Scaling: Sequential Monte Carlo and Beyond. ICML 2026.
[2] Zhu et al. Training-Free Adaptation of Diffusion Models via Doob’s h-Transform. ICML 2026.
[3] Li et al. Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding. 2024.
[4] Black et al. Training Diffusion Models with Reinforcement Learning. ICLR 2024.
My concerns have been partly addressed. While I do think the current connection between theory and experiment can be strengthened, there is merit in the current version. I have raised my score.
Thank you for taking the time to reconsider our work. Your thoughtful feedback has helped us improve the quality of the manuscript.
Response to Reviewer yfxK
We thank the reviewer for your valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all your questions.
Q1: Clarification about the position of this paper within related works
A1: We thank the reviewer for pointing out these relevant works. These methods are closely related to TRI-TSMC through reward-guided sampling. Our specific focus is how to learn and iteratively update the twisting function from a limited number of trajectories.
The key contribution of TRI-TSMC is to separate each update into constructing a KL-constrained intermediate trajectory distribution and fitting the twisting function to that distribution. The trust region controls the update size and tempers the fitting weights, allowing more trajectories to contribute to learning. Our theory characterizes the exact distribution update, while Appendix D.6 demonstrates reduced weight concentration during twist fitting. We have expanded the Related Work discussion to make these connections and differences explicit.
Q2: Pathwise stability and its connection to the variance-reduction theory
A2: Our stability analysis focuses on how importance weights are distributed across complete generation trajectories. Theorem 4.5 shows that the normalized variance of the residual importance weights decreases along the exact intermediate distributions as they move toward the target.
Appendix D.6 provides complementary evidence from the twist-fitting step. Under the same sampling and optimization setup, trust-region tempering reduces the fitting-weight CV² from 14.378 to 0.446 and increases ESS/K from 0.0666 to 0.692. The largest normalized weight decreases from 0.973 to 0.171. With 16 trajectories, the effective sample size therefore increases from approximately one to eleven, allowing more trajectories to contribute to the update. PPL also improves from 33.390 to 31.694.
Together, the theory explains the reduction in residual-weight variance under the exact distribution update, while Table 6 demonstrates reduced weight concentration during twist fitting. These weight diagnostics provide evidence of stability beyond average generation performance.
Q3: In Appendix D.6 only the case with and without trust region is compared. Is it possible to address the intermediate values as well?
A3: We provide a trust-region radius sweep in Appendix D.7, Table 7. In the GPT-2 mixed-reward setting, we vary ϵ over (0.10,0.20,0.50) while keeping the remaining settings fixed.
A smaller ϵ imposes a tighter constraint on each update, while a larger ϵ allows a larger change in the proposal. The intermediate value ϵ=0.20 achieves the lowest PPL among these settings.
Together with the controlled ablation in Appendix D.6, these results directly support our claims that trust-region tempering reduces weight concentration during twist fitting and improves alignment performance. Appendix D.6 demonstrates the benefit of introducing the trust region, while Table 7 shows how its strength affects the resulting performance.