- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents an MCMC algorithm for sampling from a tilted distribution of discrete diffusion models, i.e. p(x) e^{-r(x)} where p(x) is a pretrained diffusion model and r(x) is an external reward model. The key motivation is that many reward tilting methods rely on denoising predictions which may not land in the clean data distribution space, especially for discrete data modality. The proposed method instead construct a diffusion-based proposal by first denoise a noisy sample and then renoise it and derive a Metropolis Hasting (MH) kernel. While the MH kernel is intractable, they develop tractable approximations. In the experiment they demonstrate better pareto frontier in {performance metric, inference time} across inference-time steering methods on peptide generation and biological sequence generation.
Critical strengths
-
The paper is written very nicely and easy to follow.
-
The proposed method is easy to implement; empirically, it seems to work well.
-
THe proposed method does not require differentiable reward, nor it requires backpropagating through the denoiser network which can be memory efficient.
Critical weaknesses
-
My biggest concern of the paper is the novelty. There are a few existing works earlier in the year that use a very similar denoise-and-renoise MCMC kernel for reward tilting or for sampling from unnormalized densities [1,2,3]
-
I also didn't find a related work discussion.
-
The MH kernel approximation deserves some more discussion rather than calling it a simplified expression, as it depends on the diffusion model training error and the SDE discretization error.
-
On a related note, I also found theorem 1 more or less a standard result of training a discriminator to estimate a density ratio. Also the estimator behind thm. 1 is not really used so I'd consider Thm. 1 a bit cosmetic. I suggest the authors to remove it unless it's proven to be empirically relevant.
[1] Sampling Data with Chains of Forward-Backward Diffusion Steps Hyunmo Kang, Noam Itzhak Levi, Corinna Elena Wegner, Daniel J. Korchinski, Matthieu Wyart
[2] Exact Global MCMC with Denoising Diffusion Mitch Hill
[3] Markov Chain Monte Carlo with Diffusion Paths Han Chen, Sifan Liu, Jun Yang
Questions for authors
-
Can the authors report the MH acceptance ratio across sampling steps as well as diffusion time in the reported experiments? It'd be helpful to gauge the efficiency of the method.
-
A related question is in the scenario where the base model has low probability in high reward region. As the proposed method does not utilize gradient information, how would the authors expect its performance when compared to gradient-based methods?
-
The results in Fig. 3 look promising. However, I wonder why if the authors can provide some insights on why the proposed method have better results given the same runtime?
4 comments
Response (2/2)
Q1. Can the authors report the MH acceptance ratio across sampling steps as well as diffusion time t in the reported experiments?
We report the acceptance rate across sampling steps (iterations) and diffusion time in Fig. E.1 (page 27) of the revised version. As shown in the figure, the acceptance rate is initially high as the proposed samples are more likely to have higher rewards than the current sample. The acceptance rate decays over the iterations and converges to around 15%. The acceptance rate is also higher for smaller forward noising times as it encourages local exploitation around the current sample.
Q2. How does the authors expect the performance to compare to gradient-based methods?
We thank the reviewer for bringing up this point. First, we would like to note that it is non-trivial to extend CSMC to utilize gradient information since the clean-sample score is intractable (which is why score-based diffusion models use the noisy score ) instead. However, even if gradient information is available, these methods may not be useful due to (1) gradient information not helping in exploring disconnected modes and (2) backpropagation being computationally expensive. As we have shown in Fig. 3, gradient-based SMC does not perform well due to the computational bottleneck introduced by backpropagation.
Q3. Why does CSMC have better results given the same runtime?
As we have explained in L458-461, CSMC is more efficient than BoN since CSMC provides intermediate guidance via the forward-reverse diffusion proposal and the accept/reject step. This allows CSMC to efficiently explore locally around high-reward samples. CSMC is also more efficient than methods based on intermediate rewards (SMC, SVDD, RERD) since inaccurate intermediate rewards may mistakenly discourage exploration around high-reward regions (L170-171). Finally, although DTS also uses clean rewards, it requires building an entire search tree which is less efficient than CSMC's forward-reverse diffusion proposal (L455-457).
[1] Kang, Hyunmo, et al. "Sampling Data with Chains of Forward-Backward Diffusion Steps." arXiv preprint arXiv:2605.27006 (2026).
[2] Hill, Mitch. "Exact Global MCMC with Denoising Diffusion." arXiv preprint arXiv:2609.00279 (2026).
[3] Chen, Han, Sifan Liu, and Jun Yang. "Markov Chain Monte Carlo with Diffusion Paths." arXiv preprint arXiv:2607.11631 (2026).
Since less than two days are left in the reviewer-author discussion, we would like to kindly remind the reviewer to let us know if your concerns are addressed or if there are any remaining concerns after our response. Thank you.
I would like to thank the authors for their reply. I have increased my score to 3.
Response (1/2)
We thank the reviewer for the review. We address the concerns below.
W1. Novelty and relation to existing works.
We thank the reviewer for bringing these works to our attention. Although these works are related, we would like to first point out that the cited works are recent unpublished works which should not warrant a basis for rejection based on the ICLR reviewer guidelines (last FAQ). Furthermore, [2] and [3] are theoretical works that only consider regular diffusion sampling on toy datasets and do not consider the task of reward-guided discrete diffusion. As such, the setting is very different from our work.
The main contribution of our paper is also different from [1]. Our main contribution is reformulating reward-guided discrete diffusion sampling as an iterative search over clean samples. This challenges the current paradigm where guidance is performed along the denoising trajectory and rely on intermediate rewards, which we show to be unreliable for scientific tasks. In contrast, [1] does not identify the intermediate reward bottleneck of reward-guided diffusion samplers and instead focus on analyzing the properties of MCMC with forward-backward diffusion proposal.
Regardless, we have included a discussion of these works in the related work section (App. C) of the revised version.
W2. Missing related work
We thank the reviewer for bringing this to our attention. In the original draft of the paper, the related work discussion was combined with the background section. In Sec. 4 and App. C of the revised version, we have added a separate related work section discussing training-based guidance methods, training-free guidance methods, and MCMC diffusion samplers.
W3. Discussion on the MH kernel approximation
As the reviewer correctly notes, the theoretical guarantee depends on the diffusion model training error and number of integration steps during sampling. These are practical considerations which may affect the stationary distribution of CSMC. We additionally report the discrepancy between the simplified and discriminator-corrected acceptance probabilities for the target chain with β=10−5 in Fig. B.3 (page 22). As shown, the acceptance probabilities are very similar for both cases for almost all CSMC iterations. The log acceptance probability in Eq. 2 is affected by two terms, the temperature-weighted reward difference and the discriminator correction term. When the temperature is low (which is the case in practice), the contribution of the first term βr(x′)−r(x) significantly outweighs the discriminator term and the discriminator term can be essentially ignored. While we acknowledge that parallel tempering uses some higher temperature chains, our target chain uses low temperature which supports the use of our simplified acceptance probability.
W4. Theorem 3.1 is a bit cosmetic.
We thank the reviewer for the suggestion. We have included theorem 3.1 for completeness since it is used when comparing the simplified and discriminator-corrected acceptance probabilities. With our new discussion, we have a direct comparison between the acceptance probabilities in Fig. B.3. However, if the reviewer still believes it is better to move theorem 3.1 to the appendix, we will do so in the revised version.