acceptodds
Under review as a conference paper at ICLR 2027

VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

Abstract

Masked discrete diffusion models perform strongly on text, code, and biological sequences, but their training objective rewards only naturalness, and retraining the generator for every new reward is expensive. Inference-time steering of a frozen model either guides the sampler by the reward gradient or searches over several trajectories, and recent samplers combine the two. Such combinations are assembled as pipelines that leave three choices at their defaults: a guidance estimate resting on one Gumbel draw per sample, a reward tilting placed without reference to the distribution the combination then targets, and a selection temperature held fixed although the spread of per-step rewards drifts. We identify that distribution and settle the three choices against it. We therefore propose Variance-reduced Guidance and Adaptive Selection (VGAS), a simple yet effective inference-time framework that reduces the variance of the guidance estimate for both reward types, applies the reward tilting in the clean-token logits, where the pretrained schedule is preserved, and sets the selection temperature per step. Across regulatory DNA, protein and small-molecule benchmarks, VGAS attains the best training-free reward and matches or surpasses a reward-fine-tuned generator.

Primary area:
generative models
Keywords:
discrete diffusion, masked diffusion models, inference-time alignment, reward guidance, sequential Monte Carlo, training-free, variance reduction, biological sequence design

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VA8jOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper rigorously examines design choices for an inference-time steering algorithm that uses gradient guidance and particle selection. It suggests the following: 1) adaptive temperature selection, 2) gradient from amortizing over gumbel noise or the leave-out-out reinforce estimator, 3) leaving out proposal correction. The paper contributes a theoretical analysis on 3). It achieves strong empirical results on 3 AI-for-science tasks.

Critical strengths
  1. The strongest aspect of the paper is the results section, where it shows clear empirical improvement on 3 standard tasks. This is useful as it is the ultimate metric by which different methods are judged.

  2. The adaptive temperature selection appears to be novel and generally useful.

Critical weaknesses

I am not sure about the paper's algorithmic novelty. Each of the three things, on its own, seems somewhat straightforward. Combined, it seems like the adaptive temperature selection is the main improvement. But again, that also feels quite straightforward.

Questions for authors

I like the empirical results; this is the strongest part of the paper.

However, I would like a clearer conceptual/algorithmic story. It sounds like you've examined many algorithmic details (e.g. gradient guidance pre or post softmax, adaptive temperature selection, variance over gumbel noise, proposal correction). This is nice. In terms of a major, novel takeaway/result, it seems like that would be adaptive temperature selection? In and of itself, it doesn't seem particularly novel/groundbreaking. But you are getting better empirical results due to this, whereas adding it directly to other methods helps but is also insufficient?

If you could help me understand: 1) is adaptive temperature selection the main contribution, 2) is it sufficient to integrate this with other methods to achieve good performance, or does it uniquely achieve good performance with your method, and 3) an explanation for the answer in 2 - then I'd be able to understand the paper's contributions better.

For now, I am going to select 2 b/c I don't clearly undersand these points. However, I'm quite open to raising my score if you can help me contextualize the paper's contributions and significance.

4 comments
AuthorsOpenReview ↗

Reply to Reviewer VA8j (2/2)

[VA8j] asked whether integrating adaptive temperature selection into other methods is sufficient for good performance, or whether good performance is achieved only with our method.

Thank you for this question. We would kindly refer the reviewer to Table 4b of the original manuscript (Table 17 in the revised one), where AMC is already integrated into SMC-DDM. To test this further on other methods, we added two samplers, SMC and TDS. As shown in Table R1, with all other settings left at their defaults, replacing only the selector by AMC raises the reward of every sampler.

Table R1. Selector replaced by AMC (DNA, 3 seeds). Each sampler keeps its proposal, and the row below it replaces only the selector by AMC. †: re-run together with its AMC variant.

SamplerPred-med ↑ATAC (%) ↑3-mer ↑
SMC †4.09 ± 0.0826.0 ± 10.10.73 ± 0.09
+ AMC5.13 ± 0.2228.0 ± 1.60.78 ± 0.06
TDS †4.75 ± 0.3547.8 ± 7.60.83 ± 0.07
+ AMC5.17 ± 0.1952.1 ± 5.30.86 ± 0.05
SMC-DDM5.71 ± 0.0441.8 ± 0.50.53 ± 0.02
+ AMC5.85 ± 0.0357.5 ± 1.00.90 ± 0.01

AMC therefore generalizes beyond our sampler: with no other setting changed, it raises the reward of a sampler without guidance (SMC), of samplers that keep the proposal correction (TDS, SMC-DDM), and of our own proposal (Table 5a), which we find encouraging. At the same time, AMC alone is not sufficient: with AMC, the other samplers reach 5.1 to 5.9, whereas the best result, 7.33, is obtained with our full design, which validates the design.

Revision: Table R1 replaces the earlier Table 4b and is now Table 5b in Sec. 5.

[VA8j] asked for an explanation of this answer, namely why AMC helps on every sampler and yet the best performance is reached only with our method.

AMC helps on every sampler because all of them prescribe the selection temperature before sampling, and AMC replaces it with a measured one. The best performance requires our method for two reasons.

First, a selector can only choose among the candidates that its proposal produces. Our guided proposal alone, without any selection, already reaches 6.35 (Table 14), above the other samplers with AMC at their default settings (5.13 to 5.85).

Second, the targets differ, where the target is the distribution that the samples of a method follow (Sec. 3.2). TDS and SMC-DDM keep the proposal correction, which cancels the guidance in the target, so only the selection tilts their target toward the reward (Lemma 1). Our method omits the correction, so the guidance and the selection both tilt the target (Proposition 1). Table R2 summarizes this difference.

Table R2. What tilts the target toward the reward in each sampler, with the Pred-med under AMC. ✓: enters the target. △: used in the proposal, but cancelled in the target by the correction. ✗: not used.

SamplerGuidanceSelectionPred-med with AMC
SMC✗✓5.13
TDS△✓5.17
SMC-DDM△✓5.85
Ours✓✓7.33

As Table R2 shows, ours is the only sampler in which both the guidance and the selection enter the target, and it reaches the highest reward under AMC. This is the chain of our first answer: AMC is general, and the best result is reached when it is paired with our guided proposal.

Revision: Sec. 3.2 states this consequence where the effective temperature is introduced.

We hope that this clarifies the contribution and its significance, and we would be glad to run any further ablation during the discussion period.

AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank Reviewer VA8j for the constructive questions on the main contribution and on the role of adaptive temperature selection. Our responses and the revised paper address them as follows:

  • Main contribution as one chain: Sec. 1 and Sec. 3 now present variance-reduced guidance, the omission of the proposal correction, and Adaptive Monte Carlo (AMC) as one chain, in which each step is the reason for the next, with AMC as the main component (new Figure 2).
  • AMC on other samplers: With only the selector replaced by AMC, the reward rises on SMC (4.09 to 5.13), TDS (4.75 to 5.17) and SMC-DDM (5.71 to 5.85), so AMC is general and extends to existing samplers (Table 5b in Sec. 5).
  • Why the best result needs our method: Ours is the only sampler in which both the guidance and the selection enter the target, and it reaches 7.33 under AMC, against 5.13 to 5.85 for the others (Sec. 3.2, Sec. 5).

We appreciate the reviewer's time and would welcome any remaining questions. Changes in the revised paper are marked in blue.

Reviewer_VA8jOpenReview ↗

Reviewer Response

Thanks. I have updated my score to 3.

Upon thinking about it, although the algorithmic novelty isn't huge, I do think the algorithm's simplicity will make it something that stands the test of time & is actually used, compared to very complex techniques that are not compatible with each other and get ignored.

Dear Reviewer VA8j,

Thank you very much for taking the time to read our responses and for raising your score. We are truly grateful for your kind words, and we are especially glad that you see value in the simplicity of our method and in its potential to be used in practice.

We sincerely appreciate your thoughtful evaluation.

Reviewer_mrE1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the problem of reward-guided sampling for masked discrete diffusion models. It analyzes design choices in an SMC framework which uses a gradient-based reward-guided proposal. Namely it proposes: 1) to use a different estimator for the gradient - a Rao-Blackwellized estimate for differentiable rewards, and RLOO for non-differentiable rewards, 2) to use the same SMC weights as if the proposal were the base model, changing the effective temperature of the reward-tilted target, and 3) an adaptive SMC weight temperature which standardizes by the std of the particle set. These choices are validated in experiments over DNA sequence generation, inverse protein folding and small molecule design. They outperform baselines such as regular SMC, and reward-guided sampling.

Critical strengths
  • The adaptive MC approach to dynamically changing the selection temperature in SMC is potentially promising (in particular Fig 2 is interesting)
  • Overall the ideas are organized and presented logically
Critical weaknesses
  • The novelty/originality of the work is limited: the paper makes marginal (and previously studied) improvements to a relatively well-studied approach to reward-alignment in discrete diffusion models:

    • It uses a (gradient) reward guided proposal within SMC which is used in (Ou et al., 2026), and fits into the overall frameworks described in most diffusion-based SMC works eg. FK-Steering (Singhal et al., 2025) and TDS (Wu et al., 2023). The work acknowledges this, so my concern is not of mis-claiming novelty, but rather that the work studies minor aspects of improvement for this setting
  • Some aspects of the chosen methods merit further justification or study:

    • In 3.1, the use of the RLOO estimator for advantage over the standard baseline is not well-motivated.
    • In 3.2 the decision to leave the SMC weights the same as if the proposal was the base model is also not well motivated. The claims in 278-283 should be studied in more detail (ie. is there a real advantage to splitting the effective temperature in this way, compared to just allocating them to either the weights or the reward guidance?)
  • The experimental settings are for smaller problems (DNA sequences, proteins, small molecules) - I think this would be reasonable for a more theoretically oriented work, but since this work is more marginal in that aspect, a more comprehensive experimental evaluation is reasonable (in particular, for larger scale experiments).

  • In experiments some of the advantages of the proposed method are not clear:

    • Do the baselines use a target with the same temperature as the approximate ‘effective temperature’ of VGAS (since a different temperature could change the target and expected statistics)? If not - it is important to compare with matched targets.
    • It is unclear for AMC whether the advantage of the adaptive temperature is due to the more controlled ESS (as in Fig 2) or due to the different target it has. An ablation against an SMC method with a matched target, and along auxiliary metrics such as base-model likelihood, would be helpful here.
  • (less important than the above): there are sections where notation or acronyms are not defined in advance in the paper eg: the AMC acronym appears before its definition - line 310-311

Ou, Z., Pani, C., & Li, Y. (2026). Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Design. arXiv. Retrieved from http://arxiv.org/abs/2505.22524

Singhal, R., Horvitz, Z., Teehan, R., Ren, M., Yu, Z., McKeown, K., & Ranganath, R. (2025). A General Framework for Inference-time Scaling and Steering of Diffusion Models. arXiv. Retrieved from http://arxiv.org/abs/2501.06848

Wu, L., Trippe, B. L., Naesseth, C. A., Blei, D. M., & Cunningham, J. P. (2024). Practical and Asymptotically Exact Conditional Sampling in Diffusion Models. arXiv. Retrieved from http://arxiv.org/abs/2306.17775

Questions for authors
  • How can κ\kappa be estimated in practice?
  • Do the baselines compared to, such as SMC, use the same “effective” temperature as VGAS? ie. κγ+1/α\kappa \gamma + 1/\alpha - or is it a different temperature.
9 comments
AuthorsOpenReview ↗

Reply to Reviewer mrE1 (1/3)

We sincerely thank Reviewer mrE1 for the detailed technical reading, for seeing promise in the adaptive selection temperature, and for finding the ideas logically organized. We answer in three parts.

[mrE1] asked whether the baselines use a target at the same effective temperature as VGAS, and noted that a comparison with matched targets is important.

Thank you for this question. A comparison at matched targets needs a common temperature scale, but each method sets its temperature through its own parameter, so such a scale is not available. We therefore match the methods on the base-model likelihood (App-LL), the auxiliary metric that the reviewer suggested, in two directions (Table R3): (i) the baselines at our likelihood, and (ii) our sampler at the likelihood of SMC and TDS, which do not reach ours at any temperature.

Table R3. Comparison at matched base-model likelihood (DNA, 3 seeds). In (i), SMC-DDM is at its Table 1 setting and TreeG-G is interpolated within its guidance-scale sweep. In (ii), VGAS uses a milder setting, (γ,α)=(30,0.25)(\gamma,\alpha)=(30,0.25) with systematic resampling, and SMC-DDM is brought to this likelihood through its KL weight. The row with the correction kept is VGAS at its Table 1 setting with the proposal correction applied in the weights, which also brings it to this likelihood.

DirectionMethodApp-LLPred-med ↑Margin of VGAS
(i) at our likelihoodVGAS (Table 1)−276.3 ± 0.77.33 ± 0.18
SMC-DDM (Table 1)−278.3 ± 0.25.71 ± 0.04+1.62
TreeG-G−276.3 (matched)5.70 ± 0.01+1.63
(ii) at the likelihood of SMC and TDSVGAS (milder setting)−258.4 ± 1.47.18 ± 0.08
VGAS with the proposal correction kept−257.3 ± 0.46.96 ± 0.08+0.22
TDS at its best α\alpha (highest among all SMC and TDS settings)−255.9 ± 1.25.88 ± 0.13+1.31
SMC-DDM (at this likelihood)−261.8 ± 0.54.82 ± 0.03+2.36

In (i), SMC-DDM is already at our likelihood in Table 1, where VGAS is higher by 1.62, and TreeG-G brought to our likelihood remains 1.63 below. In (ii), VGAS brought to the likelihood of SMC and TDS is higher than the best of all their settings by 1.31, and than SMC-DDM at this likelihood by 2.36. Keeping the proposal correction in our own sampler also lowers the reward at this likelihood (second row of (ii)), which we discuss in part 2/3. The gain therefore persists whenever the likelihood is matched, so it is not explained by a sharper target.

Revision: Table R3 is added to Sec. 5, with the sweeps of TreeG-G, SMC and TDS in the appendix.

[mrE1] asked whether the advantage of AMC comes from the more controlled ESS (Figure 3) or from the different target it has.

Thank you for this question. The advantage comes from the target, not from the ESS. First, AMC does not raise the ESS: at α=0.5\alpha = 0.5, about 8 of 20 particles survive a resampling step, against about 14 under the fixed temperature, since AMC keeps selecting in the late steps (Figure 3b). Second, the ESS of fixed-temperature SMC is set by α\alpha, so if the ESS were the cause, some fixed α\alpha would match AMC. Table R4 tests this: AMC is higher at every α\alpha. What differs is the target: by Proposition 2, the per-step temperature gives a target that no fixed temperature reproduces, at an equal or higher likelihood (Table R5).

Table R4. Pred-med of the same guided proposal under fixed-temperature SMC and under AMC, at each selection temperature α\alpha (DNA, 3 seeds). These are the data of Figure 4.

Selection temperature α\alpha0.050.10.250.5124
Pred-med, fixed-temperature SMC7.127.156.956.866.606.566.44
Pred-med, AMC7.447.467.467.386.926.746.66

Revision: Table R4 is added to Appendix E.5 and cited at Figure 4.

AuthorsOpenReview ↗

Reply to Reviewer mrE1 (2/3)

[mrE1] requested an ablation against an SMC method with a matched target, along auxiliary metrics such as base-model likelihood.

Thank you for this suggestion. We would kindly refer the reviewer to Table 14, where this ablation is already reported together with the base-model likelihood: the proposal, γ\gamma and α\alpha are held fixed, and only the selector is exchanged. Table R5 reproduces these rows.

Table R5. Fixed-temperature SMC versus AMC on the same proposal (DNA, 3 seeds, reproduced from Table 14).

ProposalSelectorPred-med ↑ATAC (%) ↑App-LL ↑
Differentiable (Gumbel-Rao)SMC, fixed α\alpha6.84 ± 0.1597.9−279.3 ± 0.9
AMC7.33 ± 0.1899.1−276.3 ± 0.7
Non-differentiableSMC, fixed α\alpha6.33 ± 0.1086.0−275.8 ± 1.0
AMC6.61 ± 0.0489.5−274.7 ± 0.2

Hence, with the proposal and the selection temperature matched, AMC raises the reward while the base-model likelihood is equal or higher. Table R4 extends the comparison to every fixed temperature.

Revision: Sec. 3.3 now refers to this comparison in Table 14.

[mrE1] noted that leaving the SMC weights as if the proposal were the base model (Sec. 3.2) is not well motivated, and asked whether splitting the effective temperature between guidance and selection has a real advantage over allocating it entirely to the weights or to the guidance (lines 278 to 283).

Thank you for this request. The motivation is where the reward tilt is placed. With the proposal correction, the whole tilt must pass through the selection weights (Lemma 1). Without it, the guidance carries part of the tilt (Proposition 1). Lines 278 to 283 claim that this split matters: weights alone must concentrate on a few particles, whereas guidance first moves the particles toward high-reward regions. As the reviewer asked, we measured this claim directly (Table R6).

Table R6. Where the reward acts (DNA, 3 seeds). Surviving particles: mean effective sample size of the resampling step, out of N=20N{=}20. The first and third rows use systematic resampling.

Reward acts throughγ\gammaα\alphaPred-med ↑ATAC (%) ↑Surviving particles
Selection only00.15.72 ± 0.1526.93.2
Guidance only11000none6.35 ± 0.0187.3no selection
Both1000.57.31 ± 0.0496.37.9
Both (Table 1 setting)110000.57.33 ± 0.1899.17.4 at the last step (Figure 3b)

With selection alone, about 3 of 20 particles survive each resampling step and the reward stops at 5.72. With guidance alone, it reaches 6.35. With both, it reaches 7.3 while 7 to 8 particles survive, at a mild and at the Table 1 guidance strength. Neither part alone reaches the reward of the combination, so the split has a real advantage.

The first alternative of the reviewer, allocating everything to the weights, is to keep the proposal correction. At matched likelihood (Table R3), our own sampler with the correction kept remains below the one without it (6.96 against 7.18), as do TDS and SMC-DDM, which also keep it. Keeping the correction is therefore not the better choice in practice.

Revision: Sec. 3.2 now states this motivation, and a new paragraph in Sec. 5 tests the claim of lines 278 to 283 with Table R6 (Table 8, with the settings in Appendix E.6).

[mrE1] asked how κ\kappa can be estimated in practice.

Thank you for this question. κ\kappa is the scale that converts the guidance strength γ\gamma into inverse temperature (Proposition 1), and it can be estimated in two steps.

  1. During sampling, we accumulate for each sample the log-ratio Φγ\Phi_\gamma between the guided and the unguided probabilities of the tokens it reveals (Eq. (24), closed form in Appendix G.1). This uses quantities that the sampler already computes, with no extra denoiser or reward call.
  2. After sampling, we fit a line Φγ≈κγ,r+c\Phi_\gamma \approx \kappa\gamma\\,r + c over the samples. The slope divided by γ\gamma is κ^\hat\kappa.

Table R7. κ^\hat\kappa on DNA (Φγ\Phi_\gamma regressed on the guidance reward, M=8M{=}8, 3 seeds).

γ\gamma0.250.51241030
κ^\hat\kappa0.1300.1340.1290.1270.1270.1420.141

In Table R7, κ^\hat\kappa stays constant over a 120-fold range of γ\gamma (within 12%), as Proposition 1 predicts: the slope of Φγ\Phi_\gamma grows in proportion to γ\gamma. One estimate therefore suffices within this range. κ\kappa is a diagnostic and is not needed to run the method: γ\gamma is selected on a validation split, and α\alpha is shared across the three domains.

Revision: the procedure and Table R7 are added to Appendix A after Remark 1.

AuthorsOpenReview ↗

Reply to Reviewer mrE1 (3/3)

[mrE1] noted that the use of the RLOO estimator in Sec. 3.1, in place of the standard baseline, is not well motivated.

Thank you for pointing this out. The standard baseline of GILC-PG standardizes each reward by the mean and the standard deviation of its own group (Algorithm 6), so the baseline depends on the sample being weighted. The leave-one-out baseline uses the other n−1n-1 samples instead: it is independent of the sample, keeps the estimate unbiased, and is the natural plug-in for the variance-optimal constant baseline (Lemma 3, Remark 3), at no extra reward evaluation. In Table 19, it raises Pred-med from 6.61 to 6.96. It is thus the counterpart, for non-differentiable rewards, of Gumbel-Rao. We have made this motivation explicit in the main text.

Revision: Sec. 3.1 now states this motivation.

[mrE1] noted that the novelty is limited, as the work makes marginal and previously studied improvements to a well-studied setting, while clarifying that this is not a concern about mis-claiming.

We appreciate this distinction. We agree that the guided proposal within SMC follows prior work (Ou et al., 2026; Singhal et al., 2025; Wu et al., 2023), which the manuscript cites. The contribution is the chain below rather than a single ingredient, and AMC is its main component.

Variance-reduced guidance → target of the combination → adaptive selection (AMC)

  1. The guidance enters the target. Existing samplers retain the proposal correction (Wu et al., 2023; Ou et al., 2026). We omit it, and Lemma 1 and Proposition 1 identify the target that results: the reward-tilted distribution at inverse temperature κγ+1/α\kappa\gamma + 1/\alpha.
  2. The target fixes the role of the selection. The selection must act with the same strength at every step, which a constant α\alpha does not provide, since the spread of the per-step values falls along denoising (Figure 3a).
  3. AMC resolves this. AMC measures this spread from the particles and sets the temperature from it. Existing samplers prescribe the temperature before sampling (Table 27), and adaptive tempering in classical SMC solves for a preset ESS (Appendix F.1, Table 28).

Two results support the significance of this design. AMC is general: it extends to existing samplers (SMC, TDS and SMC-DDM) and improves each of them at their default settings (Table R1 in our reply to Reviewer VA8j). In addition, the full method exceeds the fine-tuned DRAKES in reward on DNA and protein without any training.

Revision: Sec. 1 now states the contribution as this chain, which the new Figure 2 shows.

[mrE1] noted that the experiments are on smaller problems and suggested a more comprehensive evaluation, in particular at larger scale.

Thank you for this suggestion. As a first extension beyond biological sequences, we added an open-ended language experiment: toxicity control with a masked language model pretrained on OpenWebText (MDLM), whose vocabulary is about fifty thousand tokens, more than ten thousand times that of DNA. Table R8 reports the result.

Table R8. Toxicity control on MDLM-OWT (3 seeds). Held-out classifier: a second toxicity classifier, not used as the reward.

MethodReward ↑Toxicity, reward classifier ↑Toxicity, held-out classifier ↑Perplexity ↓
Pretrained−9.26 ± 0.390.020.07122
Best-of-N−9.47 ± 0.230.010.04124
SMC−4.32 ± 0.350.440.49139
GILC-DB−8.60 ± 0.040.040.0785
TreeG-G−2.83 ± 0.030.680.48170
VGAS-GR (ours)−1.98 ± 0.610.690.4759

VGAS raises both the reward and the toxicity under the held-out classifier far above the pretrained model, and its perplexity stays below that of the pretrained model. This shows that VGAS generalizes beyond biological sequences and molecules, to open-ended text.

Revision: this experiment is added as Sec. 4.4 with its table (Table 4), and the protocol is in Appendix I.

Edit (Nov 19): the Best-of-NN row of Table R8 is updated, and the sentence below the table follows the revised Sec. 4.4; the other rows are unchanged. Details are in our follow-up reply to Reviewer mrE1.

[mrE1] noted that some acronyms, such as AMC (lines 310 to 311), are used before they are defined.

Thank you for catching this. AMC is now defined at its first use, and we checked the remaining acronyms and symbols.

We hope that these responses address the concerns of the reviewer, and we would be glad to provide further analysis during the discussion period.

Reviewer_w6WEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses training-free reward alignment in masked discrete diffusion models, where combining gradient guidance with Sequential Monte Carlo (SMC) ususally has high variance, and other problems. The authors propose VGAS, which introduces (1) Gumbel-Rao and RLOO variance-reduced gradients, (2) schedule-preserving logit-space guidance that eliminates heuristic proposal corrections, and (3) an adaptive-temperature selection rule calibrated to per-step reward spread. The method is tested and validated on DNA, protein, and QM9 benchmarks, VGAS consistently outperforms prior inference-time samplers and fine-tuning baselines (e.g., DRAKES).

Critical strengths
  1. Proposition 1 establishes the theoretical grounding with effective inverse temperature κγ+1/α\kappa \gamma + 1/\alpha, providing justification for a previously practicice that is more heuristic
  2. The ideas of Gumbel-Rao and RLOO variance-reduced gradients are both intuitive and somewhat straightforward, since they are quite standard as variance reduction techniques.
  3. These practical techniques can make consistent empirical gains across DNA, protein, and molecular domains without requiring extra denoiser forward passes, which is impressive.
Critical weaknesses
  1. Although VGAS avoids extra base denoiser evaluations, evaluating NN particles alongside MM Gumbel samples and reward backward passes increases latency and memory use. I dont think such additional steps can be trivially light if the authors dont include empirical evaluation on memory and time comparisons against simpler SMC baselines.
  2. It seems all the experiments focus exclusively on biological sequences and chemical graphs. While I can understand that AI4Science is genrally discrete so having much room for MLM, it is well known that many variants of MLM have made it possible to be applied to natural language, i.e. MLDM (and it's dominantly getting more attention than classical MLM). However, it is not evaluated on any primary open-ended natural language generation benchmark.
Questions for authors
  1. Under a fixed wall-clock budget, how should compute be allocated between the particle count NN and the variance-reduction sample budget MM (or nn)?
  2. While I appreciate the theoretical justification, the current presentation needs some major improvements, with pages 3-6 all used for background derivations. I dont think incorporating most fundamentals from past works is essential, and the Proposition 1,2 and Lemma 1,2 are all somewhat hard to follow. I encourage the authors to update presentation of the flow.
3 comments
AuthorsOpenReview ↗

Reply to Reviewer w6WE (1/2)

We sincerely thank Reviewer w6WE for the positive assessment, in particular for recognizing the theoretical grounding of Proposition 1 and the consistent gains obtained without extra denoiser forward passes. We answer in two parts.

[w6WE] noted that evaluating NN particles with nn Gumbel samples and reward backward passes increases latency and memory, and requested an empirical comparison of time and memory against simpler SMC baselines.

Thank you for this request. We measured wall-clock time and peak memory under identical conditions (Table R9).

Table R9. Measured cost on DNA (one A6000, batch of 32 sequences, 128 denoising steps), each baseline at the configuration of its Table 1 result.

MethodTime per sample (s) ↓Peak memory (GB) ↓Pred-med ↑
Pretrained0.040.90.18
SMC0.291.14.26
TDS0.732.74.48
SVDD2.581.15.30
TreeG-G4.648.45.96
SMC-DDM4.9831.35.71
GILC-DB2.206.36.21
GILC-PG1.778.24.86
VGAS-GR (n=1n{=}1)0.572.36.64
VGAS-GR (n=5n{=}5)1.454.57.01
VGAS-GR (n=10n{=}10, as in Table 1)2.466.37.33
VGAS-RO (n=20n{=}20, as in Table 1)1.938.26.96

Among the samplers that combine guidance with selection, VGAS is both faster and lighter: it takes about half the time of TreeG-G and SMC-DDM (2.5 s against 4.6 and 5.0 s) and less memory (6.3 GB against 8.4 and 31.3 GB), at a higher reward. Against the simpler samplers, it is comparable to SVDD in time and slower than SMC and TDS, since it evaluates the reward nn times per step, as the reviewer anticipated. Removing the selection stage (the GILC-DB and GILC-PG rows) leaves the memory unchanged and the time within 0.3 s, so this cost is that of the guided proposal. A smaller nn lowers both time and memory while the reward stays above every baseline of Table 1 (rows n=1n{=}1 and n=5n{=}5). We state this cost in the revised Limitations, and reducing it is left for future work.

Revision: Table R9 is added to Appendix B.1 (Table 12), with a compact version (Table 7) and a paragraph on the inference cost in Sec. 5.

[w6WE] asked how compute should be allocated between the particle count NN and the sample budget nn under a fixed wall-clock budget.

Thank you for this practical question. The two quantities affect the cost differently.

NN does not affect the cost per sample. In our sampler, every particle is returned as a sample, and changing NN from 4 to 32 leaves the time per sample within 9% and the memory unchanged.

nn sets the cost. Table R9 gives the time and memory at n=1n = 1, 55 and 1010, and Table R10 the reward along nn.

Table R10. Reward and memory along nn (DNA, VGAS-GR, 3 seeds). These runs use N=16N{=}16 with systematic resampling, so n=10n{=}10 reads 7.17 here against 7.33 in Table 1. Peak memory is at a batch of 32, as in Table R9.

nnPeak memory (GB) ↓Pred-med ↑ATAC (%) ↑
12.36.64 ± 0.0587.7 ± 1.2
54.57.01 ± 0.0696.4 ± 1.4
10 (as in Table 1)6.37.17 ± 0.0497.6 ± 2.2
209.97.32 ± 0.0698.3 ± 2.2

The reward increases with nn, and most of the gain is obtained by n=5n{=}5 to 1010. Since NN does not change the cost per sample, the budget is spent on nn, and we recommend nn up to about 10. Under a tight budget, n=1n{=}1 takes less than a quarter of the time and about one third of the memory of n=10n{=}10, both below TDS, and still exceeds every baseline of Table 1.

Revision: Table R10 is added to Appendix B.1.

AuthorsOpenReview ↗

Reply to Reviewer w6WE (2/2)

[w6WE] noted that the experiments are limited to biological sequences and chemical graphs, without an open-ended natural language generation benchmark.

Thank you for this suggestion. As a first extension beyond biological sequences, we added toxicity control with a masked language model pretrained on OpenWebText (MDLM), whose vocabulary is about fifty thousand tokens, more than ten thousand times that of DNA. The reward is the log-probability of a toxicity classifier, and a second classifier, not used as the reward, is held out for evaluation (Table R8).

Table R8. Toxicity control on MDLM-OWT (3 seeds).

MethodReward ↑Toxicity, reward classifier ↑Toxicity, held-out classifier ↑Perplexity ↓
Pretrained−9.26 ± 0.390.020.07122
Best-of-N−5.64 ± 1.620.200.31139
SMC−4.32 ± 0.350.440.49139
GILC-DB−8.60 ± 0.040.040.0785
TreeG-G−2.83 ± 0.030.680.48170
VGAS-GR (ours)−1.98 ± 0.610.690.4759

VGAS raises both the reward and the toxicity under the held-out classifier far above the pretrained model, and its perplexity stays below that of the pretrained model. This shows that VGAS generalizes beyond biological sequences and molecules, to open-ended text.

Revision: this experiment is added as Sec. 4.4 with its table (Table 4), and the protocol is in Appendix I.

Edit (Nov 19): the Best-of-NN row of Table R8 is updated; the other rows are unchanged. Details are in our follow-up reply to Reviewer mrE1.

[w6WE] suggested improving the presentation, noting that pages 3 to 6 are devoted to background derivations and that Propositions 1 and 2 and Lemmas 1 and 2 are hard to follow.

Thank you for this feedback. To address it, we revised the presentation as follows.

  1. Background. Sec. 2 is condensed to the parts needed for our results, and the reparameterization inherited from prior work is moved from Sec. 3.1 to the appendix.
  2. Lemmas 1 and 2, Propositions 1 and 2. Each is now preceded by a statement of the result in plain words, and the formal hypotheses are collected in the appendix.
  3. Overview. Sec. 3 opens with a short overview of how the three parts of the method connect, together with a new figure of this chain (Figure 2).

We hope that these responses address the concerns of the reviewer, and we would be glad to discuss any of them further.

AuthorsOpenReview ↗

A Kind Reminder for Comments

We sincerely thank Reviewer w6WE for the constructive feedback on the computational cost, the allocation of the budget, and the scope of the benchmarks. Our responses and the revised paper address these points as follows:

  • Time and memory: We measured every method on the same hardware. VGAS-GR takes 2.46 s per sample with 6.3 GB at its Table 1 setting, and with n=1n{=}1 it takes 0.57 s with 2.3 GB while its reward (6.64) still exceeds every baseline (Table 7 in Sec. 5, Table 12 in Appendix B.1).
  • Allocating NN and nn: A new sweep over nn reports the reward and the memory at each setting, from 6.64 at n=1n{=}1 to 7.32 at n=20n{=}20 (Appendix B.1).
  • Natural language benchmark: A new experiment on toxicity control with a language model (Sec. 4.4, Table 4), where VGAS reaches the highest reward and the lowest perplexity among the compared methods.
  • Presentation: Sec. 1 and Sec. 3 are reorganized around one chain of three steps, each stated with its reason (new Figure 2).

We appreciate the reviewer's time and would welcome any remaining questions. Changes in the revised paper are marked in blue.

Comments on the paper

2 comments
AuthorsOpenReview ↗

A Kind Reminder for Comments

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their constructive feedback and careful evaluation of our work. We have posted point-by-point responses and uploaded a revised manuscript with additional experiments and clarifications that address the concerns raised. The key changes are summarized in our individual responses, and all revisions in the manuscript are marked in blue.

We would greatly appreciate the Area Chair's help in inviting the reviewers to consider our responses and to continue the discussion. We welcome any remaining questions and would be glad to provide further clarification.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

Follow-up Before the Discussion Period Closes

Dear Area Chair and Reviewers,

As the discussion period closes soon, we would like to kindly follow up on our responses and the revised manuscript. We have addressed every point raised in the reviews, and a short summary of the changes is posted under each review.

We would be grateful if the reviewers could let us know whether our responses resolve their concerns. We would also sincerely appreciate the Area Chair's help in encouraging the reviewers to share their feedback before the deadline. We remain available to answer any remaining question promptly.

Thank you very much for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.