acceptodds
Under review as a conference paper at ICLR 2027

Iterative Value-Guided Tilting for Fine-Tuning Discrete Diffusion Models

10 views

Abstract

Discrete diffusion models have demonstrated considerable success across generative tasks ranging from natural language to biological sequences. However, fine-tuning these models on fine-grained black-box rewards, such as binding affinity or molecular property scores, remains challenging. We propose a novel algorithm for discrete diffusion models that requires only black-box reward evaluations and iteratively alternates between learning a soft-value function from samples of the current model and updating the generative model using the resulting value estimates. This iterative procedure progressively steers the model toward higher-reward distributions without requiring reward gradients or a pre-collected reward-labeled dataset. We validate our method on DNA sequence design and protein inverse folding tasks, outperforming comparable methods that rely only on black-box reward evaluations while remaining competitive with approaches that additionally exploit reward gradients or labeled data. Using a simple two-dimensional example, we further illustrate the benefit of successive rounds of value estimation and model optimization over a single optimization round.

Primary area:
generative models
Keywords:
generative models, discrete diffusion, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7GqAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes to progressively tilt discrete diffusion models, motivating their annealing framing as a way to extrapolate towards rewards that steer the model substantially away from the base model. The basic annealing framing aligns closely with Chen et al. 2026 and Wan et al 2026. In each annealing iteration, the authors learn an approximate conditional soft value from rollouts of the current model and then distill this approximation to a further tilted version of the denoiser. A critical part of the approach is the conditional soft value estimation analogously to any other steering technique. Empirical comparisons do not include the most similar methods.

Critical strengths

The approach seems technically clear and well-executed, a combination of ideas from the literature. The algorithms and steps are clearly described.

Critical weaknesses

Theoretical results are clarifying but do not support the annealing approach nor the extrapolation motivation. For example, the bounds blow up with additional annealing steps counter to the motivation that it is helpful to progressively approach the desired tilt. This seems like a consequence of the theoretical approach that relies on bounded errors of each fitting step. The extrapolation to "low density" regions is essentially assumed away by uniform bounds. That said, the method itself aligns well with the motivation even if it is not directly supported by the theory.

Proposition 3.1 is a direct restatement of (9) so perhaps worth restating it as a derivation of the denoising target rather than as a stand alone theoretical "result".

Perhaps the main deficiency of the paper is that there are no comparisons to the most similar approaches (DTM or DGM).

Questions for authors

In section 3, the distilling target is evaluated on re-noised samples from the current model. Since the current model is used to sample clean x_0 in any case, why not use the trajectories from the model also as points where to evaluate the loss? Aren't those exactly the x_t's where the additional tilt corrections would matter? I understand the argument for re-noising but not why the generated trajectories are discarded.

What is the rollout sample efficiency of learning the conditional soft value or the distillation step to any prescribed accuracy? These are the key recurring heavy steps. Theorem 5.1 also relies on these accuracies.

The reward bound directly controls the iteration blowup but appears appropriately as product delta_j B_r in theorem 5.1 suggesting that finer annealing steps might carry some benefit. Further analysis of this effect would be helpful.

4 comments

W2: Proposition 3.1 is a direct restatement of (9) so perhaps worth restating it as a derivation of the denoising target rather than as a stand alone theoretical "result".

We thank the reviewer for this suggestion and will restate Proposition 3.1 accordingly.

W3: Comparisons with DTM and DGM

Regarding DGM, we have reproduced the exact algorithm under comparable computational budgets on DNA and protein, and report the corresponding results below.

To further the analysis, DGM is closely related to doing a single phase of IVT: it learns the same conditional soft value function but instead uses it for inference-time guidance, without distilling the correction into the model. This absence of distillation prevents DGM, in its current formulation, from performing successive tilting phases, an extension that the authors do not consider.

Our single-phase ablation (Fig. 2) therefore also provides a closely related, albeit not identical, proxy for DGM. It allows us to assess the benefits of progressive tilting over a single correction under matched compute budgets. This ablation provides complementary evidence for the benefit of our multi-phase approach, alongside our direct experimental comparison with DGM. We will make this connection between 1-phase IVT and DGM explicit in the main body in the revision.

Regarding DTM, we have reproduced this baseline under matched computational budgets on both DNA and protein benchmarks. We provide results below, along with computational budgets.

To ensure fair comparison, we have made sure to compare our method to the existing baselines in matched compute settings (matched total wall-clock time on both DNA and proteins benchmarks).

  • DNA (§6.1):
MethodRecorded training time (h)Training rolloutsMean HepG2 activity ↑ATAC-positive (%) ↑3-mer correlation ↑JASPAR correlation ↑Mean approximate log likelihood ↑
DRAKES11.41128,0005.44593.390.82720.8971−269.15
DTM10.025,242,8807.11689.060.85700.9083−258.04
IVT9.805,242,8806.97497.340.84080.9332−239.85
DGM11.105,242,8803.4525.520.73680.9181−221.38
  • Proteins (§6.2):
MethodRecorded training time (h)Training rolloutsMean stability ↑Positive stability (%) ↑Foldability (%) ↑Mean backbone RMSD (Å) ↓Joint success (%) ↑
DRAKES4.8750,3000.91286.4191.471.12279.10
DTM4.90260,3521.66891.8086.521.21978.36
IVT4.53260,3521.43797.8181.661.50579.47
DGM4.86260,352−0.38339.8290.521.13737.09

Notably, under these comparable wall-clock time compute budgets, IVT achieves the best results on several key metrics, including ATAC positivity and JASPAR correlation on DNA, as well as positive stability and joint success rate (main evaluation metric) on proteins.

Q1: Why re-noise samples instead of reusing generated trajectories?

We discard the intermediate states because we do not need to keep a particular trajectory to evaluate the loss. What matters is matching the student denoiser to the target at samples Xt∼ptX_t \sim p_t. If the states visited during generation follow ptp_t, they are equally valid points at which to evaluate the tilt correction. We already keep the rewarded X0X_0’s and can cheaply draw a fresh XtX_t each time we reuse one, so we simply re-noise. Empirically, we found that discarding the trajectories / intermediate states did not degrade performance.

Q2: Rollout sample efficiency of value estimation and distillation

We thank the reviewer for raising this interesting question, which concerns the statistical sample complexity of learning the conditional soft value and the distillation steps. Our theoretical analysis instead focuses on the discretisation errors and how they accumulate across successive tilting phases, following a similar approach to early theoretical analyses of diffusion models. Establishing rollout sample-complexity guarantees for achieving prescribed learning and distillation accuracies is an interesting direction that we leave for future work.

Q3: Effect of the tilting increments in Theorem 5.1

W1: Theoretical results are clarifying but do not support the annealing approach nor the extrapolation motivation. For example, the bounds blow up with additional annealing steps counter to the motivation that it is helpful to progressively approach the desired tilt. This seems like a consequence of the theoretical approach that relies on bounded errors of each fitting step. The extrapolation to "low density" regions is essentially assumed away by uniform bounds. That said, the method itself aligns well with the motivation even if it is not directly supported by the theory.

We thank the reviewer for this insightful observation. We agree that the annealing step size involves a trade-off between the error introduced by each tilt update and the accumulation of approximation errors over multiple iterations.

As indicated by Theorem 5.1, the reward-dependent contributions to the one-step error recursion involve factors such as e4δjBre^{4\delta_j B_r}, e6δjBre^{6\delta_j B_r}, and δjBre6δjBr\delta_j B_r e^{6\delta_j B_r}. Thus, reducing the annealing increments decreases these contributions, particularly the term proportional to Ej\sqrt{E_j}.

On the other hand, finer annealing requires more iterations, and the model-fitting and critic-fitting errors accumulate across these iterations. Moreover, the multiplicative coefficient of the existing denoising error does not converge to one as δj\delta_j decreases. Therefore, although smaller annealing increments reduce the reward-dependent error contributions at each step, they do not necessarily improve the overall theoretical bound when the accumulated approximation errors and error propagation are taken into account.

We will clarify this trade-off in the revised version. In particular, our theoretical result suggests that the annealing step size should balance the reward-induced approximation error against the computational and statistical costs of repeatedly fitting the critic and distilling the model.

We also note that a potentially more favorable characterization of the annealing effect may be obtained through an alternative analysis based on the total variation distance, under weaker distributional assumptions. Such an analysis could yield bounds whose dependence on the maximal annealing increment is more explicit and potentially better behaved as the schedule is refined. We have explored this direction, and would be happy to incorporate the corresponding results if the reviewer considers this additional theoretical analysis valuable for the paper. A sharper characterization of the optimal annealing schedule remains an interesting direction for future work.

Thank you for all the clarifications. The results appear a little mixed. I will keep my original slightly positive/borderline assessment.

Dear reviewer, thank you for your time and consideration. We would also like to highlight our additional large-vocabulary text generation experiment, included in the latest revision, in case you missed it.

Under comparable wall-clock training budgets, IVT outperforms DTM on both held-out sentiment and diversity of positive continuations:

MethodTrain reward ↑Held-out sentiment ↑Distinct positive ↑
Pretrained0.3850.1903.55
DTM0.6500.2987.91
IVT0.9190.62221.19

We hope you can also take these additional results into account in your final assessment. Thank you again for your answer and constructive feedback.

Reviewer_NYpKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies the problem of fine-tuning a diffusion model to sample from a reward-tilted variant of its training distribution p(x0)p(x_0). It constructs a sequence of target distributions p(x0)exp⁡(ajr(x0))p(x_0)\exp(a_j r(x_0)) with aja_j a parameter from 0 to a final tilt AA. Training consists of multiple phases during which the denoiser is tuned to sample from the target with parameter aja_j. Each phase consists of drawing samples from the current denoiser, fitting a value function using these rollouts, and then using the value function to update the denoiser to target a higher tilt parameter. The work establishes that this training procedure results in a correct sampler, if all the relevant losses go to 0. The method is evaluated on a synthetic experiment, DNA sequence design, and inverse protein folding. The method generally outperforms other methods which query a black-box reward.

Critical strengths
  • Reward-tilting is an important problem, and any method which can more reliably sample high-reward points with low prior probability, is helpful
  • The ablation of Fig 2 is nice, and establishes the utility of training in multiple phases.
  • Thm 5.1 is a nice consistency result for the overall training procedure
Critical weaknesses
  • The work is missing more detailed comparisons with very closely related baselines in DGM (Wan et al, 2026) and DTM (Chen et al., 2026)
    • In my opinion the discussion of these related works is very important and should be contained in the main text rather than the appendix
    • This method deviates from these in small ways, and it is important to justify these deviations. The method of Wan et al (2026) should be compared to as a baseline. Similarly, the benefits of a learned value function should be tested by comparing to DTM - the paper speculates about a bias-variance trade-off but doesn’t investigate this
  • The presented experiments don’t convincingly establish the utility of the method:
    • It is unclear whether the other baselines have the same target as the current method (eg. the value of AA ) - so its unclear whether some deviations in metrics are due to this or, or due to the methods themselves
    • Additionally, it's unclear whether the baselines for 6.2 and 6.3 are compute-matched. Reporting relevant metrics such wall-clock time, denoiser calls, and reward-calls would be helpful here
    • Larger scale experiments, particularly with a larger vocabulary are needed to demonstrate the scalability of the method - particularly since the soft-value network needs to estimate L x K values (depending on the vocabulary)
  • There are a few issues in terms of presentation of the main text:
    • a discussion of limitations and directions for future work is missing
    • an overall conclusion about the work is missing
    • Some sections such as the preliminaries in Sec 2 contain more detail than is necessary in my opinion (such as details about the LOO denoiser)
Questions for authors
  • Are the baselines in 6.2 and 6.3 compute matched?
  • Do the baselines for DNA and Protein folding target the same distribution as this method (or an approximation to it)?
6 comments

We would like to thank the reviewer for their constructive feedback. We address their concerns below.

W1: Comparisons with DGM (Wan et al., 2026) and DTM (Chen et al., 2026)

Regarding DGM, we have reproduced the exact algorithm under comparable computational budgets on DNA, and report the corresponding results below.

To further the analysis, DGM is closely related to doing a single phase of IVT: it learns the same conditional soft value function but instead uses it for inference-time guidance, without distilling the correction into the model. This absence of distillation prevents DGM, in its current formulation, from performing successive tilting phases, an extension that the authors do not consider.

Our single-phase ablation (Fig. 2) therefore provides a closely related, albeit not identical, proxy for DGM. It allows us to assess the benefits of progressive tilting over a single correction under matched compute budgets. This ablation provides complementary evidence for the benefit of our multi-phase approach, alongside our direct experimental comparison with DGM. We will make this connection between 1-phase IVT and DGM explicit in the main body in the revision.

Regarding DTM, we have reproduced this baseline under matched computational budgets on both DNA and protein benchmarks. We provide results below, along with computational budgets.

W2 / Q1: Fairness of the experimental comparisons and computational budgets

To ensure fair comparison, we have made sure to compare our method to the existing baselines in matched compute settings (matched total wall-clock time on both DNA and proteins benchmarks). In particular, we reproduced DRAKES and additionnally implemented DTM (Chen et al., 2026), a recent and closely related online RL method for discrete diffusion models, under similar total wall clock-times.

We report the corresponding comparisons below:

  • DNA (§6.1):
MethodRecorded training time (h)Training rolloutsMean HepG2 activity ↑ATAC-positive (%) ↑3-mer correlation ↑JASPAR correlation ↑Mean approximate log likelihood ↑
DRAKES11.41128,0005.44593.390.82720.8971−269.15
DTM10.025,242,8807.11689.060.85700.9083−258.04
IVT9.805,242,8806.97497.340.84080.9332−239.85
DGM11.105,242,8803.4525.520.73680.9181−221.38
  • Proteins (§6.2):
MethodRecorded training time (h)Training rolloutsMean stability ↑Positive stability (%) ↑Foldability (%) ↑Mean backbone RMSD (Å) ↓Joint success (%) ↑
DRAKES4.8750,3000.91286.4191.471.12279.10
DTM4.90260,3521.66891.8086.521.21978.36
IVT4.53260,3521.43797.8181.661.50579.47
DGM4.86260,352−0.38339.8290.521.13737.09

Under these comparable wall-clock time compute budgets, IVT achieves the best results on several key metrics, including ATAC positivity and JASPAR correlation on DNA, as well as positive stability and joint success rate (main evaluation metric) on proteins.

Q2: Do the baselines target the same reward-tilted distribution?

The baselines cast the problem through the lenses of reward-tilting. But the value of A is a hyperparameter that was tuned and selected independently for each method. Assuming the baselines were properly tuned, the value of A, and thus the target distribution, depends on the method chosen: the results we report are taken directly from the papers, which we assume to have properly tuned their baselines. As such, not all baselines were evaluated with the same value of A as IVT, but rather with the best A corresponding to each method.

For DTM, which we independently reproduced, we have found the best value of A to be the same as IVT.

W2: Scalability to longer sequences and larger vocabularies

Most existing reward fine-tuning benchmarks for text and code focus on sparse binary rewards, such as correctness. In these settings, the primary objective is typically to obtain a correct answer rather than to sample from a reward-tilted distribution. This makes rejection-sampling fine-tuning a natural approach: generate samples, retain correct ones, and fine-tune on them, although potentially at the cost of reduced diversity.

In contrast, IVT aims to approximate the reward-tilted distribution, preserving diversity while favoring higher-reward samples. This distributional objective is particularly challenging for long sequences, where maintaining coverage of diverse high-reward samples becomes increasingly difficult. We therefore focused on biological sequence generation with continuous-valued reward oracles, where this distributional perspective is particularly relevant. We acknowledge that extending IVT to longer sequences and larger vocabularies remains an important direction for future work. We have launched experiments in these settings, with encouraging preliminary results. However, their substantial computational requirements (in terms of training time and GPU resources) mean that complete results are not yet available. We intend to include these results in the rebuttal once they become available.

Presentation and organization

We thank the reviewer for these suggestions. We will add a conclusion and a discussion of limitations and future directions, move the comparisons with DGM and DTM to the main text and inform you once we have revised our version incorporating these.

I thank the authors for their rebuttal. The revisions made to the paper are important, and in particular the added DTM baseline evaluated at matched compute is very helpful.

I still maintain some concerns:

  1. The selection of AA - this is treated as a hyperparameter tuned for IVT, but it is not clear if this is the way AA is treated in the baselines. In particular, AA is sometimes treated as a 'setting' of the task in some works rather than a hyperparameter (to my knowledge this appears to be the case in DRAKES, which doesn't appear to specify a selection procedure). If it is treated as a selection parameter, there may be multiple ways of doing so - one expects reward to grow as AA increases, but this may reduce diversity - so the exact selection procedure may not be based on reward and can vary. Because of this, AA may not be tuned in a uniform way across baselines. I consider the DTM baseline to be very useful (due to the similar selection of AA), but the other baselines don't necessarily form a matched comparison with the method. Currently, it is difficult to eliminate the confounding role of AA in the baselines.

    • additionally: the criterion used to select AA for IVT doesn't appear to be detailed
  2. The results show mixed results on the DNA task, particularly since DTM seems to be a strong baseline. In my opinion this justifies a closer study of the pros and cons of the differences between the proposed method and DTM

  3. The scalability - I understand that sparse binary correctness rewards are not the target problem for this work. But for instance language with learned rewards (such as through a classifier for eg. helpfulness reward) still appears to fit in the desired problem setting (ie. to sample with high reward while preserving diversity). Currently there does not seem to be a revision of the paper updated with larger scale settings. The importance of such a benchmark is to assess whether training remains feasible in this larger vocabulary setting, or if this is a limitation of the method.

Reviewer_piLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Iterative Value-Guided Tilting (IVT), an algorithm for fine-tuning discrete diffusion models with black-box rewards. IVT progressively tilts the generative distribution across multiple phases. In each phase, the model samples trajectories to fit an intermediate soft-value network under a generalized KL (gKL) loss, and then updates the discrete denoiser parameters toward the locally tilted distribution. The authors provide non-asymptotic bounds tracking total variation error across phases and demonstrate empirical gains over black-box baselines (e.g., REINFORCE, AR-PPO) and single-step tilting on regulatory DNA generation and protein inverse folding (ProteinGym).

Critical strengths
  1. Single-step exponential tilting often fails in discrete combinatorial spaces due to severe importance-weight degeneracy and out-of-distribution drift. It makes sense to decompose into smaller, sequential value-guided steps.

  2. Theorem 1 provides a rigorous phase-by-phase total variation bound that explicitly accounts for both value estimation errors and generative model regression errors across iterations. This provides theoretical justification for the method raised in this work.

Critical weaknesses
  1. First of all, this is a costly move. Training an intermediate value network and subsequently fine-tuning the generative diffusion denoiser at every single phase incurs substantial wall-clock and sample overhead compared to token-level RL baselines or inference-time methods. A clearer discussion of the practical compute tradeoff is needed. This will largely determines if using much more compute anad time is beneficial for much better performance.

  2. While the biological sequence benchmarks (DNA promoter activity and ProteinGym inverse folding) are appropriate, evaluating only short sequences under relatively smooth oracles leaves open whether IVT scales effectively to long discrete sequences (such as code or text).

Questions for authors
  1. How does the total wall-clock training time of IVT compare directly against online RL baselines (e.g., discrete PPO, DRAKES) under the exact same oracle query budget?

  2. How does IVT handle sparse binary rewards (e.g., exact match or strict viability filters) where intermediate soft-value estimation might become flat? It seems the experiments included have relatively continuous and smooth landscapes

2 comments

We would like to thank the reviewer for their constructive comments. We address their concerns below.

W1. First of all, this is a costly move. Training an intermediate value network and subsequently fine-tuning the generative diffusion denoiser at every single phase incurs substantial wall-clock and sample overhead compared to token-level RL baselines or inference-time methods. A clearer discussion of the practical compute tradeoff is needed. This will largely determines if using much more compute anad time is beneficial for much better performance.

Q1. How does the total wall-clock training time of IVT compare directly against online RL baselines (e.g., discrete PPO, DRAKES) under the exact same oracle query budget?

To ensure a fair comparison, we evaluate all methods under comparable total wall-clock training times on both DNA and protein benchmarks. Specifically, we reproduced DRAKES and additionally implemented two closely related methods under comparable compute budgets: DGM (Wan et al., 2026), an inference-time guidance method, and DTM (Chen et al., 2026), an online RL method for discrete diffusion models.

Regarding oracle usage, the best configurations of DTM and IVT use the same number of reward evaluations. DRAKES may be more sample-efficient, but this comes at a significant computational cost: it requires backpropagating through the reward oracle, making each evaluation substantially more expensive. Moreover, this assumes access to reward gradients, which neither IVT nor DTM requires.

Thus, comparing oracle queries alone would overlook both computational cost and differences in oracle access. Evaluating under similar wall-clock budgets provides a more comprehensive comparison of these trade-offs.

Regarding inference-time methods, these are designed to improve generation without further fine-tuning the generative model, whereas IVT aims to fine-tune the model itself. We therefore view these approaches as orthogonal and complementary research directions, rather than directly comparable alternatives. Nevertheless, we agree with the reviewer that this family of methods deserves discussion, and we will include it in the revised manuscript.

As for other methods, lack of available code made it so that we focused on reporting the results available in their papers.

We report the corresponding comparisons below:

  • DNA (§6.1):
MethodRecorded training time (h)Training rolloutsMean HepG2 activity ↑ATAC-positive (%) ↑3-mer correlation ↑JASPAR correlation ↑Mean approximate log likelihood ↑
DRAKES11.41128,0005.44593.390.82720.8971−269.15
DTM10.025,242,8807.11689.060.85700.9083−258.04
IVT9.805,242,8806.97497.340.84080.9332−239.85
DGM11.105,242,8803.4525.520.73680.9181−221.38
  • Proteins (§6.2):
MethodRecorded training time (h)Training rolloutsMean stability ↑Positive stability (%) ↑Foldability (%) ↑Mean backbone RMSD (Å) ↓Joint success (%) ↑
DRAKES4.8750,3000.91286.4191.471.12279.10
DTM4.90260,3521.66891.8086.521.21978.36
IVT4.53260,3521.43797.8181.661.50579.47
DGM4.86260,352−0.38339.8290.521.13737.09

This highlights an important trade-off: DRAKES may be more oracle-efficient, whereas IVT and DTM can exploit substantially more oracle evaluations within the same computational budget. Notably, under these comparable computational budgets, IVT achieves the best results on several key metrics, including ATAC positivity and JASPAR correlation on DNA, as well as positive stability and joint success rate (main evaluation metric) on proteins.

Q2. How does IVT handle sparse binary rewards (e.g., exact match or strict viability filters) where intermediate soft-value estimation might become flat? It seems the experiments included have relatively continuous and smooth landscapes

Our method was designed specifically for continuous-valued, dense reward, as stated in our problem setting. We do not believe that framing the problem as reward guidance (i.e. targeting p0A(x0)∝p0base(x0)exp(Ar(x0))p_0^A(x_0) \propto p_0^{base}(x_0) exp(Ar(x_0)) with AA an hyperparameter to define, is the right framing for binary rewards. In fact, in such settings, the actual target we are interested in is rather the distribution conditioned on success p0target(x0)∝p0base(x0)1r(x0)=1p_0^{\mathrm{target}}(x_0)\propto p_0^{\mathrm{base}}(x_0)\mathbf{1}{r(x_0)=1}. Exponential reward tilting is a proxy to this distribution when A→+∞A \to + \infty only.

When considering sparse binary rewards, this ideal target can be approached more directly through rejection-sampling fine-tuning: sampling from the current model, retaining only the samples satisfying the correctness/viability criterion (r(x0)=1r(x_0) = 1), and fine-tuning on these samples, and redoing this over multiple iterations.

W2. While the biological sequence benchmarks (DNA promoter activity and ProteinGym inverse folding) are appropriate, evaluating only short sequences under relatively smooth oracles leaves open whether IVT scales effectively to long discrete sequences (such as code or text).

Most existing reward fine-tuning benchmarks for text and code focus on sparse binary rewards, such as correctness. In these settings, the primary objective is typically to obtain a correct answer rather than to sample from a reward-tilted distribution. As discussed in Q2, this makes rejection-sampling fine-tuning a natural and effective approach, although potentially at the cost of reduced sample diversity.

In contrast, IVT aims to approximate the reward-tilted distribution, preserving diversity while favoring higher-reward samples. This distributional objective is particularly challenging for long sequences, where maintaining coverage of diverse high-reward samples becomes increasingly difficult. We therefore focused on biological sequence generation with continuous-valued reward oracles, where this distributional perspective is particularly relevant. We acknowledge that extending IVT to longer sequences and larger vocabularies remains an important direction for future work. We have launched experiments in these settings, with encouraging preliminary results. However, their substantial computational requirements (in terms of training time and GPU resources) mean that complete results are not yet available. We intend to include these results in the rebuttal once they become available.

Comments on the paper

2 comments

Dear reviewers,

Thank you again for your thoughtful feedback and for taking the time to review our paper. We have revised the manuscript in response to your comments and marked the main changes in purple.

The revised manuscript includes comparisons with DTM and DGM on both DNA and protein benchmarks under comparable wall-clock training budgets. We have also updated the tables in our rebuttal responses to include the protein DGM results that were previously pending. We added a Related Work section explaining the connection between DGM and single-phase IVT and a conclusion discussing limitations and future work.

If you have a chance to read our responses, we’d appreciate hearing whether they address your main concerns or whether any points remain open. A brief note on any point that would benefit from further clarification would be very helpful, and we’d be happy to follow up here.

Best,
The authors

AuthorsOpenReview ↗

2nd rebuttal revision - Addition of a text generation experiment

Dear reviewers,

A brief follow-up to let you know that we have further updated the revised manuscript to include an additional text generation experiment with a large vocabulary (50,257 tokens). In this experiment, we fine-tune a pretrained masked diffusion language model to generate positive continuations using classifier-based rewards, and evaluate the results using a separate sentiment classifier. IVT outperforms DTM under comparable wall-clock training budgets in this experiment, achieving higher held-out sentiment (0.622 vs. 0.298) and greater diversity of positive continuations (21.19 vs. 7.91).

The changes are highlighted in purple. We hope these additional results are helpful and would be happy to discuss them further or address any remaining concerns.

Thank you again for your time and valuable feedback.

Best,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.