acceptodds
Under review as a conference paper at ICLR 2027

IDRF: Inverse-Distilled Reward Fine-Tuning of Masked Discrete Diffusion Models

Abstract

Masked discrete diffusion models offer a promising alternative to autoregressive generation, but iterative sampling can be costly, and intractable sequence likelihoods complicate reward fine-tuning. We introduce IDRF, a framework for reward fine-tuning of few-step masked discrete diffusion generators. Starting from a standard reverse-KL-regularized reward objective, IDRF replaces the intractable sequence-level KL penalty with an inverse-distillation regularizer. We prove that the population inverse-distillation loss upper-bounds the KL between complete sequences of any length. IDRF optimizes a trajectory-based surrogate of this loss without reference-model rollouts, so the student keeps its own few-step sampler. We view few-step generation as a finite-horizon Markov decision process and opti- mize reward with a clipped policy-gradient objective over the student’s denoising trajectories. We evaluate IDRF across biological, text, and image domains. In each domain, it reaches high reward with up to 32× fewer sampling steps and avoids the reward hacking that reward-only fine-tuning exhibits

Primary area:
generative models
Keywords:
masked discrete diffusion, reward fine-tuning, inverse distillation, few-step generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_cppWOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Practical reward fine-tuning algorithms often use a KL regularization term which ensures that the learned policy does not drift “too far” from some reference one. For discrete diffusion models, this regularization term is intractable. This work demonstrates that the inverse distillation loss from IDLM [Li et al., 2026] upper bounds this KL regularization when we replace data distributions from IDLM with the distribution induced by the reference policy. This provides a principled motivation for reward fine-tuning with the new proposed IDRF loss which replaces the KL with the IDLM-inspired loss. The authors demonstrate that reward fine-tuning with their proposed loss allows for competitive few step reward-guided generation across several domains.

Critical strengths
  • This paper deals with an important problem for masked diffusion models.
  • The results demonstrate strong small NFE reward guidance compared to other methods.
Critical weaknesses

Theoretical derivation / Method presentation

I found parts of the Methods section to be hard to follow. Below I detail some examples of issues I had. I recommend that the authors restructure this section to better motivate and explain their methodology.

  • Given that the main theoretical contribution is extending the IDLM [Li et al., 2026] result from one token to full sequences (and then applying it to prefp_{\text{ref}} as opposed to pdatap_{\text{data}}), in the main text, it would be useful for the authors to provide intuition regarding why the original IDLM result fails to establish the practical sequence-level result and how the current work gets around / resolves that issue. Perhaps I am missing something but in practice LID\mathcal{L}_{\mathrm{ID}} from equation 7 in the current manuscript seems to be equivalent to summing token level losses from IDLM.
  • The importance weighting term ρikℓ\rho_{ik\ell} is not well introduced nor is it sufficiently motivated that it the fully factorized ratio of GθG_\theta and GθoldG_{\theta_{\text{old}}} is valid. For example, the derivation in d2 [Wang et al., 2026] is more comprehensive in this respect.
  • I did not understand the purpose nor the placement of Section 3.2 in the methods section. I do not see how the MDP formulation contributes to the overall presentation of the method. As far as I can tell the “practical objective” and the training algorithm could presented almost identically even without Section 3.2.

Baselines

I think the authors should have a more systematic approach to selecting baselines. While I understand that each experiment section comes with prior work, e.g., DRAKES in the DNA experiments, it would be better to consistently compare to similar Diffusion Reward fine-tuning methods. Specifically, I believe the authors should consistently compare to a reasonable subset of the following methods:

  • d1 [Zhao et al., 2025], wd1 [Tang et al., 2025], coupledGRPO [Gong et al., 2025]
  • TraceRL [Wang et al., 2025] / d2 [Wang et al., 2026] (the latter of which was also omitted from the Related Works section and should be included).

I recognize that some of these, e.g., diffuGRPO, are already included in the DNA experiment, but adding them in the image and NLP experiments would substantially strengthen the empirical results in my opinion.

Questions for authors
  • The ID loss upper bounds KL and is therefore motivated as a useful surrogate for regularized reward fine-tuning. However it would be interesting to see a targeted experiment / analysis comparing ID with KL in cases where the KL is indeed tractable, e.g., for true any-order diffusion LMs, as explained in d2 [Wang et al., 2026], or perhaps in a smaller setting where KL can be computed without prohibitive computation resources.
  • I am curious if the authors tried this method on a large DLM like LLaDA / Dream7B for benchmark tasks, i.e., following the setup of d1, wd1, d2, etc. If so, are there preliminary results that can be added to this work?
5 comments
AuthorsOpenReview ↗

Response to Reviewer cppW (1/4)

Dear Reviewer, thank you for your careful review and constructive suggestions. We are glad that you find the problem important and the few-step results strong. Your comments made the method section clearer and the comparison more systematic. All changes are marked in blue in the revised paper. Below we respond to each point.

[W1.a] Why the IDLM result does not cover the practical sequence-level loss, and how Proposition 1 resolves it.

Thank you for raising this. You read Eq. (7) correctly. It is the same sum of token-level losses that IDLM [1] trains with. We keep the loss unchanged. Our contribution is the guarantee for it.

What IDLM proves. IDLM proves its bound for a single token. It notes that the same argument applies if the whole sequence is treated as one state with a joint reverse process. For the sum of token-level losses used in practice, it leaves the analysis to future work ([1], App. D.2), because the tokens of a sequence are dependent. Applying the single-token result position by position controls each token separately, and the joint law of the sequence stays open.

How Proposition 1 closes the gap. IDLM argues through path measures, a general view that covers several kinds of discrete diffusion. To close the gap, we specialize to masked diffusion and view it as any-order autoregression. This narrower view allows us to show the following. In continuous time the reverse process reveals the masks one at a time, in a uniformly random order σ\sigma. Let πf,σ(x)\pi_{f,\sigma}(x) be the probability of generating xx in this order with denoiser ff. Then the NELBO is an autoregressive log-loss averaged over orders, and the law of the reference sampler is the same average:

LNELBO(f,pθ)=EσEx∼pθ[−log⁡πf,σ(x)],pref=Eσπfref,σ.\mathcal L_{\mathrm{NELBO}}(f,p_\theta)=\mathbb E_{\sigma} \mathbb E_{x\sim p_\theta}\big[-\log \pi_{f,\sigma}(x)\big],\qquad p_{\mathrm{ref}}=\mathbb E_{\sigma} \pi_{f_{\mathrm{ref}},\sigma}.

The Bayes denoiser of pθp_\theta generates pθp_\theta in every order, so the inner minimum is the entropy of pθp_\theta. What remains is a KL between full sequences for each order, and convexity of the KL gives

LID(θ)=EσDKL(pθ∥πfref,σ) ≥ DKL(pθ∥pref).\mathcal L_{\mathrm{ID}}(\theta)=\mathbb E_{\sigma} D_{\mathrm{KL}}\big(p_\theta \Vert \pi_{f_{\mathrm{ref}},\sigma}\big)\ \ge\ D_{\mathrm{KL}}\big(p_\theta \Vert p_{\mathrm{ref}}\big).

Averaging over reveal orders is the step that extends the single-token bound to dependent tokens. The bound involves only the law pθp_\theta, so it holds for any sequence length and any student sampler.

From p∗p^\ast to prefp_{\mathrm{ref}}. IDLM assumes a reference that is optimal for the data. Proposition 1 holds for any reference and bounds the KL to prefp_{\mathrm{ref}}, the law of the reference's own sampler. Equality holds when the reference is the Bayes denoiser of that law. This is the case that matters for fine-tuning, where the reference is a trained network.

In the revised paper. We agree that this intuition belongs in the main text. Section 3.1 now contains the paragraph "Why the bound holds" with the two identities above, and Section 2.2 states what IDLM leaves open. The full proof is in Appendix A.

AuthorsOpenReview ↗

Response to Reviewer cppW (2/4)

[W1.b] The importance weight, and why the fully factorized ratio is valid.

We agree that the ratio was introduced too quickly. Section 3.2 now derives it from the sampler.

Rollout probability. The student is the standard ancestral sampler run with KK steps. At step kk each masked position is revealed with a fixed probability bkb_k, and its token is drawn from Gθℓ(⋅∣xtk)G_\theta^\ell(\cdot\mid x_{t_k}). The denoiser never predicts the mask token, so a masked position either stays masked or is revealed as a clean token. With UkU_k the set of positions revealed at step kk, the probability of a rollout τ\tau is

Pθ(τ)=c(τ)∏k=1K∏ℓ∈UkGθℓ(x0ℓ∣xtk),P_\theta(\tau)=c(\tau)\prod_{k=1}^{K}\prod_{\ell\in U_k}G_\theta^\ell\big(x_0^\ell\mid x_{t_k}\big),

where c(τ)c(\tau) collects the factors bkb_k and 1−bk1-b_k and does not depend on θ\theta (Eq. 9, with the derivation in Appendix B).

The ratio. The factor c(τ)c(\tau) cancels, so the likelihood ratio between the current and the old student is exactly a product of per-token ratios:

Pθ(τ)Pθold(τ)=∏k=1K∏ℓ∈Ukρkℓ,ρkℓ=Gθℓ(x0ℓ∣xtk)Gθoldℓ(x0ℓ∣xtk).\frac{P_\theta(\tau)}{P_{\theta_{\mathrm{old}}}(\tau)}=\prod_{k=1}^{K}\prod_{\ell\in U_k}\rho_{k\ell},\qquad \rho_{k\ell}=\frac{G_\theta^\ell(x_0^\ell\mid x_{t_k})}{G_{\theta_{\mathrm{old}}}^\ell(x_0^\ell\mid x_{t_k})}.

This is the trajectory likelihood of d2 [2] with one segment per sampling step. We state this in Appendices C.6 and E.2, where d2 is run in this exact setting.

How we use it. The full product is exact and has high variance, since it multiplies up to LL factors. As in PPO, Eq. (10) sums the per-token ratios and clips each one. At θ=θold\theta=\theta_{\mathrm{old}} this has the same gradient as the product (Appendix B). The ratios only matter when a rollout batch is reused. This is the case for DNA, with two updates per batch. For images and text each batch serves one update, every ratio equals one, and Eq. (10) is plain RLOO. The revised Section 3.2 states this explicitly.

[W1.c] Purpose and placement of Section 3.2 (MDP formulation).

Thank you for pointing this out. We agree that the purpose of this section may have sounded unclear.

Why the MDP view matters. The likelihood of a sequence, pθ(x0)p_\theta(x_0), is intractable, because it sums over all unmasking trajectories. Viewed as an MDP, the sampler has states xtkx_{t_k} and a policy with tractable per-step probabilities. The log-probability of a whole trajectory, log⁡Pθ(τ)\log P_\theta(\tau), can therefore be computed exactly (Eq. 9), and the policy gradient of the reward needs only this quantity.

In the revised paper. To make this section easier to read and understand, we restructured it and clarified the unclear points. Section 3.2 is now "Training the few-step student". It introduces the few-step sampler, derives the rollout probability, and builds the reward term on it. The MDP view is stated as a remark with a reference to AGRPO [7].

AuthorsOpenReview ↗

Response to Reviewer cppW (3/4)

[W2] A systematic comparison with diffusion reward fine-tuning methods.

We agree that a consistent set of baselines strengthens the results. We implemented d1 [3], wd1 [4] and d2 [2] in our codebase and trained them in all three domains.

Setup. Every baseline uses the same few-step sampler, initialization, rollouts, optimizer and evaluation as IDRF. Only the loss changes. d1 and wd1 score tokens at the fully masked state. d2 uses one segment per sampling step, which makes its trajectory likelihood exact. On images it scores four of the five transitions per update, as our reward loss does. Each row is a single training run, as for our own rows. For DNA and images, d1 and d2 use β=0.04\beta=0.04, the value reported for both methods ([2], [4]), and wd1 its best ψ\psi. For text, each baseline is shown at the weight whose reward is closest to that of IDRF. Full sweeps over β\beta and ψ\psi are in Appendices C.6, D.1 and E.2, with samples for images and text. In the tables below, bold marks the best value among d1, wd1, d2 and IDRF.

Table A: Regulatory-DNA design, 8 sampling steps (rows of Table 1 in the paper).

MethodPred-Activity ↑ATAC-Acc (%) ↑3-mer Corr. ↑
RLOO w/o LID\mathcal L_{\mathrm{ID}}7.6837.80.378
d14.1417.30.888
wd15.040.50.696
d27.2689.20.735
IDRF (ours)7.2295.30.683

Table B: CLIP-guided MaskGIT, 4 sampling steps (rows of Table 2). CLIP is the optimized reward, and the other metrics are held out.

MethodCLIP ↑ImageReward ↑PickScore ↑HPSv2.1 (×100) ↑
Reference, 4 steps14.00−2.1718.2414.67
RLOO w/o LID\mathcal L_{\mathrm{ID}}26.12−0.4920.6122.57
wd130.100.1920.9022.44
d124.59−0.7320.1821.14
d219.28−1.8019.1517.91
IDRF (ours)27.740.8822.1127.72

Table C: Sentiment steering of review text, 32 sampling steps (rows of Table 3). log⁡P(1⋆)\log P(1\star) is the optimized reward.

Methodlog⁡P(1⋆)\log P(1\star) ↑GenPPL ↓LengthEOS (%) ↑GM ↓
Real one-star reviews−0.95053.332.2100.00.03
Reference, 128 steps−5.07091.727.8100.02.45
RLOO w/o LID\mathcal L_{\mathrm{ID}}−0.00876.9126.89.428.81
wd1−0.01459.7124.488.522.69
d2−0.038139.770.2100.010.07
d1−0.077107.664.5100.08.22
IDRF (ours)−0.04465.444.1100.010.86

Results.

  • DNA. d1 and wd1 stay far from IDRF, in line with the 128-step diffu-GRPO row that Table 1 already contained (activity 5.86, ATAC-Acc 33.0%). d2 comes close, with activity 7.26 and ATAC-Acc 89.2%, against 7.22 and 95.3% for IDRF.
  • Images. wd1 reaches a higher CLIP than IDRF (30.10 against 27.74) and is below it on every held-out metric, with ImageReward 0.19 against 0.88. d1 and d2 stay below the reward-only ablation. The difference is clearly visible in the samples (Figure 4 and Figure 7 in Appendix D.1, the first sample of each class with no selection). IDRF produces clear objects that match the prompt. The wd1 images are dominated by the prompt colour with little object detail, and the d1 and d2 images are cluttered with no clear or very fuzzy objects.
  • Text. d1 and d2 keep EOS at 100%, and their GenPPL (108 and 140) is above that of the reference (91.7). wd1 writes reviews about four times as long as real ones. IDRF is the only fine-tuned model that is close to real reviews in both GenPPL and length.

Each baseline falls short in at least one domain, while IDRF is strong in all three.

Why the baselines behave this way. d1 and d2 both regularize with a token-level KL to the reference network. d1 evaluates it at the fully masked state, and d2 at every state of the rollout. In both cases the term is minimized by the reference network itself, so a larger β\beta moves the student back toward the reference sampled in the same few steps. On images this sampler is weak at 4 steps (first row of Table B), and at β=1\beta=1 both methods return to its level (Appendix D.1). LID\mathcal L_{\mathrm{ID}} regularizes toward prefp_{\mathrm{ref}} itself (Proposition 1, Corollary 1). It needs one pass through the reference per rollout for any KK, while the KL of d2 evaluates the reference on every segment, KK passes in this setting. We added this comparison to Section 3.2.

Related Work. d2 and coupled-GRPO [5] are added to Section 4, next to d1, wd1 and TraceRL [6].

Reviewer_jbTFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IDRF for reward fine-tuning of masked discrete diffusion models into few-step generators. The K-step sampler is cast as a finite-horizon MDP and optimized with an RLOO-style clipped policy-gradient surrogate, regularized by an inverse-distillation gap between a frozen reference and a trainable auxiliary using the student's own rollout states. Proposition 1 shows that the population inverse-distillation loss, under an optimal auxiliary, upper-bounds the sequence-level KL from the student to the reference's continuous-time endpoint distribution, for arbitrary sequence lengths and student samplers. The practical regularizer instead applies to a mixture of one-step completion laws across rollout depths, of which the endpoint law is one component. Experiments on DNA design, CLIP-guided image generation, and review sentiment steering show that IDRF achieves higher rewards than the references with fewer sampling steps, while better preserving quality than same-step reward-only fine-tuning. The DNA and text experiments also illustrate degeneration that the optimized rewards and likelihood-based evaluation metrics fail to reveal.

Critical strengths

Sequence-Level Justification for Inverse-Distillation Regularization: Proposition 1 extends the inverse-distillation bound to sequences of arbitrary length and any student sampler, with an arbitrary reference denoiser and an optimal auxiliary. The random-reveal-order proof provides a clear connection between the population inverse-distillation loss and the sequence-level KL. This gives a principled motivation for regularizing reward fine-tuning when sequence likelihoods are intractable, while allowing the student to retain its own few-step sampler.

Cross-Domain Evidence of Reduced Reward Hacking: The experiments cover regulatory-DNA design, CLIP-guided image generation, and review sentiment steering, and evaluate properties beyond the optimized rewards. Compared with reward-only fine-tuning at the same sampling budget, IDRF improves DNA ATAC-Acc from 37.8% to 95.3%, achieves higher scores on all three held-out image reward models, and preserves EOS emission and reduces repetition in text. The regularization sweep and on-policy-state ablation further support the practical contribution of the inverse-distillation term.

Critical weaknesses

Limited Matched Comparisons with Alternative Regularization Strategies: The same-step reward-only ablations demonstrate the benefit of the inverse-distillation term, but do not establish its advantage over reverse-transition KL regularization at the same sampling budget or over inverse distillation followed by reward fine-tuning. Such comparisons would help distinguish the benefits of inverse-distillation regularization from those of reference regularization or few-step distillation more generally. For review sentiment steering, a reward-matched early-stopping baseline would also be informative, since reward saturation precedes the reported degeneration.

Incomplete Evaluation of Diversity and Robustness: Section 6 acknowledges that IDRF produces less diverse MaskGIT samples than the reference, but this reduction is not quantified. The per-sequence nucleotide entropy and normalized token entropy reported for DNA and text characterize within-sample composition and repetition, rather than diversity across samples. Consequently, the extent to which IDRF preserves distributional coverage remains unclear. The image evaluation also covers only ten fixed class-prompt pairs, and the reported variability does not establish robustness across independent training runs: the DNA standard deviations are computed over sampling seeds for fixed checkpoints, while the IDRF configurations use single training seeds.

Questions for authors

Matched Regularization Comparisons: How does IDRF compare with reverse-transition KL regularization at the same sampling budget, or with inverse distillation followed by reward fine-tuning? These comparisons would help clarify the specific benefit of the proposed inverse-distillation regularizer.

Early Stopping in Review Sentiment Steering: How does reward-only fine-tuning perform when stopped near reward saturation and compared with IDRF at a matched reward level? Does the advantage in EOS emission and repetition persist under this comparison?

Diversity Across Samples: Can the authors quantify the reduction in MaskGIT sample diversity acknowledged in Section 6 and assess diversity across generated DNA sequences and reviews? This would complement the reported per-sequence entropy metrics and clarify the trade-off between reward, sample quality, and distributional coverage.

Robustness Across Training Runs and Image Prompts: How consistent are the main results across independent training seeds? Do the image-quality improvements persist on a broader set of class-prompt pairs beyond the ten used in the current experiments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer jbTF (1/3)

Dear Reviewer, thank you for your valuable comments. Your questions come down to three checks of the method: whether a transition-KL regularizer would work as well, how much diversity IDRF costs, and whether the results hold across training seeds and new prompts. We ran all three and report the results below.

[W1, Q1] Comparison with reverse-transition KL regularization at the same sampling budget.

Thank you for this suggestion. We ran the comparison with reverse-transition KL at the same sampling budget.

Setup. Path-KL keeps our reward loss and replaces LID\mathcal L_{\mathrm{ID}} with β\beta times the KL between the student and the reference reverse transitions, summed over all transitions of the rollout. This is the regularizer of DRAKES [1] applied to the few-step sampler. The sampling steps, initialization, optimizer, number of iterations and evaluation are those of IDRF. We sweep six values of β\beta. Every row has five training seeds, and we report mean ± SD over them.

Table A: CLIP-guided MaskGIT, 4 sampling steps. CLIP is the optimized reward, and the other metrics are held out.

MethodTraining seedsCLIP ↑ImageReward ↑PickScore ↑HPSv2.1 (×100) ↑
Reference, 4 steps–14.00−2.1718.2414.67
RLOO w/o LID\mathcal L_{\mathrm{ID}}525.64 ± 1.50−0.28 ± 0.2920.49 ± 0.2623.27 ± 0.46
Path-KL, β=0.001\beta=0.001526.39 ± 1.07−0.44 ± 0.6320.49 ± 0.4722.13 ± 2.48
Path-KL, β=0.005\beta=0.005526.57 ± 0.57−0.24 ± 0.1720.69 ± 0.1723.53 ± 0.63
Path-KL, β=0.01\beta=0.01525.28 ± 1.21−0.56 ± 0.3420.36 ± 0.2222.27 ± 0.99
Path-KL, β=0.04\beta=0.04524.06 ± 0.64−1.01 ± 0.1219.99 ± 0.1421.13 ± 0.29
Path-KL, β=0.1\beta=0.1521.84 ± 0.94−1.40 ± 0.1919.58 ± 0.1819.63 ± 0.74
Path-KL, β=0.5\beta=0.5516.65 ± 0.15−2.05 ± 0.0418.67 ± 0.0416.07 ± 0.27
IDRF (ours)526.94 ± 0.340.79 ± 0.1021.70 ± 0.1327.16 ± 0.43

Table B: Regulatory-DNA design, 8 sampling steps.

MethodTraining seedsPred-Activity ↑ATAC-Acc (%) ↑3-mer Corr. ↑
RLOO w/o LID\mathcal L_{\mathrm{ID}}57.93 ± 0.2026.1 ± 7.60.527 ± 0.117
Path-KL, β=0.001\beta=0.00157.99 ± 0.3475.6 ± 26.10.498 ± 0.120
Path-KL, β=0.005\beta=0.00558.04 ± 0.2933.0 ± 15.30.559 ± 0.044
Path-KL, β=0.01\beta=0.0157.66 ± 0.1752.8 ± 19.00.645 ± 0.013
Path-KL, β=0.05\beta=0.0556.68 ± 0.2687.6 ± 1.40.719 ± 0.058
Path-KL, β=0.1\beta=0.155.69 ± 0.2682.3 ± 3.40.845 ± 0.032
Path-KL, β=0.5\beta=0.550.76 ± 0.088.0 ± 1.30.722 ± 0.039
IDRF (ours)57.15 ± 0.1292.0 ± 2.60.668 ± 0.020

Results.

  • Images. No value of β\beta raises the held-out metrics above reward-only fine-tuning beyond seed noise. The best setting, β=0.005\beta=0.005, has ImageReward −0.24 against −0.28 for RLOO. A larger β\beta moves the student back to the four-step reference. On these five seeds IDRF has ImageReward 0.79 ± 0.10, PickScore 21.70 ± 0.13 and HPSv2.1 27.16 ± 0.43, above every Path-KL setting on all three held-out metrics. Its CLIP is also higher, 26.94 against 26.57 for the best Path-KL setting. The samples agree with the metrics. With Path-KL the images stay blurred, as with reward-only fine-tuning, while IDRF produces sharp objects that match the prompt in the same four steps.

  • DNA. With a small β\beta the activity stays high and ATAC-Acc is low and unstable across seeds, from 33% to 76% on average. At β=0.05\beta=0.05 Path-KL reaches its best ATAC-Acc, 87.6%, with activity 6.68. IDRF is higher on both, with ATAC-Acc 92.0% and activity 7.15. Path-KL has a higher 3-mer correlation for β≥0.05\beta\ge 0.05.

  • Text. d2 [2] with one segment per sampling step uses the same regularizer. Its row in Table 3 has GenPPL 139.7 against 65.4 for IDRF at a matched reward.

  • Cost. Path-KL evaluates the reference at every transition of the rollout, five passes on images, eight on DNA and 33 on text. LID\mathcal L_{\mathrm{ID}} needs one reference pass per rollout for any number of steps.

Why Path-KL behaves this way. The transition KL is minimized by the reference network itself, so it pulls the student toward the reference sampled in the same few steps. This sampler is weak at four steps (first row of Table A). LID\mathcal L_{\mathrm{ID}} regularizes toward prefp_{\mathrm{ref}}, the law of the full reference sampler (Proposition 1).

AuthorsOpenReview ↗

Response to Reviewer jbTF (2/3)

[W1, Q1] Inverse distillation followed by reward fine-tuning.

We ran this two-stage variant directly. On images (4 training seeds, 80 + 80 iterations vs. 80 for IDRF), it reaches ImageReward −1.60 ± 0.59 and CLIP 20.7 ± 4.8, against 0.82 ± 0.09 and 27.1 ± 0.2 for joint IDRF with the same seeds. The reward stage lowers ImageReward relative to its own starting point on three of four seeds and collapses on one (CLIP 14.0). On DNA (one seed), it reaches a comparable reward and ATAC-Acc (7.37 / 94.9% vs. 7.15 / 92.0% for IDRF) but loses the distributional match: 3-mer correlation 0.476 vs. 0.668 and entropy 1.05 vs. 1.27 nats, with ATAC-Acc passing through 6.8% during training. Since the regularizer is absent during the reward stage, nothing constrains its drift. Keeping LID\mathcal L_{\mathrm{ID}} active during reward optimization is what prevents it.

[W1, Q2] Early stopping of reward-only fine-tuning at a matched reward.

Thank you for this question. Both runs of Table 3 were evaluated every 100 updates on 1,000 samples, so we can compare them directly. The reward of RLOO is closest to that of IDRF at update 2,700.

Table C: Review text, 32 sampling steps, matched reward.

Modellog⁡P(1⋆)\log P(1\star) ↑GenPPL ↓LengthEOS (%) ↑
Real one-star reviews−0.95053.332.2100.0
RLOO w/o LID\mathcal L_{\mathrm{ID}}, stopped at update 2,700−0.046155.277.199.3
IDRF (ours)−0.04465.444.1100.0

At a matched reward IDRF is much closer to real reviews, with GenPPL 65.4 against 155.2 and length 44.1 against 77.1. The early-stopped RLOO model still emits EOS and its repetition is close to that of IDRF, so the gap at this point is in fluency and length. Training RLOO further breaks the review format. From update 2,700 to 10,000 its reward changes by less than 0.04 nats, while the mean length grows from 77 to 127 tokens and EOS falls from 99.3% to 9.4%.

[W2, Q3] Diversity across samples.

Thank you for this suggestion. We measured diversity across samples in all three domains. In text and DNA IDRF keeps most of the diversity, where reward-only fine-tuning loses it. On images the reduction noted in Section 6 is recovered by sampling the same student with more steps.

Table D: Images. Mean pairwise LPIPS [3] between the 100 samples of a class, averaged over the classes of Table 2. The IDRF columns use the five training seeds of Table A, with the same 4-step students sampled with more steps and no retraining.

Sampling stepsReference, LPIPS ↑IDRF, LPIPS ↑Reference, ImageReward ↑IDRF, ImageReward ↑
40.5960.352 ± 0.034−2.170.79 ± 0.10
160.6740.439 ± 0.036−1.620.66 ± 0.10
320.6980.481 ± 0.036−1.440.61 ± 0.08
640.7150.514 ± 0.036−1.350.53 ± 0.12
1280.7250.532 ± 0.038−1.280.48 ± 0.10

At 4 steps IDRF is the only model with a positive ImageReward (0.79, against −2.17 for the reference and −0.28 for reward-only fine-tuning, whose LPIPS is 0.524). With more steps the same student reaches LPIPS 0.532, and its ImageReward stays positive at every number of steps.

Table E: Text. 2,748 samples per model and the 2,748 held-out real one-star reviews. distinct-2 [4] and Self-BLEU-4 [5] use the first 32 tokens of each review. The embedding distance is the mean pairwise cosine distance between all-mpnet-base-v2 sentence embeddings [6].

Reviewsdistinct-2 ↑Self-BLEU-4 ↓Embedding distance ↑
Real one-star reviews0.4830.3190.813
Reference, 128 steps (all ratings)0.4900.2880.803
RLOO w/o LID\mathcal L_{\mathrm{ID}}, 32 steps0.1130.7670.237
IDRF (ours), 32 steps0.3220.5340.671

IDRF keeps 83% of the embedding distance of real one-star reviews and 67% of their distinct-2. Reward-only fine-tuning keeps 29% and 23%. Reward fine-tuning narrows the distribution by design, since the samples become strongly negative (log⁡P(1⋆)=−0.04\log P(1\star)=-0.04 against −0.95-0.95 for real reviews), and the regularizer is what preserves the diversity.

DNA. The mean pairwise Hamming distance between 640 generated sequences is 0.692 ± 0.006 for IDRF over five training seeds, close to the 0.750 of random sequences. Reward-only fine-tuning has 0.607 ± 0.042.

AuthorsOpenReview ↗

Response to Reviewer jbTF (3/3)

[W2, Q4] Robustness across training seeds and class-prompt pairs.

Thank you for this question.

Broader set of class-prompt pairs. We trained on ten new class-prompt pairs that are not in Table 2: "A castle outdoors", "a yellow bell pepper", "a white four-poster", "a red hornbill", "a close-up photo of a Tibetan mastiff", "a brown wire-haired fox terrier on a plain background", "A bagel with a shallow depth of field", "A coral fungus with a shallow depth of field", "A backpack in natural light" and "a clean studio shot of a steel drum".

Table F: New class-prompt pairs, 4 sampling steps.

MethodTraining seedsCLIP ↑ImageReward ↑PickScore ↑HPSv2.1 (×100) ↑
Reference, 4 steps–16.33−1.8618.4515.76
RLOO w/o LID\mathcal L_{\mathrm{ID}}527.11 ± 0.42−0.27 ± 0.2220.17 ± 0.1521.33 ± 0.73
IDRF (ours)527.16 ± 0.570.37 ± 0.1420.84 ± 0.1024.45 ± 0.54

The improvements persist. At the same CLIP, IDRF is above reward-only fine-tuning and the reference on all three held-out metrics, with ImageReward 0.37 against −0.27 and −1.86.

Training seeds. We report several training seeds in all three domains: five per method on images (Table F), five per method on DNA (Table G) and five for IDRF on text (Table H).

Table G: Regulatory DNA, 8 sampling steps.

MethodTraining seedsPred-Activity ↑ATAC-Acc (%) ↑3-mer Corr. ↑
RLOO w/o LID\mathcal L_{\mathrm{ID}}57.93 ± 0.2026.1 ± 7.60.527 ± 0.117
IDRF (ours)57.15 ± 0.1292.0 ± 2.60.668 ± 0.020

IDRF has ATAC-Acc 92.0 ± 2.6%, against 26.1 ± 7.6% for reward-only fine-tuning.

Table H: Amazon one-star reviews, 32 sampling steps.

MethodTraining seedslog⁡P(1⋆)\log P(1\star) ↑GenPPL ↓LengthEOS (%) ↑GM ↓
RLOO w/o LID\mathcal L_{\mathrm{ID}} (Table 3)1−0.00876.9126.89.428.81
IDRF (Table 3)1−0.04465.444.110010.86
IDRF (ours), new seeds5−0.047 ± 0.00762.1 ± 2.155.9 ± 5.5100 ± 011.03 ± 0.90

The five new seeds give log⁡P(1⋆)=−0.047±0.007\log P(1\star)=-0.047 \pm 0.007, GenPPL 62.1±2.162.1 \pm 2.1, EOS 100% on every seed and GM 11.03±0.9011.03 \pm 0.90, close to the row of Table 3 (−0.044, 65.4, 100% and 10.86).


Citations:

[1] Wang et al., Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design, ICLR 2025.

[2] Wang et al., d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation, ICML 2026.

[3] Zhang et al., The Unreasonable Effectiveness of Deep Features as a Perceptual Metric, CVPR 2018.

[4] Li et al., A Diversity-Promoting Objective Function for Neural Conversation Models, NAACL 2016.

[5] Zhu et al., Texygen: A Benchmarking Platform for Text Generation Models, SIGIR 2018.

[6] Reimers and Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP 2019.


Concluding remarks: We thank the reviewer for questions that sharpened the paper. Three findings stand out. First, a transition-KL regularizer at the same sampling budget does not reproduce the effect of LID\mathcal L_{\mathrm{ID}}. No weight improves the held-out image metrics over reward-only fine-tuning beyond seed noise, and on DNA IDRF reaches ATAC-Acc 92.0% over five training seeds against 87.6% for the best transition-KL setting. Second, the diversity that IDRF gives up on images at 4 steps is regained at sampling time with more steps, and in text and DNA IDRF keeps most of the diversity that reward-only fine-tuning loses. Third, the gains hold on new class-prompt pairs and across training seeds. We hope these results resolve the points raised in the review, and we are glad to clarify anything further.

Thank you,

The Authors

Reviewer_vUeAOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes INVERSE-DISTILLED REWARD FINE-TUNING (IDRF) for fine-tuning mask diffusion language model (MDLM) on given reward model while achieving distillation. The paper leverages the distillation framework Inverse-distilled Diffusion Language Models (IDLM) and Inverse Bridge Matching Distillation (IBMD) as a regularization for keeping the fine-tuned model similar to original model and achieve distillation. The experimental results show that the IDRF achieves comparable reward with much less sampling steps.

Critical strengths
  1. Easy to read
  2. Comprehensive experimental results
  3. Good results under limited sampling steps
Critical weaknesses
  1. Algorithm mainly comes from existing works, IDLM. IDRF is a policy graident with IDLM as regularization.
  2. The sampling step in the experiments are not fair between IDRF and baselines. Could you provide the result of IDRF with 128 steps in Table 1, 2, and 3? Does the performance grow with more sampling steps?
Questions for authors

Same as weakness

2 comments
AuthorsOpenReview ↗

Response to Reviewer vUeA (1/2)

Dear Reviewer, thank you for the review and for the positive assessment of our experiments. You raise two points: what IDRF adds to IDLM, and whether the comparison is fair in the number of sampling steps. We address both and report IDRF sampled with up to 128 steps in all three domains.

[W1] What IDRF adds to IDLM.

Your summary is accurate. IDRF is a policy gradient regularized with the inverse-distillation loss of IDLM [1]. Our contributions are the following.

1. A guarantee that turns this loss into a regularizer. IDLM proves its bound for a single token and leaves the sequence-level loss used in practice to future work ([1], App. D.2). Proposition 1 proves LID(θ)≥DKL(pθ∥pref)\mathcal L_{\mathrm{ID}}(\theta)\ge D_{\mathrm{KL}}(p_\theta\Vert p_{\mathrm{ref}}) for full sequences, for any trained reference and any student sampler. It gives a tractable way to keep a reward-tuned model close to the reference, whose sequence likelihood is intractable.

2. Reward fine-tuning directly in the few-step regime. IDLM has no reward. We derive the exact probability of a KK-step rollout (Eq. 9) and optimize the reward on the same trajectories that the regularizer uses. Training needs no pretraining data, no reward gradients and no reference rollouts.

3. Evidence that this combination is what works in few steps. One alternative is a KL between reverse transitions, as in DRAKES [2] and d2. Another is a two-stage pipeline that first distills with LID\mathcal L_{\mathrm{ID}} and then fine-tunes on the reward. We ran both on images.

Table A: Alternatives to IDRF on CLIP-guided MaskGIT, 4 sampling steps. Mean ± SD over training seeds.

MethodSeedsCLIP ↑ImageReward ↑PickScore ↑HPSv2.1 (×100) ↑
Reward only (RLOO)525.64 ± 1.50−0.28 ± 0.2920.49 ± 0.2623.27 ± 0.46
Reward + transition KL, best β\beta526.57 ± 0.57−0.24 ± 0.1720.69 ± 0.1723.53 ± 0.63
Distill with LID\mathcal L_{\mathrm{ID}}, then reward420.73 ± 4.76−1.60 ± 0.5918.95 ± 1.2116.68 ± 5.54
IDRF (ours)526.94 ± 0.340.79 ± 0.1021.70 ± 0.1327.16 ± 0.43

IDRF is the only row with a positive ImageReward, and it is the best on every metric. The revised paper also compares IDRF with d1, wd1 and d2 in all three domains (Tables 1 to 3).

AuthorsOpenReview ↗

Response to Reviewer vUeA (2/2)

[W2] Sampling steps of IDRF and the baselines.

The fine-tuned baselines use the same steps as IDRF. In Tables 1 to 3 the rows RLOO, d1, wd1 and d2 are trained and sampled with the same 8, 4 and 32 steps as IDRF. Only the loss differs. The rows with more steps are the references, the published DNA baselines (128 steps) and ReDGE. In these comparisons the larger budget is on the side of the baselines.

IDRF with 128 steps. We sampled trained IDRF models with up to 128 steps in all three domains, with no retraining.

Table B: Regulatory-DNA design, IDRF trained with 8 steps. Mean ± SD over five training seeds.

StepsPred-Activity ↑ATAC-Acc (%) ↑3-mer Corr. ↑
87.15 ± 0.1292.0 ± 2.60.668 ± 0.020
167.35 ± 0.0986.8 ± 5.50.648 ± 0.024
327.39 ± 0.0982.4 ± 6.40.638 ± 0.027
647.41 ± 0.1081.1 ± 6.90.633 ± 0.028
1287.42 ± 0.1078.1 ± 8.20.631 ± 0.029

Table C: CLIP-guided MaskGIT, IDRF trained with 4 steps. Mean ± SD over the five seeds of Table A, no guidance. LPIPS is the mean pairwise distance between images of one class and measures diversity.

StepsCLIP ↑ImageReward ↑PickScore ↑HPSv2.1 (×100) ↑LPIPS ↑
426.94 ± 0.340.79 ± 0.1021.70 ± 0.1327.16 ± 0.430.352 ± 0.034
1626.78 ± 0.350.66 ± 0.1021.62 ± 0.1727.22 ± 0.430.439 ± 0.036
3226.64 ± 0.360.61 ± 0.0821.58 ± 0.1727.14 ± 0.400.481 ± 0.036
6426.47 ± 0.470.53 ± 0.1221.54 ± 0.1926.98 ± 0.440.514 ± 0.036
12826.34 ± 0.430.48 ± 0.1021.50 ± 0.2026.89 ± 0.400.532 ± 0.038
Reference, 128 steps, w=3w=3 (Table 2)23.11−0.7321.1025.05–

Table D: Sentiment steering, IDRF trained with 32 steps (the model of Table 3).

Stepslog⁡P(1⋆)\log P(1\star) ↑GenPPL ↓LengthEOS (%) ↑Norm. ent. ↑GM ↓
32−0.04465.444.1100.00.90210.86
64−0.03962.338.4100.00.9139.78
128−0.04763.134.0100.00.9218.95
Real one-star−0.95053.332.2100.00.9480.03

Does the performance grow with more steps? The optimized reward changes little with the step count, and the effect on the held-out metrics depends on the domain.

  • Text. More steps help. The reward stays at the same level and the reviews move closer to real ones in GM (10.86 to 8.95), length (44.1 to 34.0, real 32.2) and entropy.
  • Images. The quality metrics decrease slowly (ImageReward from 0.79 to 0.48) and diversity grows (LPIPS from 0.352 to 0.532). At 128 steps IDRF stays above the 128-step reference with guidance on all four quality metrics.
  • DNA. Activity grows from 7.15 to 7.42 and ATAC-Acc decreases from 92.0% to 78.1%. Reward-only fine-tuning trained and sampled with 128 steps keeps 0.1% ATAC-Acc (Table 1).

IDRF applies the regularizer on the states that its KK-step sampler visits (Section 3.2), so the training budget KK is its natural setting and the cheapest one. More steps remain an option at inference.

Citations

[1] Li et al., IDLM: Inverse-distilled Diffusion Language Models, ICML 2026.

[2] Wang et al., Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design, ICLR 2025.

Concluding remarks

IDRF builds on the loss of IDLM and adds the sequence-level guarantee, few-step reward fine-tuning, and evidence that the alternatives do not reach its quality in this regime. Every fine-tuned baseline samples with the same steps as IDRF. Sampled with 128 steps, IDRF improves on text, stays above the 128-step reference on images, and reaches a higher activity on DNA at a lower ATAC-Acc. We hope this clarifies both points, and we are glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.