- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Practical reward fine-tuning algorithms often use a KL regularization term which ensures that the learned policy does not drift “too far” from some reference one. For discrete diffusion models, this regularization term is intractable. This work demonstrates that the inverse distillation loss from IDLM [Li et al., 2026] upper bounds this KL regularization when we replace data distributions from IDLM with the distribution induced by the reference policy. This provides a principled motivation for reward fine-tuning with the new proposed IDRF loss which replaces the KL with the IDLM-inspired loss. The authors demonstrate that reward fine-tuning with their proposed loss allows for competitive few step reward-guided generation across several domains.
Critical strengths
- This paper deals with an important problem for masked diffusion models.
- The results demonstrate strong small NFE reward guidance compared to other methods.
Critical weaknesses
Theoretical derivation / Method presentation
I found parts of the Methods section to be hard to follow. Below I detail some examples of issues I had. I recommend that the authors restructure this section to better motivate and explain their methodology.
- Given that the main theoretical contribution is extending the IDLM [Li et al., 2026] result from one token to full sequences (and then applying it to as opposed to ), in the main text, it would be useful for the authors to provide intuition regarding why the original IDLM result fails to establish the practical sequence-level result and how the current work gets around / resolves that issue. Perhaps I am missing something but in practice from equation 7 in the current manuscript seems to be equivalent to summing token level losses from IDLM.
- The importance weighting term is not well introduced nor is it sufficiently motivated that it the fully factorized ratio of and is valid. For example, the derivation in d2 [Wang et al., 2026] is more comprehensive in this respect.
- I did not understand the purpose nor the placement of Section 3.2 in the methods section. I do not see how the MDP formulation contributes to the overall presentation of the method. As far as I can tell the “practical objective” and the training algorithm could presented almost identically even without Section 3.2.
Baselines
I think the authors should have a more systematic approach to selecting baselines. While I understand that each experiment section comes with prior work, e.g., DRAKES in the DNA experiments, it would be better to consistently compare to similar Diffusion Reward fine-tuning methods. Specifically, I believe the authors should consistently compare to a reasonable subset of the following methods:
- d1 [Zhao et al., 2025], wd1 [Tang et al., 2025], coupledGRPO [Gong et al., 2025]
- TraceRL [Wang et al., 2025] / d2 [Wang et al., 2026] (the latter of which was also omitted from the Related Works section and should be included).
I recognize that some of these, e.g., diffuGRPO, are already included in the DNA experiment, but adding them in the image and NLP experiments would substantially strengthen the empirical results in my opinion.
Questions for authors
- The ID loss upper bounds KL and is therefore motivated as a useful surrogate for regularized reward fine-tuning. However it would be interesting to see a targeted experiment / analysis comparing ID with KL in cases where the KL is indeed tractable, e.g., for true any-order diffusion LMs, as explained in d2 [Wang et al., 2026], or perhaps in a smaller setting where KL can be computed without prohibitive computation resources.
- I am curious if the authors tried this method on a large DLM like LLaDA / Dream7B for benchmark tasks, i.e., following the setup of d1, wd1, d2, etc. If so, are there preliminary results that can be added to this work?
5 comments
Response to Reviewer cppW (2/4)
[W1.b] The importance weight, and why the fully factorized ratio is valid.
We agree that the ratio was introduced too quickly. Section 3.2 now derives it from the sampler.
Rollout probability. The student is the standard ancestral sampler run with steps. At step each masked position is revealed with a fixed probability , and its token is drawn from . The denoiser never predicts the mask token, so a masked position either stays masked or is revealed as a clean token. With the set of positions revealed at step , the probability of a rollout is
where collects the factors and and does not depend on (Eq. 9, with the derivation in Appendix B).
The ratio. The factor cancels, so the likelihood ratio between the current and the old student is exactly a product of per-token ratios:
This is the trajectory likelihood of d2 [2] with one segment per sampling step. We state this in Appendices C.6 and E.2, where d2 is run in this exact setting.
How we use it. The full product is exact and has high variance, since it multiplies up to factors. As in PPO, Eq. (10) sums the per-token ratios and clips each one. At this has the same gradient as the product (Appendix B). The ratios only matter when a rollout batch is reused. This is the case for DNA, with two updates per batch. For images and text each batch serves one update, every ratio equals one, and Eq. (10) is plain RLOO. The revised Section 3.2 states this explicitly.
[W1.c] Purpose and placement of Section 3.2 (MDP formulation).
Thank you for pointing this out. We agree that the purpose of this section may have sounded unclear.
Why the MDP view matters. The likelihood of a sequence, , is intractable, because it sums over all unmasking trajectories. Viewed as an MDP, the sampler has states and a policy with tractable per-step probabilities. The log-probability of a whole trajectory, , can therefore be computed exactly (Eq. 9), and the policy gradient of the reward needs only this quantity.
In the revised paper. To make this section easier to read and understand, we restructured it and clarified the unclear points. Section 3.2 is now "Training the few-step student". It introduces the few-step sampler, derives the rollout probability, and builds the reward term on it. The MDP view is stated as a remark with a reference to AGRPO [7].
Response to Reviewer cppW (3/4)
[W2] A systematic comparison with diffusion reward fine-tuning methods.
We agree that a consistent set of baselines strengthens the results. We implemented d1 [3], wd1 [4] and d2 [2] in our codebase and trained them in all three domains.
Setup. Every baseline uses the same few-step sampler, initialization, rollouts, optimizer and evaluation as IDRF. Only the loss changes. d1 and wd1 score tokens at the fully masked state. d2 uses one segment per sampling step, which makes its trajectory likelihood exact. On images it scores four of the five transitions per update, as our reward loss does. Each row is a single training run, as for our own rows. For DNA and images, d1 and d2 use , the value reported for both methods ([2], [4]), and wd1 its best . For text, each baseline is shown at the weight whose reward is closest to that of IDRF. Full sweeps over and are in Appendices C.6, D.1 and E.2, with samples for images and text. In the tables below, bold marks the best value among d1, wd1, d2 and IDRF.
Table A: Regulatory-DNA design, 8 sampling steps (rows of Table 1 in the paper).
| Method | Pred-Activity ↑ | ATAC-Acc (%) ↑ | 3-mer Corr. ↑ |
|---|---|---|---|
| RLOO w/o | 7.68 | 37.8 | 0.378 |
| d1 | 4.14 | 17.3 | 0.888 |
| wd1 | 5.04 | 0.5 | 0.696 |
| d2 | 7.26 | 89.2 | 0.735 |
| IDRF (ours) | 7.22 | 95.3 | 0.683 |
Table B: CLIP-guided MaskGIT, 4 sampling steps (rows of Table 2). CLIP is the optimized reward, and the other metrics are held out.
| Method | CLIP ↑ | ImageReward ↑ | PickScore ↑ | HPSv2.1 (×100) ↑ |
|---|---|---|---|---|
| Reference, 4 steps | 14.00 | −2.17 | 18.24 | 14.67 |
| RLOO w/o | 26.12 | −0.49 | 20.61 | 22.57 |
| wd1 | 30.10 | 0.19 | 20.90 | 22.44 |
| d1 | 24.59 | −0.73 | 20.18 | 21.14 |
| d2 | 19.28 | −1.80 | 19.15 | 17.91 |
| IDRF (ours) | 27.74 | 0.88 | 22.11 | 27.72 |
Table C: Sentiment steering of review text, 32 sampling steps (rows of Table 3). is the optimized reward.
| Method | ↑ | GenPPL ↓ | Length | EOS (%) ↑ | GM ↓ |
|---|---|---|---|---|---|
| Real one-star reviews | −0.950 | 53.3 | 32.2 | 100.0 | 0.03 |
| Reference, 128 steps | −5.070 | 91.7 | 27.8 | 100.0 | 2.45 |
| RLOO w/o | −0.008 | 76.9 | 126.8 | 9.4 | 28.81 |
| wd1 | −0.014 | 59.7 | 124.4 | 88.5 | 22.69 |
| d2 | −0.038 | 139.7 | 70.2 | 100.0 | 10.07 |
| d1 | −0.077 | 107.6 | 64.5 | 100.0 | 8.22 |
| IDRF (ours) | −0.044 | 65.4 | 44.1 | 100.0 | 10.86 |
Results.
- DNA. d1 and wd1 stay far from IDRF, in line with the 128-step diffu-GRPO row that Table 1 already contained (activity 5.86, ATAC-Acc 33.0%). d2 comes close, with activity 7.26 and ATAC-Acc 89.2%, against 7.22 and 95.3% for IDRF.
- Images. wd1 reaches a higher CLIP than IDRF (30.10 against 27.74) and is below it on every held-out metric, with ImageReward 0.19 against 0.88. d1 and d2 stay below the reward-only ablation. The difference is clearly visible in the samples (Figure 4 and Figure 7 in Appendix D.1, the first sample of each class with no selection). IDRF produces clear objects that match the prompt. The wd1 images are dominated by the prompt colour with little object detail, and the d1 and d2 images are cluttered with no clear or very fuzzy objects.
- Text. d1 and d2 keep EOS at 100%, and their GenPPL (108 and 140) is above that of the reference (91.7). wd1 writes reviews about four times as long as real ones. IDRF is the only fine-tuned model that is close to real reviews in both GenPPL and length.
Each baseline falls short in at least one domain, while IDRF is strong in all three.
Why the baselines behave this way. d1 and d2 both regularize with a token-level KL to the reference network. d1 evaluates it at the fully masked state, and d2 at every state of the rollout. In both cases the term is minimized by the reference network itself, so a larger moves the student back toward the reference sampled in the same few steps. On images this sampler is weak at 4 steps (first row of Table B), and at both methods return to its level (Appendix D.1). regularizes toward itself (Proposition 1, Corollary 1). It needs one pass through the reference per rollout for any , while the KL of d2 evaluates the reference on every segment, passes in this setting. We added this comparison to Section 3.2.
Related Work. d2 and coupled-GRPO [5] are added to Section 4, next to d1, wd1 and TraceRL [6].
Response to Reviewer cppW (1/4)
Dear Reviewer, thank you for your careful review and constructive suggestions. We are glad that you find the problem important and the few-step results strong. Your comments made the method section clearer and the comparison more systematic. All changes are marked in blue in the revised paper. Below we respond to each point.
[W1.a] Why the IDLM result does not cover the practical sequence-level loss, and how Proposition 1 resolves it.
Thank you for raising this. You read Eq. (7) correctly. It is the same sum of token-level losses that IDLM [1] trains with. We keep the loss unchanged. Our contribution is the guarantee for it.
What IDLM proves. IDLM proves its bound for a single token. It notes that the same argument applies if the whole sequence is treated as one state with a joint reverse process. For the sum of token-level losses used in practice, it leaves the analysis to future work ([1], App. D.2), because the tokens of a sequence are dependent. Applying the single-token result position by position controls each token separately, and the joint law of the sequence stays open.
How Proposition 1 closes the gap. IDLM argues through path measures, a general view that covers several kinds of discrete diffusion. To close the gap, we specialize to masked diffusion and view it as any-order autoregression. This narrower view allows us to show the following. In continuous time the reverse process reveals the masks one at a time, in a uniformly random order σ. Let πf,σ(x) be the probability of generating x in this order with denoiser f. Then the NELBO is an autoregressive log-loss averaged over orders, and the law of the reference sampler is the same average:
LNELBO(f,pθ)=EσEx∼pθ[−logπf,σ(x)],pref=Eσπfref,σ.
The Bayes denoiser of pθ generates pθ in every order, so the inner minimum is the entropy of pθ. What remains is a KL between full sequences for each order, and convexity of the KL gives
LID(θ)=EσDKL(pθ∥πfref,σ) ≥ DKL(pθ∥pref).
Averaging over reveal orders is the step that extends the single-token bound to dependent tokens. The bound involves only the law pθ, so it holds for any sequence length and any student sampler.
From p∗ to pref. IDLM assumes a reference that is optimal for the data. Proposition 1 holds for any reference and bounds the KL to pref, the law of the reference's own sampler. Equality holds when the reference is the Bayes denoiser of that law. This is the case that matters for fine-tuning, where the reference is a trained network.
In the revised paper. We agree that this intuition belongs in the main text. Section 3.1 now contains the paragraph "Why the bound holds" with the two identities above, and Section 2.2 states what IDLM leaves open. The full proof is in Appendix A.