acceptodds
Under review as a conference paper at ICLR 2027

RFG: Self-Improving Diffusion Large Language Models with Reward-Free Guidance

Abstract

Diffusion Large Language Models (dLLMs) have shown strong reasoning capabilities, yet further improving them typically requires costly post-training with additional data and supervision. We ask whether a post-trained dLLM can improve itself at inference time without additional training, data, or reward models. This requires a guidance signal from the checkpoints alone that is well-defined on the partially masked intermediate states of dLLMs, which existing methods fail to provide. Here we propose Reward-Free Guidance (RFG), a training-free framework for inference-time self-improvement of dLLMs that derives such a signal from the checkpoints themselves. We theoretically demonstrate that a reward signal for a partially masked state can be parameterized by the log-likelihood ratio between a post-trained policy dLLM and its reference model. In practice, this signal can be obtained using off-the-shelf checkpoints alone. Extensive experiments show that RFG consistently improves state-of-the-art post-trained dLLMs by up to 16.1%. Furthermore, despite being entirely training-free, RFG delivers gains that rival or even surpass resource-intensive reinforcement learning techniques.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion large language models, self-improving LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jYaxOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The reasoning capabilities of Diffusion Large Language Models (dLLMs) are typically improved through additional post-training, which is computationally expensive. To address this issue, the paper proposes a method for further enhancing model capability after post-training has already been completed. By using the logits of the pre- and post-training models during inference to compute a guidance signal, the method enables a dLLM to self-improve at inference time using only existing checkpoints.

Critical strengths

The paper proposes an efficient inference-time optimization and guidance algorithm that can further improve the capabilities of dLLMs.

Critical weaknesses

First, there is a problem with the contribution of the paper. The paper mentions statements such as “Furthermore, RFG matches or exceeds the gains delivered by resource-intensive RL post-training while remaining fully training-free,” emphasizing that RFG is more resource-saving. However, using RFG requires a model that has already completed RL post-training. It is not reasonable to claim that this part of the cost does not exist simply because one can directly use an RL-post-trained model trained by others.

Second, there is a problem with the related work of the paper. As the authors mention CFG, the related work of this work should be inference-time guidance methods, rather than post-training algorithms such as GRPO.

Questions for authors

Compared with a broader range of inference-time guidance methods, is the advantage of RFG clear? If the authors can demonstrate that RFG is an effective inference-time guidance method, I would be willing to raise my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jYax (1/2): Weaknesses (W1, W2)

We thank the reviewer for the feedback and for recognizing RFG as an efficient algorithm that further improves the capabilities of dLLMs. We address each concern below.


TL;DR

  • W1 (training cost). Using RFG does not require RL checkpoints, as it works with SFT checkpoints as well. Our claim based on Table 1 compares RFG and RL on the same existing SFT checkpoint, which is a fair comparison. RFG here does not use any checkpoint that has undergone RL post-training.
  • W2 (related work). Due to the page limit, the initial submission left the complete related work that covers inference-time guidance methods in its Appendix D. The revision moves it back into Section 5 and adds more inference-time guidance methods. Post-training algorithms such as GRPO are covered because they produce the checkpoints RFG uses.
  • Q (guidance baselines). The advantage of RFG is clear across a broader range of guidance methods. With A-CFG and FoCore added to CFG, RFG is the best or tied-best method in all 48 settings, and the only one that improves the original model in every setting.

Contribution and Comparison with RL Post-Training (W1)

The reviewer's concern is our claim that RFG "matches or exceeds the gains delivered by resource-intensive RL post-training while remaining fully training-free". The reviewer reads RFG as "requiring a model that has already completed RL post-training", so that the claim omits the cost of that training.

We share the reviewer's view that training cost for RL checkpoints must be accounted for, but we respectfully clarify that this reading does not match our setup.

Using RFG does not require RL checkpoints, as it works with SFT checkpoints as well, and the claim involves exactly that case: RFG applied to an existing SFT checkpoint, compared with RL post-training of the same checkpoint.

  • RFG works with either SFT or RL checkpoints. Our experiments use both, and half of the settings use SFT checkpoints that have undergone no RL (LLaDA-Instruct, Dream-Instruct). RFG improves all of them (Tables 2 to 5), so RL post-training is not a prerequisite for RFG.

  • The claim is made under a fair comparison in Table 1. It compares two ways of improving the same existing SFT checkpoint, LLaDA-Instruct:

    • RFG: guidance on this SFT checkpoint rather than an RL checkpoint. Its reference is the base LLaDA, so no model used by RFG in Table 1 has undergone any RL post-training.
    • RL: further RL training of this SFT checkpoint.

    Both methods thus start from the same existing checkpoint, so the comparison is fair and supports the claim. The revised caption of Table 1 states this setup explicitly.

We also note that

  • Comparison with RL (Table 1) serves as a reference for RFG's effectiveness. It places RFG's absolute gains next to those of RL post-training, which makes them more meaningful.
  • We do apply RFG to RL checkpoints, in the main tables (Tables 2 to 5) and not in Table 1. These results show that RFG and RL are complementary, not substitutes. In practice the two are composable, as RFG yields further gains on top of RL checkpoints.

Scope of the Related Work (W2)

The reviewer notes that, since the paper relates RFG to CFG, its related work should cover inference-time guidance methods and not post-training algorithms such as GRPO.

We'd like to clarify that the paper does cover inference-time guidance. Due to the page limit, the main text of the initial submission held only a summary of the works we build on. The complete related work was left to Appendix D, which discusses inference-time guidance (classifier guidance, CFG, autoguidance, contrastive decoding, guidance for discrete diffusion). We include the post-training methods as well because they are a core part of our setting: they produce the SFT or RL checkpoints for RFG (e.g., d1-LLaDA and diffuCoder are trained with GRPO).

With the additional page allowed for the revision, we've moved the complete related work back into Section 5 and extended it with further inference-time guidance methods, including A-CFG and FoCore for dLLMs and Diffusion Blend and RLG for image diffusion.

AuthorsOpenReview ↗

Response to Reviewer jYax (2/2): Questions

Comparison with a Broader Range of Inference-Time Guidance Methods (Q)

The reviewer asks whether the advantage of RFG is clear against a broader range of inference-time guidance methods, and states a willingness to raise the score if RFG is shown to be effective.

Yes. RFG is the best or tied-best method in all 48 model-benchmark-length settings of Tables 2 to 5, and the only one that improves the original model in every setting. The initial submission compared RFG with CFG (Tables 2 to 5) and policy-only sharpening (Table 9). We have added two recent inference-time guidance methods designed for dLLMs, A-CFG (Li et al., 2025) and FoCore (Feng et al., 2026), on all 48 settings. Like RFG, they need no training and no external reward model. CFG, A-CFG, FoCore, and RFG share the same cost of two forward passes per denoising step, and they differ in the second branch of the extrapolation form (Table R3).

Table R3: Guidance methods on post-trained dLLMs over the 48 settings of Tables 2 to 5. Changes are in accuracy points relative to the original model.

MethodSecond branchSettings improvedMean change
CFGSame model, prompt masked8 of 48−3.7
A-CFGSame model, low-confidence tokens re-masked4 of 48−2.9
FoCoreSame model, high-information-density tokens re-masked24 of 48+0.2
RFGReference checkpoint, same input48 of 48+2.7
  • Against the original model. CFG and A-CFG lower average accuracy, FoCore leaves it nearly unchanged, and RFG improves every setting.
  • Head to head. RFG is above CFG in 47 settings with one tie, above A-CFG in all 48, and above FoCore in 46 with two ties. No method is above RFG in any setting.

Note that all methods are evaluated in the same setting as our main results in Tables 2 to 5. The A-CFG paper tests pre-trained models with few-shot prompts, and the FoCore paper relies on different prompts and the full MBPP test set, whereas we use the sanitized one. Absolute numbers therefore differ from those in the original papers. Despite this, FoCore’s ranking above CFG and A-CFG is preserved in our runs (21 and 23 of the 24 settings shared with its paper). Our original-model baseline is higher than the FoCore paper’s in 19 of these 24 settings. We consider this the fairest test of inference-time guidance, since each method must improve a stronger original model.

We thank the reviewer for this question. The revised paper adds A-CFG and FoCore to the experiments in revised Appendix.

Full Results (also in revised Appendix)

BenchmarkModelLengthOriginalCFGA-CFGFoCoreRFG
GSM8KLLaDA-Instruct12876.372.674.876.377.2
25679.880.777.981.381.3
51283.781.581.382.884.6
Dream-Instruct12867.963.665.268.674.2
25680.972.275.480.282.1
51284.880.181.984.085.1
d1-LLaDA12879.377.477.678.579.8
25682.581.380.882.184.7
51283.682.282.982.584.6
LLaDA-1.512876.375.776.976.378.0
25681.682.079.681.382.1
51284.881.384.084.085.3
MATH-500LLaDA-Instruct12832.423.830.232.834.4
25638.434.035.637.439.6
51242.639.040.043.644.0
Dream-Instruct12832.218.420.632.033.6
25643.622.831.641.246.4
51250.429.641.251.252.6
d1-LLaDA12834.021.229.235.037.4
25638.833.836.439.641.6
51242.639.440.841.643.8
LLaDA-1.512832.023.428.831.634.0
25638.633.635.039.440.6
51242.036.839.844.044.0
HumanEvalLLaDA-Instruct12832.326.831.132.933.5
25639.637.836.040.240.9
51245.139.040.945.747.6
Dream-Instruct12852.457.359.154.960.4
25661.066.557.961.666.5
51262.264.059.861.665.2
LLaDA-1.512832.925.630.534.137.8
25639.638.436.639.041.5
51245.739.640.243.946.3
diffuCoder12870.168.965.972.073.8
25669.570.772.070.776.2
51269.570.772.671.378.7
MBPPLLaDA-Instruct12851.049.048.249.854.1
25654.949.848.254.155.6
51249.849.444.750.652.9
Dream-Instruct12865.061.963.864.268.1
25664.263.463.465.469.6
51264.664.662.365.470.4
LLaDA-1.512851.848.647.953.354.1
25654.552.946.754.557.2
51250.651.043.251.055.6
diffuCoder12872.470.871.673.974.3
25672.471.270.872.475.9
51272.871.671.672.475.5

References

  • Li, P., et al. (2025). Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking. NeurIPS 2025.
  • Feng, J., et al. (2026). Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast. arXiv:2605.01373.

We hope these results and clarifications address the reviewer's concerns and welcome further discussion.

Thank you for your response. The new experiments show that RFG improves the responses of post-trained models. I’m willing to raise my score.

However, I still have concerns about how the paper presents its contribution. Both SFT and RL take extra compute to train a model from a base checkpoint. That cost still exists even if the model developers release the SFT checkpoint. RL can start from a base checkpoint alone, and RFG needs both the base checkpoint and the post-trained one. This is a limitation of RFG, it’s why I’m still not convinced by the comparison with RL. I don’t think being “training-free” is a contribution in this paper.

We thank the reviewer for the prompt reply and for re-evaluating the paper. Following the reviewer's comment, we'll make these points explicit in the paper:

  • RFG needs a post-trained (SFT / RL) checkpoint, so it builds on the cost of post-training and does not remove it.
  • The comparison with RL is only a reference for the size of RFG's gains. It does not show that RFG removes the cost of post-training.

We thank the reviewer again for helping make the paper's claims more precise.

Reviewer_ZKEqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reward-Free Guidance (RFG) to further improve the generation quality of post-trained Diffusion Language Models without additional training or an external reward model. The method constructs an implicit trajectory reward from the log-likelihood ratio between the post-trained model and a base reference model, and uses a Markov decomposition to obtain stepwise process rewards. During actual decoding, RFG linearly extrapolates the logits of the two models via (1+w)ℓθ−wℓref(1+w)\ell_\theta-w\ell_{\mathrm{ref}}, thereby strengthening the post-trained model’s predictive preference relative to the base model. Experiments cover multiple SFT/RL post-trained LLaDA and Dream series models and demonstrate certain performance improvements on mathematical reasoning, code generation, and instruction-following tasks.

Critical strengths
  1. The research direction is of practical value. How to leverage existing base and post-trained models to further improve inference quality without additional training is a question worth exploring.
  2. The method is simple and the experiments are relatively broad. RFG can directly use existing checkpoints and achieves performance improvements across multiple models, training stages, and generation tasks.
Critical weaknesses
  1. The core method is highly related to existing image diffusion guidance work, and its novelty needs to be further clarified. RFG uses ℓRFG=(1+w)ℓθ−wℓref\ell_{\mathrm{RFG}}=(1+w)\ell_\theta-w\ell_{\mathrm{ref}} to extrapolate predictions between the base model and the post-trained model. Similar two-model guidance ideas have been studied in the image diffusion literature. For example, Autoguidance [1] uses the prediction difference between a stronger and a weaker model to improve generation quality; DB-KLA in Diffusion Blend [2] and Reinforcement Learning Guidance (RLG) [3] further investigate how to combine a pretrained model with an RL-finetuned model to adjust the strength of reward alignment at inference time. In particular, RLG differs from this paper only in the parameterization of its linear combination coefficients and similarly interprets model extrapolation from the perspectives of implicit rewards and KL-regularized RL. Although the above methods are mainly designed for continuous diffusion, whereas RFG operates on discrete token logits, the authors are advised to further clarify the special problems that this transfer must address in the dLLM setting and the additional algorithmic or theoretical contributions relative to existing guidance methods.

  2. There is a lack of performance comparisons under a strictly unified inference compute budget. RFG requires running the post-trained model and the reference model separately at each denoising step, so its model forward computation is significantly higher than that of directly using the post-trained model. Although the paper compares with CFG, which also requires two forward passes, this is not yet sufficient to demonstrate the benefit of RFG relative to increasing the sampling compute of a standard model. The authors are advised to fix the total Number of Function Evaluations (NFE)—for example, by halving the number of RFG denoising steps while keeping the total number of forward calls the same—and then compare with the original model. They should also report accuracy and actual inference latency under different NFEs to verify whether the additional reference forward pass yields sufficient performance gains.

[1] Karras, T., et al. (2024). Guiding a Diffusion Model with a Bad Version of Itself. NeurIPS 2024.

[2] Cheng, M., et al. (2026). Diffusion Blend: Inference-Time Multi-Preference Alignment for Diffusion Models. ICLR 2026. First released in 2025.

[3] Jin, L., et al. (2025). Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance. arXiv:2508.21016.

[4] Yuan, L., et al. (2025). Free Process Rewards without Process Labels. ICML 2025.

Questions for authors

See Weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZKEq (1/3): W1

We thank the reviewer for the careful review and for recognizing the practical value of improving inference from existing checkpoints without additional training, as well as the broad scope of our experiments, the simplicity of RFG's implementation, and its gains across models, training stages, and tasks. We address each concern below.


Novelty Relative to Guidance in Image Diffusion (W1)

The reviewer's concern is what RFG contributes beyond two-model guidance for image diffusion, where Autoguidance [1], DB-KLA [2], and RLG [3] also extrapolate between two models.


TL;DR

  • The extrapolation form is shared with [1-3]. We claim no novelty for the form, which is already explicitly stated in Sec. 3.2 & 3.3.
  • The transfer to dLLMs must handle a reward defined on discrete transitions and the selection of positions, neither of which arises in continuous diffusion.
  • We contribute theoretically a closed-form process reward on partially masked states of dLLMs, and empirically show that only the contrast between checkpoints improves post-trained dLLMs in all 48 settings.

RFG’s novelty lies in addressing challenges specific to dLLMs that continuous diffusion does not pose:

  • The reward must be defined on transitions. RLG[3]'s reward is a log-ratio of noisy marginal densities and acts through its gradient, a scaled difference of scores. The process reward of [4] scores left-to-right prefixes. A partially masked sequence has no such gradient and no prefix, and the likelihood of a response is a sum over denoising trajectories. We define the reward on the trajectory and prove that it decomposes exactly into transition rewards βlog⁡pθ(xt−1∣xt)pref(xt−1∣xt)\beta\log\frac{p_\theta(x_{t-1}\mid x_t)}{p_{\mathrm{ref}}(x_{t-1}\mid x_t)}, each computed in closed form from the two models' logits (Prop. 3.1).
  • A transition selects positions as well as tokens. Under confidence-based decoding, guidance changes the unmasking order as well as the tokens, and a revealed token is never revised. Image diffusion has no counterpart. Our response to Reviewer 6ufE (W2, Q2) shows that a position rule shared by the policy and the reference cancels in the reward, and that at the selected positions, the reward-tilted token distribution is exactly the guided softmax that Alg. 1 passes to its decoder.

RFG's two models require no additional training and need not come from RL. Autoguidance [1] trains a deliberately degraded model, while RFG reuses an existing checkpoint. DB-KLA[2] and RLG[3] are formulated for RL-trained models, while our experiments extend this to SFT checkpoints.

Empirically, RFG is the only guidance method compared that improves post-trained dLLMs in all 48 settings. Results on images do not establish that guidance of this form improves reasoning in a dLLM. The form has been carried over to dLLMs: CFG is the direct transfer from image diffusion, and A-CFG (Li et al., 2025) and FoCore (Feng et al., 2026) adapt it. All three share RFG’s cost (two forward passes per step) and differ in the second branch. We compare them in our response to Reviewer jYax and the revised Appendix. The gain therefore comes from the policy/reference contrast that RFG’s formulation prescribes, and not from extrapolation as such.

We thank the reviewer for raising this point and have revised the paper accordingly: Section 5 discusses [1-3], Section 3.3 restates the contribution, and the experiments add A-CFG and FoCore.

References

  • Li, P., et al. (2025). Adaptive Classifier-Free Guidance via Dynamic Low-Confidence Masking. NeurIPS 2025.
  • Feng, J., et al. (2026). Focus on the Core: Empowering Diffusion Large Language Models by Self-Contrast. arXiv:2605.01373.
AuthorsOpenReview ↗

Response to Reviewer ZKEq (2/3): W2 (a) Matched-NFE comparison

Comparison under a Unified Inference Compute Budget: NFE and Latency (W2)

The reviewer requests comparing RFG with increasing the sampling compute of a standard model, and asks to (a) fix the total Number of Function Evaluations (NFE) and compare with the original model, and (b) report actual inference latency.

We answer in the two parts the reviewer requests: (a) accuracy under matched NFE, and (b) measured latency.


TL;DR

  • (a) Increasing sampling compute helps the original model at first, but the gains saturate. RFG performs better in all 16 model-benchmark settings under a matched budget of 1024 NFEs, with each method at its best configuration within the budget.
  • (b) Equal NFE does not mean equal latency, because standard scaling increases both the number of evaluations and the sequence length processed by each evaluation. From the original model at 512 NFEs, standard scaling to 1024 NFEs takes 3.06× to 3.57× the latency. RFG reaches the same 1024 NFEs with its two forward passes realized in parallel on two GPUs, at no more than 1.07×. In a hardware-matched setting on the same single GPU as the original model, RFG takes 1.68× to 1.97×.

(a) Matched NFE: RFG raises the best accuracy beyond where standard sampling-compute scaling saturates

We fix the total NFE as the reviewer suggests. A denoising step costs the original model one NFE and RFG two, so at the same NFE we compare RFG at half the steps of the original model. With the standard samplers here, at least one token is unmasked per step, so the generation length LL is no less than the number of steps TT. Therefore, scaling the NFE of the original model, i.e., its steps, requires scaling its generation length at the same time. We keep L=TL=T for both methods (same as our main tables), so one token is unmasked per step. This avoids the errors locked in when several tokens are unmasked at once, and LLaDA’s official repository reports that the model performs best in this setting.

Increasing the sampling compute of the original model helps at first and then saturates. The results of the original model at 1024 NFEs (L=T=1024L=T=1024) are shown in Table R1: accuracy changes by −0.6% on average relative to 512 NFEs and falls in 11 of 16 settings. Our main tables therefore stop at 512, the last length that still improves the original model on average, and the saturation is already visible on MBPP at LL of 256 vs 512.

We set the matched budget to 1024 NFEs for two reasons: L=T=1024L=T=1024 is the setting LLaDA uses to report official results on GSM8K, MATH, HumanEval, and MBPP for its base model, and it gives the original model enough NFEs to reach its best accuracy, since it has saturated by then. We compare the two methods (i) at exactly 1024 NFEs each, the original model at L=T=1024L=T=1024 against RFG at L=T=512L=T=512, and (ii) at their best configurations within the budget, to avoid forcing either method to use a longer generation when a shorter one performs better: the best result among L=T∈{128,256,512,1024}L=T\in\lbrace128,256,512,1024\rbrace for the original model and among L=T∈{128,256,512}L=T\in\lbrace128,256,512\rbrace for RFG, so neither exceeds 1024 NFEs.

Results are in Table R1. At exactly 1024 NFEs each, RFG exceeds the original model in all 16 settings, by +5.1% on average. At their best configurations, RFG still reaches a higher best accuracy in all 16 settings, by +3.7% on average and up to +12.3%.

Table R1: Accuracy under a matched budget of 1024 NFEs: at exactly 1024 NFEs, and each method's best result within 1024 NFEs.

BenchmarkModelOriginal @ 1024 NFEsRFG @ 1024 NFEsRel. Gain @ 1024 NFEsBest Original (L)Best RFG (L)Rel. Gain (best vs. best)
GSM8KLLaDA-Instruct83.384.6+1.6%83.7 (512)84.6 (512)+1.1%
GSM8KDream-Instruct83.885.1+1.6%84.8 (512)85.1 (512)+0.4%
GSM8Kd1-LLaDA83.584.6+1.3%83.6 (512)84.7 (256)+1.3%
GSM8KLLaDA-1.584.385.3+1.2%84.8 (512)85.3 (512)+0.6%
MATH-500LLaDA-Instruct42.444.0+3.8%42.6 (512)44.0 (512)+3.3%
MATH-500Dream-Instruct49.852.6+5.6%50.4 (512)52.6 (512)+4.4%
MATH-500d1-LLaDA41.843.8+4.8%42.6 (512)43.8 (512)+2.8%
MATH-500LLaDA-1.543.444.0+1.4%43.4 (1024)44.0 (512)+1.4%
HumanEvalLLaDA-Instruct43.947.6+8.4%45.1 (512)47.6 (512)+5.5%
HumanEvalDream-Instruct61.665.2+5.8%62.2 (512)66.5 (256)+6.9%
HumanEvalLLaDA-1.541.546.3+11.6%45.7 (512)46.3 (512)+1.3%
HumanEvaldiffuCoder70.178.7+12.3%70.1 (128, 1024)78.7 (512)+12.3%
MBPPLLaDA-Instruct51.052.9+3.7%54.9 (256)55.6 (256)+1.3%
MBPPDream-Instruct65.470.4+7.6%65.4 (1024)70.4 (512)+7.6%
MBPPLLaDA-1.551.855.6+7.3%54.5 (256)57.2 (256)+5.0%
MBPPdiffuCoder72.475.5+4.3%72.8 (512)75.9 (256)+4.3%
AuthorsOpenReview ↗

Response to Reviewer ZKEq (3/3): W2 (b) Measured latency

(b) Measured latency: RFG runs its two forward passes in parallel, and reaches the same NFE at lower latency than scaling even on a single GPU

Equal NFE does not mean equal latency. NFE counts forward passes, not how long each takes or whether they can run at the same time. Standard scaling increases both steps and generation length, so the original model's additional forward passes are sequential and each processes a longer sequence. RFG's additional forward passes leave the sequence length unchanged and run in parallel with the policy's. The NFE-matched comparison in (a) is therefore conservative for RFG, and we report the measured latency below.

RFG's two forward passes per step are realized in parallel. The policy and reference models read the same current state and neither requires the other's output, so we place them on two GPUs and run concurrently. As shown in Table R2, RFG at L=T=512L=T=512 runs at no more than 1.07× the latency of the original model in the same L=T=512L=T=512 setting, while improving accuracy in all 16 model-benchmark settings by +2.6 points on average.

In contrast, increasing the sampling compute of the original model is inherently sequential, because each denoising step takes the output of the previous step as input. Scaling the original model from 256 to 512 NFEs takes 2.81× to 3.19× the latency for +2.2 points on average, and from 512 to 1024 NFEs takes 3.06× to 3.57× for −0.3. Each doubling costs more than 2× because it not only doubles the number of steps but also lengthens the sequence that every forward pass processes.

Hardware-matched setting. Parallel RFG uses a second GPU, so we also restrict RFG to the same single GPU as the original model, where its two passes run sequentially. RFG only adds a second forward pass per step and leaves the sequence length unchanged, so it takes 1.68× to 1.97× the latency of the original model at L=T=256L=T=256 and 512512, less than the corresponding doubling in every setting. With the hardware and the NFE both matched, RFG at L=T=512L=T=512 takes 0.49× to 0.61× the latency of the original model at L=T=1024L=T=1024 and performs better in all 16 settings. This sequential comparison demonstrates that the latency advantage under matched-NFE does not depend on having a second GPU.

Table R2: Latency of adding NFEs as a multiplier of the latency before the change (A100-80GB, batch size 1, averaged over benchmarks). Absolute latency values are in revised Appendix. Each entry is the average over the benchmarks of that model. Ranges quoted in the text are the minimum and maximum over the 16 individual model-benchmark settings, so they extend slightly beyond the table entries.

ModelOriginal, 256 → 512Original, 512 → 1024Original, 512 → RFG, 512 & one GPU (sequential)Original, 512 → RFG, 512 & two GPUs (parallel)
NFE256 → 512512 → 1024512 → 1024512 → 1024
LLaDA-Instruct3.08×3.44×1.96×1.02×
Dream-Instruct2.92×3.46×1.88×1.05×
d1-LLaDA3.16×3.41×1.69×1.05×
LLaDA-1.53.07×3.44×1.96×1.01×
diffuCoder2.88×3.28×1.86×1.03×
Mean accuracy gain (points)+2.2−0.3+2.6+2.6

Summary. RFG offers two advantages over increasing the sampling compute of the original model.

  • Under matched NFE, scaling steps and length is effective at small budgets but it saturates. With enough NFEs for the original model to reach its best accuracy, RFG exceeds that accuracy in all 16 settings under the same NFE budget.
  • RFG achieves higher accuracy with lower measured latency than a scaled original model at the same 1024 NFEs. With its two forward passes realized in parallel on two GPUs, RFG runs at no more than 1.07× the latency of the original model at L=T=512L=T=512. In a hardware-matched setting on one GPU, RFG takes 1.68× to 1.97×, versus 3.06× to 3.57× for scaling the original model to the same NFE.

We thank the reviewer for raising this point and have revised the paper accordingly: a new discussion reports the matched-NFE comparison and the measured latency and notes that scaling steps and length could be the better use of NFEs at some smaller budgets before saturation, and a new Limitations paragraph states the additional compute RFG requires.


We hope these clarifications and results address the reviewer's concerns and welcome further discussion.

We thank the reviewer again for the careful review. As the discussion period is closing, we would like to follow up on our response and revision.

It addresses the two concerns raised:

  • The novelty relative to two-model guidance in image diffusion (W1), where we clarify the contributions specific to dLLMs and add the A-CFG and FoCore baselines as empirical evidence
  • The comparison under a unified compute budget (W2), with new matched-NFE and latency results (Tables R1 and R2).

We would be grateful if the reviewer could let us know whether these results and clarifications resolve the concerns.

Reviewer_6ufEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reward-Free Guidance (RFG), which combines the logits of a post-trained diffusion language model and its base reference model to improve generation without additional training or an external reward model. The method is motivated by a decomposition of trajectory-level likelihood ratios into step-wise rewards and evaluated on mathematical reasoning, code generation, and instruction following.

Critical strengths
  1. The method is simple to implement using existing checkpoints and can further improve models that have already undergone RL post-training.
  2. The improvements at a fixed number of denoising steps are consistent across models and tasks. Comparisons with CFG and policy-only sharpening also provide evidence for the usefulness of the reference model.
Critical weaknesses
  1. The trade-off between inference cost and performance remains unclear. RFG requires two model forward passes per denoising step, so a natural alternative is to spend the same budget on more denoising steps with the original model. Comparing the reported results at matched forward-pass counts, this alternative performs better in most settings. This does not diminish the benefit of avoiding additional training, but leaves RFG’s practical advantage over simply increasing denoising steps insufficiently established.
  2. The connection between the theoretical formulation and the implemented sampler needs clarification. The likelihood-ratio derivation holds when conditioning on a common set of positions to unmask. However, confidence-based decoding can change both token predictions and the unmasking order after guidance is applied. It is unclear whether the transition distributions in the theory account for this position-selection process. Since the process-reward interpretation is a central contribution, this distinction matters for assessing its explanatory value.
Questions for authors
  1. Under matched inference budgets or measured latency, in which settings does RFG offer an advantage over increasing the original model’s denoising steps?
  2. Do the theoretical transition distributions include position selection? If the derivation assumes a shared position-selection rule, how does that assumption correspond to the implementation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6ufE (1/4): W1, Q1 (a) matched budget

We thank the reviewer for the detailed review and for recognizing RFG's ease of implementation, its ability to further improve RL-post-trained models, the consistency of its gains across models and tasks, and the evidence from the CFG and policy-only sharpening results for the usefulness of the reference model. We address each concern below.


Inference Cost–Performance Trade-off (W1, Q1)

The reviewer's concern is whether RFG’s additional reference-model pass is a better use of inference resources than increasing denoising steps. The reviewer asks how RFG is advantageous under (a) matched inference budget or (b) measured latency.

We address both aspects raised by the reviewer: (a) best accuracy under matched inference budget, and (b) measured decoding latency.


TL;DR

  • (a) More steps or a longer generation help the original model at first, but the gains saturate. RFG reaches a higher best accuracy in all 16 model-benchmark settings, with each method at its best configuration within the same budget of 1024 forward passes.
  • (b) RFG runs its additional reference-model pass in parallel, so on two GPUs it takes at most 1.07× the latency of the original model in the same setting at the largest generation length of our main tables.

(a) Matched inference budget: RFG raises the best accuracy beyond where step and length scaling saturates

We evaluate the standard scaling strategy the reviewer mentions: increasing steps (TT) and, after reaching one token per step, extending generation length (LL).

Increasing TT toward LL can be preferable to guidance because fewer tokens are unmasked per step, so fewer errors are locked in since an unmasked token is never revised here. Therefore, our main experiments (Tables 2 to 5) use T=LT=L, one token per step. From there the budget can only grow through a longer generation LL, which helps at first and then saturates.

The results of the original model at L=T=1024L=T=1024 are shown in Table R1: accuracy changes by −0.6% on average relative to L=512L=512 and falls in 11 of 16 settings. Our main tables therefore stop at 512, the last length that still improves the original model on average, and the saturation is already visible on MBPP at LL of 256 vs 512.

We therefore compare the best accuracy each method reaches within 1024 forward passes. We choose this budget for two reasons: L=T=1024L=T=1024 is the setting LLaDA uses to report official results on GSM8K, MATH, HumanEval, and MBPP for its base model, and it gives the original model enough budget to reach its best accuracy, since it has saturated by then. To avoid forcing either method to use a longer generation when a shorter one performs better, we compare their best configurations: for original model we choose the best result among L=T∈{128,256,512,1024}L=T\in\lbrace128,256,512,1024\rbrace, and for RFG we choose among L=T∈{128,256,512}L=T\in\lbrace128,256,512\rbrace, so neither exceeds 1024 passes.

Results are in Table R1. RFG reaches a higher best accuracy in all 16 settings, by +3.7% on average and up to +12.3%. At exactly 1024 passes each, RFG at 512 also exceeds the original model at 1024 in all 16 settings.

Table R1: Original model performance at L=T=1024L=T=1024, and each method’s best result within this same budget.

BenchmarkModelOriginal, L = T = 1024Best Original (L)Best RFG (L)Rel. Gain @ matched budget
GSM8KLLaDA-Instruct83.383.7 (512)84.6 (512)+1.1%
GSM8KDream-Instruct83.884.8 (512)85.1 (512)+0.4%
GSM8Kd1-LLaDA83.583.6 (512)84.7 (256)+1.3%
GSM8KLLaDA-1.584.384.8 (512)85.3 (512)+0.6%
MATH-500LLaDA-Instruct42.442.6 (512)44.0 (512)+3.3%
MATH-500Dream-Instruct49.850.4 (512)52.6 (512)+4.4%
MATH-500d1-LLaDA41.842.6 (512)43.8 (512)+2.8%
MATH-500LLaDA-1.543.443.4 (1024)44.0 (512)+1.4%
HumanEvalLLaDA-Instruct43.945.1 (512)47.6 (512)+5.5%
HumanEvalDream-Instruct61.662.2 (512)66.5 (256)+6.9%
HumanEvalLLaDA-1.541.545.7 (512)46.3 (512)+1.3%
HumanEvaldiffuCoder70.170.1 (128, 1024)78.7 (512)+12.3%
MBPPLLaDA-Instruct51.054.9 (256)55.6 (256)+1.3%
MBPPDream-Instruct65.465.4 (1024)70.4 (512)+7.6%
MBPPLLaDA-1.551.854.5 (256)57.2 (256)+5.0%
MBPPdiffuCoder72.472.8 (512)75.9 (256)+4.3%
AuthorsOpenReview ↗

Response to Reviewer 6ufE (2/4): W1, Q1 (b) measured latency

(b) Measured latency: RFG uses additional compute in parallel, at 1.01× to 1.07× the latency of the original model at L=T=512L=T=512

RFG can turn additional inference compute into accuracy gains with little additional wall-clock latency by evaluating its two models concurrently.

The forward passes of RFG’s policy and reference models read the same current state and neither requires the other’s output, so they can run in parallel on two GPUs with little added latency. Specifically, as shown in Table R2, RFG at L=T=512L=T=512 improves accuracy in all 16 model-benchmark settings by +2.6 points on average, while running at no more than 1.07× the latency of the original model in the same L=T=512L=T=512 setting.

In contrast, scaling steps, as the reviewer suggests, or generation length is inherently sequential, because each denoising step takes the output of the previous step as input. Scaling the original model from L=T=256L=T=256 to 512512 takes 2.81× to 3.19× the latency for +2.2 points on average, and from 512512 to 10241024 takes 3.06× to 3.57× for −0.3. Each doubling costs more than 2× because it not only doubles the number of steps but also lengthens the sequence that every forward pass processes. RFG, however, only adds a second forward pass per step and leaves the sequence length unchanged. Thus, even when no second GPU is available, RFG (sequential implementation with same single GPU as the original model) takes 1.68× to 1.97× the latency of the original model with same one GPU at L=T=256L=T=256 and 512512, less than the corresponding doubling in every setting.

Table R2: Latency of scaling the original model versus adding RFG, as a multiplier of the latency before the change (A100-80GB, batch size 1, averaged over benchmarks). Absolute latency values are in revised Appendix. Each entry is the average over the benchmarks of that model. Ranges quoted in the text are the minimum and maximum over the 16 individual model-benchmark settings, so they extend slightly beyond the table entries.

ModelOriginal, 256 → 512Original, 512 → 1024Original, 512 → RFG, 512 & one GPU (sequential)Original, 512 → RFG, 512 & two GPUs (parallel)
NFE256 → 512512 → 1024512 → 1024512 → 1024
LLaDA-Instruct3.08×3.44×1.96×1.02×
Dream-Instruct2.92×3.46×1.88×1.05×
d1-LLaDA3.16×3.41×1.69×1.05×
LLaDA-1.53.07×3.44×1.96×1.01×
diffuCoder2.88×3.28×1.86×1.03×
Mean accuracy gain (points)+2.2−0.3+2.6+2.6

Summary. RFG offers two advantages.

  • For latency-sensitive deployment, RFG improves the original model at the same LL and TT in every setting we evaluated with small measured latency overhead at L=T=512L=T=512 when additional parallel compute is available. Without parallel compute, RFG is still faster than scaling the original model to the same number of forward passes.
  • For accuracy-focused deployment, scaling steps and length is effective at small budgets but it saturates. With enough budget for the original model to reach its best accuracy, RFG exceeds that accuracy in all settings under the same budget.

We thank the reviewer for raising this point and have revised the paper accordingly: a new discussion clarifies when RFG is advantageous and notes that scaling steps and length could be preferable at some smaller budgets before saturation, and a new Limitations paragraph states the additional compute RFG requires.

AuthorsOpenReview ↗

Response to Reviewer 6ufE (3/4): W2, Q2 parts 1–2

Connection between the Theoretical Formulation and the Implemented Sampler (W2, Q2)

The reviewer asks whether the theoretical transition distributions include position selection, and how the shared-position assumption corresponds to an implementation in which guidance can change the unmasking order. We thank the reviewer for identifying this and clarify below.


TL;DR

  • The theory compares both checkpoints under a common position selector, which may itself use guided logits. Sharing that selector does not require preserving the unguided model’s unmasking order.
  • The logit transformation is exact for the resulting conditional categorical token kernel. Greedy or nucleus decoding is a subsequent operation.

1. Theory: transition kernels under a shared position selector

Let UtU_t denote the positions revealed at step tt, yt={yti}i∈Uty_t=\lbrace y_t^i\rbrace_{i\in U_t} their token assignments, and πmi(⋅∣xt)=softmax⁡(ℓmi)\pi_m^i(\cdot\mid x_t)=\operatorname{softmax}(\ell_m^i) checkpoint mm’s categorical token distribution. Under a shared position-selection rule σt\sigma_t, the categorical transition kernels are Kmσ(Ut,yt∣xt)=σt(Ut∣xt)∏i∈Utπmi(yti∣xt),m∈{θ,ref}.K_m^\sigma(U_t,y_t\mid x_t)=\sigma_t(U_t\mid x_t)\prod_{i\in U_t}\pi_m^i(y_t^i\mid x_t),\qquad m\in\lbrace\theta,\mathrm{ref}\rbrace. Here, σt\sigma_t may depend on the state, timestep, and both checkpoints’ logits, but not on the current sampled token assignments. The shared selector cancels in the transition likelihood ratio: rθt(Ut,yt∣xt)=βlog⁡Kθσ(Ut,yt∣xt)Krefσ(Ut,yt∣xt)=β∑i∈Utlog⁡πθi(yti∣xt)πrefi(yti∣xt).r_\theta^t(U_t,y_t\mid x_t)=\beta\log\frac{K_\theta^\sigma(U_t,y_t\mid x_t)}{K_{\mathrm{ref}}^\sigma(U_t,y_t\mid x_t)}=\beta\sum_{i\in U_t}\log\frac{\pi_\theta^i(y_t^i\mid x_t)}{\pi_{\mathrm{ref}}^i(y_t^i\mid x_t)}. Together with positive categorical token probabilities, the shared selector guarantees common support. The trajectory decomposition and Proposition 3.1 therefore hold exactly for the trajectory distributions induced by these kernels.

These are not the kernels obtained by independently running each checkpoint’s native confidence-based decoder. If the two checkpoints instead used different selectors in this construction, their likelihood ratio would retain an additional βlog⁡(σθ/σref)\beta\log(\sigma_\theta/\sigma_{\mathrm{ref}}) term. If deterministic selectors choose different subsets at a given state, their one-step transition supports are disjoint at that state. We now explicitly state the shared-selector and common-support assumptions.

2. Implementation: sharing the guided selector does not fix the unguided order

Algorithm 1 evaluates both checkpoints on the same current state xtx_t and prompt (lines 3–4), forms the guided logits ℓRFG=(1+w)ℓθ−wℓref,πRFG=softmax⁡(ℓRFG),\ell_{\mathrm{RFG}}=(1+w)\ell_\theta-w\ell_{\mathrm{ref}},\qquad\pi_{\mathrm{RFG}}=\operatorname{softmax}(\ell_{\mathrm{RFG}}), and invokes the denoising strategy S\mathcal S once, using these guided logits (line 6). Neither checkpoint runs a separate decoding trajectory.

For deterministic confidence-based position selection, including the temperature-zero LLaDA setting, the induced selector can be written as Ut=st(w)(xt):=st ⁣(xt,ℓRFG(xt)).U_t=s_t^{(w)}(x_t):=s_t\negthinspace\left(x_t,\ell_{\mathrm{RFG}}(x_t)\right). For fixed checkpoints and guidance strength ww, this is a well-defined state-dependent function. Instantiating both comparison kernels with σt(w)(Ut∣xt)=1 ⁣[Ut=st(w)(xt)]\sigma_t^{(w)}(U_t\mid x_t)=\mathbf1\negthinspace\left[U_t=s_t^{(w)}(x_t)\right] gives the shared-selector construction above.

Crucially, the selector is shared between the policy and reference comparison kernels, not between guided and unguided generation. There is no requirement that st(w)(xt)=st(0)(xt)s_t^{(w)}(x_t)=s_t^{(0)}(x_t). Guidance can therefore change the confidence ranking and unmasking order without contradicting the reward decomposition. The comparison evaluates both checkpoints’ token probabilities at the same visited state and positions, and it does not require their independently decoded trajectories to coincide.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.