acceptodds
Under review as a conference paper at ICLR 2027

FastGuide: Accelerating Reward Guidance for Diffusion Large Language Models

Abstract

Gradient-based reward guidance provides a flexible way to use downstream reward models to control masked diffusion language models at inference time. However, its computational cost remains high as each decoding iteration incurs expensive diffusion model forward passes and reward model backpropagation steps. To address this, we introduce FastGuide, an adaptive hybrid of parallel and autoregressive decoding to accelerate reward guidance for diffusion language models. In analogy to parallel decoding, FastGuide amortizes the cost of reward model backpropagation by computing guidance once per decoding step and reusing it to generate multiple tokens. Within each decoding step, FastGuide makes diffusion forward passes autoregressive by unmasking tokens one at a time while efficiently recomputing token distributions after each unmasking by utilizing KV caching techniques and sparse recomputation of attention. Lastly, to adapt hybrid decoding to the model's confidence, FastGuide defers any token that the model is unconfident about under its recomputed distribution. Experiments on three reward benchmarks demonstrate that FastGuide is up to 4.4x faster than reward guidance baselines while retaining similar generation quality.

Primary area:
generative models
Keywords:
text diffusion, reward guidance, alignment, parallel decoding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CSdrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FastGuide accelerates reward-guided generation in dLLMs. It reuses the guidance vector across multiple token commitments while updating the base model’s predictions after each commitment. Low-confidence tokens are deferred. Experiments on Dream and LLaDA compare generation quality and runtime against sequential and parallel decoding methods.

Critical strengths
  1. The method separates guidance updates from base model updates. Figure 3 supports this design. Reusing the guidance while updating token predictions helps reduce the quality loss of parallel generation.

  2. The paper reports substantial runtime improvements over sequential reward-guided decoding, with guidance computation included in the measurements. This demonstrates a practical improvement within the tested settings.

Critical weaknesses
  1. All generation experiments use Dream-7B-Instruct or LLaDA-8B-Instruct. There is no evaluation on a more competitive model such as SDAR with its optimized inference engine. The results therefore do not establish whether FastGuide remains useful when the underlying model and decoder are stronger.

  2. The benefit over simpler alternatives at equal runtime is unclear. On Dream/RM-Bench, FastGuide and unguided BoN take 6.2 and 6.3 seconds, respectively, but their LMUnit scores are 4.04 and 4.06. All experiments also use 128-token outputs, leaving its effectiveness on longer generation untested.

Questions for authors
  1. Does FastGuide improve quality at equal runtime on SDAR or other competitive dLLMs, using its optimized inference engine and including all guidance costs?

  2. How does FastGuide compare with BoN and DAPD at equal runtime on longer outputs, using task accuracy or code execution rather than reward-model and LLM-judge scores?

1 comment
AuthorsOpenReview ↗

Rebuttal Response

We thank the reviewer for their insightful feedback. We now respond in detail to each question.

“Does FastGuide improve quality at equal runtime on SDAR or other competitive dLLMs, using its optimized inference engine and including all guidance costs?” (Weakness 1 and Question 1)

We agree it is valuable to discuss FastGuide in the context of modern block-diffusion models such as SDAR. We note, however, that our work accelerates gradient-based reward guidance, and to the best of our knowledge there is no prior work on gradient-based reward guidance for block-diffusion models such as SDAR. This is because reward guidance hinges on constructing a completion of the full response at every intermediate sampling step, which is then passed to the reward model to compute gradients. A full-canvas diffusion model provides this completion essentially for free, since every forward pass outputs a marginal distribution at every position of the response (which APS and EntRGi exploit). A block-diffusion model exposes marginals only inside the block currently being denoised, which is not sufficient to construct a response beyond the block boundary.

Nonetheless, FastGuide itself is compatible with any response completion algorithm. To demonstrate this, we try a very simple implementation of gradient-based reward guidance on SDAR that uses a full sampled rollout at every timestep to construct a reward model input. We then compare doing sequential gradient-based reward guidance versus FastGuide using the same response completion algorithm. The below table shows results on 50 prompts from JudgeBench using the SDAR-8B-Chat optimized inference engine with block size 32 and guidance with a Skywork-Reward-V2-Qwen3-1.7B model. Our preliminary experiments show that FastGuide is able to maintain similar performance to sequential guidance while being significantly faster. In contrast, when performing parallel generation with k=8, generation quality drastically drops.

It is important to note that the reported inference time includes the time to generate rollouts at every timestep, which is clearly very expensive. It is an important direction of future work to develop efficient response completion algorithms for block-AR diffusion models.

SDAR-8B-Chat, JudgeBenchLMUnits/gen
Sequential guidance4.33318
Confidence k=81.7153
FastGuide k=8, τ=.54.2797

“How does FastGuide compare with BoN and DAPD at equal runtime on longer outputs, using task accuracy or code execution rather than reward-model and LLM-judge scores?” (Weakness 2 and Question 2)

Below, we include performance on GSM8K and MATH500 benchmarks using 256-length generations and task accuracy as a metric (Top@1 is the answer picked with highest reward model score). These results show similar trends to Table 1, where FastGuide retains a large proportion of sequential guided decoding (first row) while being around 5x faster. Furthermore, BoN sampling requires around 16 generations to be close to FastGuide performance, greatly increasing sampling time. At roughly equal runtime compared to DAPD and BoN, FastGuide has higher task accuracy. Interestingly, on the harder MATH500 benchmark, there is a larger gap between sequential and parallel decoding, possibly due to parallel decoding being harder for more complex reasoning tasks.

GSM8K (300 prompts)

MethodTop@1 accAvg accs/gen
SequentialBoN@448.3 ± 2.932.9 ± 1.421.3
BoN@1663.3 ± 2.832.9 ± 1.485.2
EntRGi64.3 ± 2.848.6 ± 2.078.1
ParallelDAPD46.0 ± 2.932.9 ± 2.013.6
FastGuide65.0 ± 2.850.5 ± 1.914.8

MATH500 (200 prompts)

MethodTop@1 accAvg accs/gen
SequentialBoN@430.5 ± 3.320.2 ± 1.721.2
BoN@1640.0 ± 3.520.2 ± 1.784.8
EntRGi44.5 ± 3.528.1 ± 2.277.0
ParallelDAPD23.5 ± 3.018.0 ± 1.910.2
FastGuide35.0 ± 3.429.2 ± 2.415.8
Reviewer_d4ZeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a strategy to accelerate gradient-based reward guidance of discrete diffusion models. Instead of computing the reward gradient after every unmasked token, it computes it once per step and reuses it for several tokens. Those tokens are then unmasked one at a time with a cheap partial recomputation of the model's predictions. On three reward benchmarks with two discrete diffusion models, it is 2.8-4.4× faster than standard sequential guided decoding with similar reward scores.

Critical strengths
  • Speeding up reward-guided decoding is an important problem and avoiding recomputing the gradient from the reward model at every step is needed. The paper shows that you can reuse guidance across many steps without too much impact.

  • I appreciate the amount of details that are provided given hyperparameters, models that were used, datasets, etc. Although there is a substantial amount of work needed to replicate the paper, and the method has a lot of important components, it seems like it would be possible to replicate it given the provided information.

Critical weaknesses
  • There are many components to the method that might not be so easily understandable. I think the paper would benefit from clearer explanation of the components, possibly supported by additional figures. Figure 2 is also hard to understand.

  • While the results are interesting, there are many other results that could be provided. How would the proposed method adapt to current block-AR diffusion models? Nowadays, many recent models perform in block-AR mode over full-canvas (which is not trivial to kv-cache and has some difficulties regarding variable length generation). Furthermore, the number of decoded tokens in the experiments is still small, at only 128 tokens. It would be interesting to also see experiments where more tokens are decoded.

Questions for authors
  • In Table 4 of the appendix, there are some results with different c models and using Llama as the reward model provides huge improvement over using Qwen. Is this because the reward model is larger? Or is there some interplay between that model using a tokenizer that has less/more overlap with the vocab of the discrete diffusion models?
3 comments
AuthorsOpenReview ↗

Rebuttal Response

We thank the reviewer for their positive evaluation of our work. We now respond in detail to each question.

“Clearer explanation of components” (Weakness 1)

We thank the reviewer for pointing this out. To improve clarity, we have added a walkthrough of Figure 2 in the appendix (in blue text) as well as a subsection in Section 3 clarifying the different components of the method.

“Adapting to current block-AR diffusion models" (Weakness 2)

We agree that it is valuable to discuss FastGuide in the context of modern block-diffusion models. We note, however, that our work accelerates gradient-based reward guidance, and to the best of our knowledge there is no prior work on gradient-based reward guidance for block-AR diffusion models. This is because reward guidance hinges on constructing a completion of the full response at every intermediate sampling step, which is then passed to the reward model to compute gradients. A full-canvas diffusion model provides this completion essentially for free, since every forward pass outputs a marginal distribution at every position of the response (which APS and EntRGi exploit). A block-diffusion model exposes marginals only inside the block currently being denoised, which is not sufficient to construct a response beyond the block boundary. Nonetheless, FastGuide itself is compatible with any response completion algorithm. It is an important direction of future work to develop efficient response completion algorithms for block-AR diffusion models.

“Experiments where more tokens are decoded” (Weakness 2)

We agree that increasing the generation length is a valuable experiment. Below, we show results of FastGuide and baseline methods expanding the generation length to 256 tokens on the JudgeBench dataset. We find the same conclusions as Table 1 in our paper, where FastGuide retains a small gap to sequential guided decoding accuracy, while being ~5x faster.

MethodTop@1Seq Gaps/gen
SeqConf (BoN)+2.75 ±.16--28.3 ±3.0
SeqConf+3.71 ±.16--99.7 ±13.0
ParallelConf+1.19 ±.18−2.5224.9 ±3.3
ParallelDAPD+2.26 ±.18−1.4614.8 ±2.0
ParallelFastGuide+3.28 ±.16−0.4319.8 ±2.7

Explaining Table 4 of Appendix and Apparent Reward Model Discrepancies (Question 1)

To clarify, each block of Table 4 uses a different reward model, and Top@1 is reported on that model's own scale. These reward models are trained with a Bradley-Terry preference objective, which determines only score differences between responses, not an absolute level, so each model's scale can have an arbitrary offset. The Llama-based scores (≈7–8) and the Qwen-based scores (≈2–3) are therefore not comparable across blocks, and the higher numbers do not indicate better generations. The table should be read within each block: the ordering of methods and the gap to the sequential baseline are the meaningful quantities. LMUnit, a held-out judge, is on a common scale from 1-5 and can be compared across blocks.

Thank you for responding to my comments.

We thank the reviewer for pointing this out. To improve clarity, we have added a walkthrough of Figure 2 in the appendix (in blue text) as well as a subsection in Section 3 clarifying the different components of the method.

This indeed improves understanding, but I would still prefer that the Figure itself was also improved.

We agree that it is valuable to discuss FastGuide in the context of modern block-diffusion models. We note, however, that our work accelerates gradient-based reward guidance, and to the best of our knowledge there is no prior work on gradient-based reward guidance for block-AR diffusion models. This is because reward guidance hinges on constructing a completion of the full response at every intermediate sampling step, which is then passed to the reward model to compute gradients. A full-canvas diffusion model provides this completion essentially for free, since every forward pass outputs a marginal distribution at every position of the response (which APS and EntRGi exploit). A block-diffusion model exposes marginals only inside the block currently being denoised, which is not sufficient to construct a response beyond the block boundary. Nonetheless, FastGuide itself is compatible with any response completion algorithm. It is an important direction of future work to develop efficient response completion algorithms for block-AR diffusion models.

I understand that some adaptations need to be added, but I don't agree with the "for free aspect". In the sense of getting the marginals of every position, yes, that is true. However, full-canvas diffusion models cannot be deployed in practice. They cannot leverage any of the inference optimizations of AR models (like KV caching) nor support flexible length generation or long context. My comment was towards seeing how the proposed method would perform with models that we could actually use in practice.

We thank the reviewer for meaningfully engaging with us during the discussion period.

This indeed improves understanding, but I would still prefer that the Figure itself was also improved.

Thanks for the feedback! We think the confusion was because Figure 2 had many moving parts and notation. We have now updated it to be as simple as possible. The figure shows illustrating that during each hybrid decoding step for each subsequent candidate, we sparsely recompute the dLLM logits, add the cached guidance vector, and use the resulting confidence to decide whether to commit or defer the token. We hope that this updated figure is far more understandable. Thank you for helping us improve the clarity of our paper!

I understand that some adaptations need to be added, but I don't agree with the "for free aspect". In the sense of getting the marginals of every position, yes, that is true. However, full-canvas diffusion models cannot be deployed in practice. They cannot leverage any of the inference optimizations of AR models (like KV caching) nor support flexible length generation or long context. My comment was towards seeing how the proposed method would perform with models that we could actually use in practice.

It is true that there are challenges to deploying full-canvas diffusion models in practice, and we acknowledge that comment. We want to highlight the preliminary experiment we performed in discussion period as response to Reviewer CSdr. Specifically, we tried a very simple implementation of gradient-based reward guidance on a modern block-AR model, SDAR, that uses a full sampled rollout at every timestep to construct a reward model input. We then compare doing sequential gradient-based reward guidance versus FastGuide using the same response completion algorithm. The below table shows results on 50 prompts from JudgeBench using the SDAR-8B-Chat optimized inference engine with block size 32 and guidance with a Skywork-Reward-V2-Qwen3-1.7B model. Our preliminary experiments show that FastGuide is able to maintain similar performance to sequential guidance while being significantly faster. In contrast, when performing parallel generation with k=8, generation quality drastically drops. It is important to note that the reported inference time includes the time to generate rollouts at every timestep, which is clearly very expensive. Nonetheless, this reinforces our point that FastGuide itself is compatible with any response completion algorithm, which is a fascinating direction for extending gradient-based guidance methods to alternative models.

SDAR-8B-Chat, JudgeBenchLMUnits/gen
Sequential guidance4.33318
Confidence k=81.7153
FastGuide k=8, τ=.54.2797
Reviewer_DSywOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FastGuide reuses reward guidance across several token commitments while refreshing base-model predictions through sparse recomputation and cached keys/values. Low-confidence tokens are deferred. Experiments on Dream-7B and LLaDA-8B show improved speed-quality trade-offs on reward-benchmark prompts.

Critical strengths
  1. The computational diagnosis is useful. Figure 3 separates base-logit reuse from guidance reuse, empirically motivating the allocation of computation.

  2. The engineering evaluation is substantial. Ablations examine sparse updates, windows, deferral, ordering, and batching. Additional guidance algorithms, reward models, and LMUnit checks broaden the evidence.

Critical weaknesses
  1. The probabilistic interpretation is underspecified. Eq. 1 adds a logit-space guidance vector, not the scalar reward. This is algebraically valid, but the inherited gradient heuristic is not connected to conditionals of a specified reward-tilted joint. Agreement with sequential EntRGi does not establish that connection.

  2. Base-logit refresh cannot generally recover reward-induced dependencies. For independent uniform binary tokens, an equality reward leaves initial marginals uniform but creates dependence. Reusing even ideal initial marginal corrections yields a uniform joint. Eq. 6 therefore needs assumptions or targeted validation beyond Figure 4's average conditional TV.

  3. KV error accumulation remains unclear. Bidirectional attention lets unmasking change higher-layer K/V outside refreshed windows. Attending globally to stale entries does not recover an exact forward pass. The paper acknowledges approximation and reports local TV, but its accumulated effect on FastGuide trajectories, especially beyond 128 tokens, remains untested.

  4. The PMI interpretation exceeds the diagnostic. Figure 4 uses unguided dLLM probabilities. A pair can have negative base-model PMI but positive PMI after reward tilting. This measures base-model compatibility; it does not establish incompatibility under the reward-guided joint.

Questions for authors
  1. Theory and methods: see weaknesses

  2. Can you add more experiments on standard math/coding benchmarks?

2 comments
AuthorsOpenReview ↗

Rebuttal Response [1 / 2] (Weaknesses 1-4)

We thank the reviewer for their insightful feedback, recognizing the usefulness of our computational diagnostic and acknowledging our substantial evaluation. We now respond in detail to each question.

“Probabilistic Interpretation Underspecified” (Weakness 1)

We agree that a probabilistic interpretation of prior work on guidance algorithms is valuable. For a reward-tilted distribution, the ideal correction to the base dLLM logits depends on the gradients of a soft value function, which measures rewards of possible completions under the dLLM denoising distribution. This is made formal in [1], which mathematically shows the connection between gradient-based guidance and sampling from the reward-tilted joint (see specifically Section 5 and Algorithm 11 for discussion on guidance for masked discrete diffusion). We acknowledge that updates such as the prior work we present in Equation 1 still rely on certain approximations and understanding their theoretical foundations to the ideal reward-tilted sampler is an important open research question. We emphasize though that this does not detract from the soundness of our work, which is focused on accelerating gradient-based guidance algorithms for masked discrete diffusion models. For clarity on prior work and its probabilistic foundation, we have added a discussion of the probabilistic interpretation of [1] in Appendix A.1.

[1] M. Uehara et al. Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Review. arXiv:2501.09685

“Base-logit refresh cannot generally recover reward-induced dependencies” (Weakness 2)

We agree with the reviewer that refreshing the base logits cannot generally recover reward-induced dependencies. However, base-logit refresh serves a different purpose in Eq. 6. Guidance can cause two tokens to be proposed to be unmasked together even when committing the first lowers the base model’s probability of the second (Figures 1 and 4b). Refreshing the base logits allows the decoder to reconsider the second proposal.

Whether the guidance correction also needs to be refreshed is a separate, empirical question, which is tested directly by Figure 3. With the base logits refreshed in both cases, reusing guidance retains most of the generation quality achieved by recomputing it. We have clarified this after Equation 6 in the revised version of the paper.

To provide a targeted validation of Equation 6 and guidance caching, we compared token choices under cached and freshly recomputed guidance on Dream-7B with Skywork-0.6B on RM-Bench. The below table shows that about 8% of committed tokens in each step change after guidance is recomputed. On the other hand, tokens that are deferred under our confidence deferral step are 4x likely to be changed after recomputing guidance, which supports using confidence deferral.

CommitsToken changed by recomputing the guidance correction
Tokens that are not deferred8.5%
Tokens that are deferred to the next step36.7%

“KV error accumulation remains unclear” (Weakness 3)

It is not true that the accumulated effect on FastGuide trajectories remains untested. Appendix D.5 in our paper studies precisely this effect of KV error accumulation, and we show that local recomputed attention is a reliable approximation of the full bidirectional attention and does not strongly affect downstream performance. Below, we show results of FastGuide and baseline methods expanding the generation length to 256 tokens on the JudgeBench dataset. We find the same conclusions as Table 1 in our paper, where FastGuide retains a small gap to sequential guided decoding accuracy, while being ~5x faster. Compared to FastGuide utilizing full bidirectional attention, these results further show the effect of KV error on the final reward.

MethodTop@1 rewardLMUnits/gen
Conf3.714.2599.7
Conf (BoN@4)2.754.1728.3
DAPD2.264.0514.8
FastGuide3.284.1919.8
FastGuide with full bidirectional attention3.654.2443.6

“PMI Interpretation exceeds diagnostic” (Weakness 4)

We deliberately measured base model compatibility in Figure 4. This is to show that guidance more often co-proposes pairs the base model disfavors, which motivates re-checking the base-model term after each commit. The question raised by the reviewer is whether the reward might prefer a pair of tokens that has low base model PMI. We believe that practically, this is rare for two reasons. First, the experiment mentioned above in Weakness 2 shows that cached guidance introduces a relatively small number of token changes, and these do not strongly affect generation quality as measured by Figure 3. Second, when the confidence of the token falls, the token is deferred and guidance is recomputed anyways.

AuthorsOpenReview ↗

Rebuttal Response [2 / 2] (Question 2)

“More experiments on standard benchmarks" (Question 2)

Below, we include performance on GSM8K and MATH500 benchmarks using 256-length generations and task accuracy as a metric (Top@1 is the answer picked with highest reward model score). These results show similar trends to Table 1, where FastGuide retains a large proportion of sequential guided decoding (first row) while being around 5x faster. Furthermore, BoN sampling requires around 16 generations to be close to FastGuide performance, greatly increasing sampling time. At roughly equal runtime compared to DAPD and BoN, FastGuide has higher task accuracy. Interestingly, on the harder MATH500 benchmark, there is a larger gap between sequential and parallel decoding, possibly due to parallel decoding being harder for more complex reasoning tasks.

GSM8K (300 prompts)

MethodTop@1 accAvg accs/gen
SequentialBoN@448.3 ± 2.932.9 ± 1.421.3
BoN@1663.3 ± 2.832.9 ± 1.485.2
EntRGi64.3 ± 2.848.6 ± 2.078.1
ParallelDAPD46.0 ± 2.932.9 ± 2.013.6
FastGuide65.0 ± 2.850.5 ± 1.914.8

MATH500 (200 prompts)

MethodTop@1 accAvg accs/gen
SequentialBoN@430.5 ± 3.320.2 ± 1.721.2
BoN@1640.0 ± 3.520.2 ± 1.784.8
EntRGi44.5 ± 3.528.1 ± 2.277.0
ParallelDAPD23.5 ± 3.018.0 ± 1.910.2
FastGuide35.0 ± 3.429.2 ± 2.415.8

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision Summary

Revision Summary

The insightful feedback by the reviewers (DSyw, d4Ze, CSdr) has improved the experimental depth and clarity of our paper. We summarize below the changes made to the main paper (shown in blue text in the revision), which address all the reviewers’ comments.

  1. In Appendix A, we add a discussion of the probabilistic foundation of guidance for diffusion models. (DSyw Weakness 1)

  2. In Section 3.1, we clarify the scope of guidance caching and regimes in which it is valid. In Section 4.2, we add targeted validation of the effect of guidance caching on token values. (DSyw Weakness 2 and 4)

  3. In Appendix D.5, we include more experiments with expanded generation length on JudgeBench. (DSyw Weakness 3, d4Ze Weakness 2)

  4. We update Figure 2 to include more explanation of the method and add Section 3.4, a short summary section to more clearly explain the different components of our algorithm. (d4Ze Weakness 1)

  5. We clarify Table 4 to avoid confusion between different reward model scales. (d4Ze Question 1)

  6. In Appendix A, we add a discussion of block-autoregressive diffusion models such as SDAR and the challenges of reward guidance for these models. (d4Ze Weakness 2, CSdr Weakness 1)

  7. In Appendix D.4, we add experiments on longer outputs on the standard math benchmarks of GSM8K and MATH500. (DSyw Question 2, d4Ze Weakness 2, CSdr Weakness 2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.