- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces two new methods to adapt the strength of classifier free guidance (CFG) in masked diffusion models. In Adaptive on/of, the CFG residual is not used in the initial decoding steps, only being applied once the null and conditional logits are sufficiently similar. In AdaptiveCap, instead of completely omitting the CFG residual, the guidance strength is regulated by a prompt specific relative residual strength. Experiments with LLaDA and Dream on three different datasets show both methods yield improvements over standard CFG.
Critical strengths
- A strength of the proposed methods is their practical applicability: they can be readily incorporated into existing masked diffusion language models, providing an accessible way to explore adaptive guidance during decoding.
- The paper is easy to follow in most parts.
Critical weaknesses
- Adaptive on/off and AdaptiveCap appear to be adaptations of the ideas in [1] from continuous diffusion to masked diffusion. Their practical utility is welcome, but the methodological development and insights presented seem limited beyond demonstrating that these ideas also hold in the masked diffusion setting.
- Some of the design choices and hyperparameter settings are not well motivated or explained. See questions 1 to 3 below.
- The experimental results are not entirely convincing. While they do demonstrate improvements over the baseline in most cases, the gains are marginal in a few cases, especially for MMLU and MATH-500 for low guidance strength . Moreover, LLaDA results include a single seed.
Minor issues
- The experiments on image generation read almost as an after thought. This is not critical, but the paper would read better if these results were either better integrated into the text or dropped entirely.
References
[1] Kynkäänniemi, Tuomas, et al. "Applying guidance in a limited interval improves sample and distribution quality in diffusion models." Advances in Neural Information Processing Systems 37 (2024): 122458-122483.
Questions for authors
- Could the authors motivate the choice of cosine similarity? I understand it has been in previous work on continuous diffusion models, but there they compute similarities between noise predictions in image or latent dimension space. In masked diffusion models, we have distributions over the vocabulary space. In that case, wouldn’t it be more natural to use a measure between distributions, like the KL divergence?
- Could the authors elaborate further on how is estimated in AdaptiveCap? The text indicates that is estimated from 25 complete generations are used for each prompt, which seems quite computationally expensive? Is that so? Can’t this be amortized somehow?
- Similarly to the previous question, the text does not go into detail on how is chosen. Was there a validation dataset to select good values for ?
- The results indicate the CFG residual in the initial steps is not always harmful, which is somewhat counterintuitive. Do the authors have any hypothesis as to why that is the case?
5 comments
Response to Reviewer buST (2/2)
W3. Size of the gains and the single LLaDA run
Some gains on generation-based MMLU are small. One possible reason is that MMLU scores only the final multiple-choice answer, even when a rationale is generated. Changes in the generation path may therefore be less visible in this metric than in code generation or mathematical reasoning.
On MATH-500, AdaptiveCap improves DREAM-7B over Full CFG by 0.2 percentage points at and by 5.6, 5.2, and 7.1 points at , , and , respectively. For LLaDA-1.5, it improves Full CFG at every tested scale, including gains of 4.4 points at and 5.0 points at .
The single LLaDA run follows the configuration stated in the Experimental Setup. It uses temperature-zero greedy decoding and a fixed unmasking schedule, making decoding deterministic under this configuration. DREAM uses stochastic decoding, so we report the mean and standard deviation over five sampling seeds.
Q4. Why might the early residual still be useful?
Our hypothesis is that the information available during early control changes as response tokens are revealed. Initially, the response is fully masked and the null-prompt branch has little response context. Once a few tokens are revealed, both branches can use them to predict the remaining positions, while the conditional branch can also use the prompt.
For example, a revealed def gives the model a clue about the structure of a code response. The conditional branch can combine this clue with the task prompt to favor a suitable function name and implementation. The residual may therefore become useful before branch similarity reaches the threshold. The MBPP 401 and MATH 197 examples in the appendix show successful outputs with residual capping, although final responses alone cannot identify which sampling steps caused the result.
Adaptive on/off removes the residual throughout this period. AdaptiveCap retains it while limiting unusually high relative strength. This provides a possible explanation for why early guidance can be useful and why reducing its strength can outperform removing it.
Summary
Reviewer buST asked what our work adds beyond interval guidance in continuous diffusion and noted that the image experiments “read almost as an after thought.” We agree that we could make the role of the image experiments clearer. The comparison helps show how the early CFG behavior we observed differs from what has been reported in continuous diffusion. In the masked diffusion language models we studied, branch alignment was weak and relative CFG residual strength was high early in sampling, when little response context was available. Both image models showed stronger early alignment. We mathematically decompose the residual strength to identify what contributes to its high early values. This analysis motivates AdaptiveCap, which keeps early guidance while limiting overly strong corrections. In the revised manuscript, we moved Figure 5 away from the sensitivity table so readers can more easily see how the image comparison helps define the scope of our findings.
The review interpreted the pilot as 25 complete generations per prompt. We clarified that AdaptiveCap uses a single pilot of 25 sampling steps per prompt to estimate its reference strength, adding about 9.8% to the 256-step generation budget. We chose the shared threshold from the observed changes in branch similarity during sampling, without tuning it on downstream task scores. We also explained why we use centered cosine similarity to measure how closely the two branches agree on relative token scores.
We reported MATH-500 gains across several guidance scales and explained that we report one LLaDA run because decoding is deterministic under our evaluation settings. We also offered a hypothesis for why early guidance can help. Once a few response tokens are revealed, the conditional branch can use them together with the prompt to predict the remaining tokens. The CFG residual may therefore become useful before branch similarity reaches the threshold.
We highlighted the revised text in blue so you can easily find the changes. We appreciate your feedback and hope you will consider the revised manuscript in your final assessment.
I thank the authors for their response to my comments and questions. In particular, I appreciate the clarifications about how the hyperparameters were selected, and the use of 25 sampling steps rather than complete generations. That said, I remain unconvinced in a few respects.
- The novelty of the paper is still not clear to me. Previous work had already studied the effect of CFG at different points of the decoding process, and while Kynkäänniemi et al. proposed restricting guidance to an interval, Rojas et al. had already explicitly considered reducing early guidance through their ramp-up schedule. Thus, reducing rather than disabling early guidance is not itself novel.
- It is still not clear why cosine similarity is the right measure here. In the rebuttal the authors emphasize the difference between token-score alignment and confidence, but why is alignment the relevant criterion, if differences in confidence affect sampling and unmasking order?
- On a similar note, I also share reviewer hAd8’s concerns regarding measuring guidance strength on raw logits: these norms vary with vocabulary-wide shifts that leave the final branch distributions and standard CFG unchanged. What justifies making the controller depend on those shifts? The argument in Appendix J is partly circular: the mean component affects generation because the controller is designed to depend on it, but that does not justify why it is a good idea to design the controller this way. The raw-versus-centered ablation provides empirical support, but does not resolve this conceptual concern.
- The experimental results are still not very convincing. While the addition of the methods introduced by Rojas et al. is welcome, the gains over the baselines are generally small or mixed, with a clear advantage only on Dream/Math-500.
Response to Reviewer buST (1/2)
Thank you for recognizing the practical value and overall clarity of the paper. We would like to clarify several points about its contribution, design choices, and experimental results.
(W1, minor comment). Relationship to prior work and the role of the image experiments
Our paper is not a direct application of Kynkäänniemi et al.’s method to masked diffusion. As noted in the Introduction and Background, Kynkäänniemi et al.’s guidance interval inspired Adaptive on/off, which serves as a comparison method. They select a fixed noise-level interval for each model and dataset using downstream metrics such as FID. We propose using branch similarity to determine the control period on each generation trajectory. We fix η = 0.87 from the observed similarity dynamics without tuning on validation performance. This shared threshold gives both controls a prompt-specific stopping point, even within the same dataset.
Our main contribution is the analysis of early CFG and the AdaptiveCap method it motivates. In the evaluated language models, early sampling combines weak alignment between the conditional and null-prompt branches with elevated relative residual strength. Equation 27 decomposes this strength into three components, and Equation 33 links the vocabulary-mean component to differences in post-normalization hidden representations. Figure 8 shows weak early alignment in the final block outputs before normalization. AdaptiveCap preserves the residual direction and caps its strength using a prompt-specific reference. Its generally higher scores than Adaptive on/off in Table 1 suggest that suppressing the early residual entirely can discard useful guidance. Reviewer hAD8 highlighted the value of the analysis for understanding CFG behavior, and Reviewer oeCp highlighted how the analysis informed practical control methods.
The image experiments clarify the scope of this finding. Stable Diffusion maintains high branch similarity, and aMUSEd starts with both high relative residual strength and high similarity. Both exhibit stronger early alignment than the evaluated language models. Early control in aMUSEd also yields inconsistent joint improvements in FID and CLIP. These comparisons show why we consider branch alignment and residual strength together when characterizing early CFG across models.
(W2, Q1). Why use centered cosine similarity?
Following Castillo et al. (2025), we use cosine similarity to compare conditional and unconditional predictions. We apply it to vocabulary logits, where the CFG residual is defined. Centering removes uniform logit shifts, so similarity reflects the alignment of relative token scores.
KL divergence is a valid way to compare probability distributions, but it answers a different question. Two branches can rank tokens similarly while assigning them different confidence levels, in which case KL divergence can still be large. We use centered cosine similarity to measure alignment in relative token scores and measure residual strength separately.
(W2, Q2, Q3). How are τ and η chosen?
The pilot uses 25 sampling steps, not 25 complete generations. For each prompt, AdaptiveCap runs a 25-step Full CFG pilot and averages the observed relative residual strengths to estimate τ. It then restores the random state and starts the main generation from the original masked response. The pilot adds approximately 9.8% to the 256-step generation budget.
Neither the pilot length nor η was selected using task labels or downstream scores. Our appendix examines pilot lengths from 5 to 40 steps and shows that the estimated reference changes more gradually beyond 25 steps, motivating K=25. We set η=0.87 from the transition in branch-similarity dynamics and use it across both models, all three datasets, all guidance scales, and all seeds. Table 3 shows that scores vary by at most 0.8 percentage points across thresholds from 0.83 to 0.91.
Thus, τ captures each prompt’s early residual strength, while η provides a shared criterion for ending early control. Reviewer hAD8 also noted the practical value of AdaptiveCap’s short pilot and training-free design.