acceptodds
Under review as a conference paper at ICLR 2027

Little Context, Large Guidance during Early Sampling in Masked Diffusion Language Models

Abstract

Masked diffusion language models apply classifier-free guidance (CFG) using a null prompt prediction as the reference, even during initial sampling when little response context is available. We show that this period forms a distinct guidance regime. The null prompt prediction changes rapidly, its alignment with the conditional prediction is weak, and the relative CFG residual strength is elevated. An algebraic decomposition and hidden representation analysis relate this elevated strength to differences between the two branches. We evaluate two adaptive controls for this regime. Adaptive on/off removes the early residual, whereas AdaptiveCap retains its direction while limiting its strength using a prompt specific reference. Both return to standard CFG once branch similarity reaches a threshold. Across two models, three benchmarks, and five guidance scales, AdaptiveCap consistently improves or matches Full CFG, with gains of up to percentage points. Comparisons with image generators show that elevated residual strength alone does not predict gains from early control. Our results indicate that when strong residuals coincide with weak branch alignment, retaining early guidance at a moderated strength is more effective than applying it unchanged or removing it entirely.

Primary area:
generative models
Keywords:
Masked diffusion language models, Classifier-free guidance, Early-stage sampling, Guidance rescaling, Inference-time calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_buSTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces two new methods to adapt the strength of classifier free guidance (CFG) in masked diffusion models. In Adaptive on/of, the CFG residual is not used in the initial decoding steps, only being applied once the null and conditional logits are sufficiently similar. In AdaptiveCap, instead of completely omitting the CFG residual, the guidance strength is regulated by a prompt specific relative residual strength. Experiments with LLaDA and Dream on three different datasets show both methods yield improvements over standard CFG.

Critical strengths
  • A strength of the proposed methods is their practical applicability: they can be readily incorporated into existing masked diffusion language models, providing an accessible way to explore adaptive guidance during decoding.
  • The paper is easy to follow in most parts.
Critical weaknesses
  • Adaptive on/off and AdaptiveCap appear to be adaptations of the ideas in [1] from continuous diffusion to masked diffusion. Their practical utility is welcome, but the methodological development and insights presented seem limited beyond demonstrating that these ideas also hold in the masked diffusion setting.
  • Some of the design choices and hyperparameter settings are not well motivated or explained. See questions 1 to 3 below.
  • The experimental results are not entirely convincing. While they do demonstrate improvements over the baseline in most cases, the gains are marginal in a few cases, especially for MMLU and MATH-500 for low guidance strength ww. Moreover, LLaDA results include a single seed.

Minor issues

  • The experiments on image generation read almost as an after thought. This is not critical, but the paper would read better if these results were either better integrated into the text or dropped entirely.

References

[1] Kynkäänniemi, Tuomas, et al. "Applying guidance in a limited interval improves sample and distribution quality in diffusion models." Advances in Neural Information Processing Systems 37 (2024): 122458-122483.

Questions for authors
  1. Could the authors motivate the choice of cosine similarity? I understand it has been in previous work on continuous diffusion models, but there they compute similarities between noise predictions in image or latent dimension space. In masked diffusion models, we have distributions over the vocabulary space. In that case, wouldn’t it be more natural to use a measure between distributions, like the KL divergence?
  2. Could the authors elaborate further on how τ\tau is estimated in AdaptiveCap? The text indicates that τ\tau is estimated from 25 complete generations are used for each prompt, which seems quite computationally expensive? Is that so? Can’t this be amortized somehow?
  3. Similarly to the previous question, the text does not go into detail on how η\eta is chosen. Was there a validation dataset to select good values for η\eta?
  4. The results indicate the CFG residual in the initial steps is not always harmful, which is somewhat counterintuitive. Do the authors have any hypothesis as to why that is the case?
5 comments
AuthorsOpenReview ↗

Response to Reviewer buST (1/2)

Thank you for recognizing the practical value and overall clarity of the paper. We would like to clarify several points about its contribution, design choices, and experimental results.

(W1, minor comment). Relationship to prior work and the role of the image experiments

Our paper is not a direct application of Kynkäänniemi et al.’s method to masked diffusion. As noted in the Introduction and Background, Kynkäänniemi et al.’s guidance interval inspired Adaptive on/off, which serves as a comparison method. They select a fixed noise-level interval for each model and dataset using downstream metrics such as FID. We propose using branch similarity to determine the control period on each generation trajectory. We fix η = 0.87 from the observed similarity dynamics without tuning on validation performance. This shared threshold gives both controls a prompt-specific stopping point, even within the same dataset.

Our main contribution is the analysis of early CFG and the AdaptiveCap method it motivates. In the evaluated language models, early sampling combines weak alignment between the conditional and null-prompt branches with elevated relative residual strength. Equation 27 decomposes this strength into three components, and Equation 33 links the vocabulary-mean component to differences in post-normalization hidden representations. Figure 8 shows weak early alignment in the final block outputs before normalization. AdaptiveCap preserves the residual direction and caps its strength using a prompt-specific reference. Its generally higher scores than Adaptive on/off in Table 1 suggest that suppressing the early residual entirely can discard useful guidance. Reviewer hAD8 highlighted the value of the analysis for understanding CFG behavior, and Reviewer oeCp highlighted how the analysis informed practical control methods.

The image experiments clarify the scope of this finding. Stable Diffusion maintains high branch similarity, and aMUSEd starts with both high relative residual strength and high similarity. Both exhibit stronger early alignment than the evaluated language models. Early control in aMUSEd also yields inconsistent joint improvements in FID and CLIP. These comparisons show why we consider branch alignment and residual strength together when characterizing early CFG across models.

(W2, Q1). Why use centered cosine similarity?

Following Castillo et al. (2025), we use cosine similarity to compare conditional and unconditional predictions. We apply it to vocabulary logits, where the CFG residual is defined. Centering removes uniform logit shifts, so similarity reflects the alignment of relative token scores.

KL divergence is a valid way to compare probability distributions, but it answers a different question. Two branches can rank tokens similarly while assigning them different confidence levels, in which case KL divergence can still be large. We use centered cosine similarity to measure alignment in relative token scores and measure residual strength separately.

(W2, Q2, Q3). How are τ\tau and η\eta chosen?

The pilot uses 25 sampling steps, not 25 complete generations. For each prompt, AdaptiveCap runs a 25-step Full CFG pilot and averages the observed relative residual strengths to estimate τ\tau. It then restores the random state and starts the main generation from the original masked response. The pilot adds approximately 9.8% to the 256-step generation budget.

Neither the pilot length nor η\eta was selected using task labels or downstream scores. Our appendix examines pilot lengths from 5 to 40 steps and shows that the estimated reference changes more gradually beyond 25 steps, motivating K=25K=25. We set η=0.87\eta=0.87 from the transition in branch-similarity dynamics and use it across both models, all three datasets, all guidance scales, and all seeds. Table 3 shows that scores vary by at most 0.8 percentage points across thresholds from 0.83 to 0.91.

Thus, τ\tau captures each prompt’s early residual strength, while η\eta provides a shared criterion for ending early control. Reviewer hAD8 also noted the practical value of AdaptiveCap’s short pilot and training-free design.

AuthorsOpenReview ↗

Response to Reviewer buST (2/2)

W3. Size of the gains and the single LLaDA run

Some gains on generation-based MMLU are small. One possible reason is that MMLU scores only the final multiple-choice answer, even when a rationale is generated. Changes in the generation path may therefore be less visible in this metric than in code generation or mathematical reasoning.

On MATH-500, AdaptiveCap improves DREAM-7B over Full CFG by 0.2 percentage points at w=0.3w=0.3 and by 5.6, 5.2, and 7.1 points at w=0.5w=0.5, 0.70.7, and 1.01.0, respectively. For LLaDA-1.5, it improves Full CFG at every tested scale, including gains of 4.4 points at w=0.1w=0.1 and 5.0 points at w=0.3w=0.3.

The single LLaDA run follows the configuration stated in the Experimental Setup. It uses temperature-zero greedy decoding and a fixed unmasking schedule, making decoding deterministic under this configuration. DREAM uses stochastic decoding, so we report the mean and standard deviation over five sampling seeds.

Q4. Why might the early residual still be useful?

Our hypothesis is that the information available during early control changes as response tokens are revealed. Initially, the response is fully masked and the null-prompt branch has little response context. Once a few tokens are revealed, both branches can use them to predict the remaining positions, while the conditional branch can also use the prompt.

For example, a revealed def gives the model a clue about the structure of a code response. The conditional branch can combine this clue with the task prompt to favor a suitable function name and implementation. The residual may therefore become useful before branch similarity reaches the threshold. The MBPP 401 and MATH 197 examples in the appendix show successful outputs with residual capping, although final responses alone cannot identify which sampling steps caused the result.

Adaptive on/off removes the residual throughout this period. AdaptiveCap retains it while limiting unusually high relative strength. This provides a possible explanation for why early guidance can be useful and why reducing its strength can outperform removing it.

Summary

Reviewer buST asked what our work adds beyond interval guidance in continuous diffusion and noted that the image experiments “read almost as an after thought.” We agree that we could make the role of the image experiments clearer. The comparison helps show how the early CFG behavior we observed differs from what has been reported in continuous diffusion. In the masked diffusion language models we studied, branch alignment was weak and relative CFG residual strength was high early in sampling, when little response context was available. Both image models showed stronger early alignment. We mathematically decompose the residual strength to identify what contributes to its high early values. This analysis motivates AdaptiveCap, which keeps early guidance while limiting overly strong corrections. In the revised manuscript, we moved Figure 5 away from the sensitivity table so readers can more easily see how the image comparison helps define the scope of our findings.

The review interpreted the pilot as 25 complete generations per prompt. We clarified that AdaptiveCap uses a single pilot of 25 sampling steps per prompt to estimate its reference strength, adding about 9.8% to the 256-step generation budget. We chose the shared threshold from the observed changes in branch similarity during sampling, without tuning it on downstream task scores. We also explained why we use centered cosine similarity to measure how closely the two branches agree on relative token scores.

We reported MATH-500 gains across several guidance scales and explained that we report one LLaDA run because decoding is deterministic under our evaluation settings. We also offered a hypothesis for why early guidance can help. Once a few response tokens are revealed, the conditional branch can use them together with the prompt to predict the remaining tokens. The CFG residual may therefore become useful before branch similarity reaches the threshold.

We highlighted the revised text in blue so you can easily find the changes. We appreciate your feedback and hope you will consider the revised manuscript in your final assessment.

I thank the authors for their response to my comments and questions. In particular, I appreciate the clarifications about how the hyperparameters were selected, and the use of 25 sampling steps rather than complete generations. That said, I remain unconvinced in a few respects.

  1. The novelty of the paper is still not clear to me. Previous work had already studied the effect of CFG at different points of the decoding process, and while Kynkäänniemi et al. proposed restricting guidance to an interval, Rojas et al. had already explicitly considered reducing early guidance through their ramp-up schedule. Thus, reducing rather than disabling early guidance is not itself novel.
  2. It is still not clear why cosine similarity is the right measure here. In the rebuttal the authors emphasize the difference between token-score alignment and confidence, but why is alignment the relevant criterion, if differences in confidence affect sampling and unmasking order?
  3. On a similar note, I also share reviewer hAd8’s concerns regarding measuring guidance strength on raw logits: these norms vary with vocabulary-wide shifts that leave the final branch distributions and standard CFG unchanged. What justifies making the controller depend on those shifts? The argument in Appendix J is partly circular: the mean component affects generation because the controller is designed to depend on it, but that does not justify why it is a good idea to design the controller this way. The raw-versus-centered ablation provides empirical support, but does not resolve this conceptual concern.
  4. The experimental results are still not very convincing. While the addition of the methods introduced by Rojas et al. is welcome, the gains over the baselines are generally small or mixed, with a clear advantage only on Dream/Math-500.
Reviewer_oeCpOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the CFG control mechanism in diffusion language models. It focuses on early-stage CFG behavior, studying the impact of CFG on generation stability and the differences between inference with and without CFG at different sampling stages. The results show that early CFG guidance can be excessively strong, and moderating its strength can improve generation performance. Furthermore, the paper introduces two CFG control mechanisms, Adaptive on/off and AdaptiveCap, to improve model generation performance.

Critical strengths
  1. The paper provides a detailed analysis of the mechanisms underlying early-stage CFG behavior in diffusion language models, supported by well-designed experiments in both the main text and appendices, including parameter sweeps and other analyses. Overall, the study is rigorous and solid.

  2. The paper goes beyond analyzing the mechanisms and proposes two practical CFG control strategies, Adaptive on/off and AdaptiveCap, achieving performance improvements across different benchmarks. This makes the work practically valuable.

Critical weaknesses

The paper analyzes early-stage generation behavior by introducing branch similarity and residual strength. However, these observations do not establish that low branch similarity or high residual strength causes degradation in generation performance. This weakens the persuasiveness of the paper's analysis of early-stage CFG behavior.

Questions for authors
  1. In Figure 3, R_t increases noticeably during the middle and later sampling stages on MBPP, while it decreases and gradually levels off on MATH-500. How do the authors explain the possible reasons for this difference?

  2. Table 2 only reports a narrow range of similarity thresholds, and the results show almost no differences. Have the authors evaluated a wider range of thresholds when selecting this key parameter?

3 comments
AuthorsOpenReview ↗

Response to Reviewer oeCp

Thank you for recognizing the detailed analysis of early CFG dynamics and the practical value of Adaptive on/off and AdaptiveCap. We appreciate your constructive feedback and address your questions below.

W1. Relationship between early CFG dynamics and performance

Low branch similarity and high residual strength alone do not establish the cause of generation errors. We compare Full CFG with Adaptive on/off, which removes the early residual, and AdaptiveCap, which limits its strength. We also compare Ramp-Up and fixed scaling under the same similarity threshold in our response to hAD8 (W1, Q1, Q3). The results, particularly on DREAM-7B MATH-500, show that how the early residual is controlled affects final performance. Our claim is that reducing its elevated strength can improve performance.

Q1. Different trends on MBPP and MATH-500 in Figure 3

Figure 3 shows the first 80 of 256 steps to make the early dynamics visible. MBPP begins to rise again within this window, whereas MATH-500 appears to level off. We examined the full sampling trajectory and found that the MATH-500 decomposition also rises later, around step 200. The extended decomposition indicates that vocabulary mean differences contribute strongly to the later increase in both tasks, suggesting a difference in timing.

Q2. Sensitivity to the similarity threshold

We extended the threshold sweep to include η = 0.81 and 0.93 at w = 0.3 and added the results to Table 3 in the revised manuscript. Across all six model and task combinations, scores vary by at most 0.8 percentage points for thresholds from η = 0.83 to 0.91. Larger changes occur at the endpoints in some settings. For example, DREAM-7B MATH-500 scores 43.4% at η = 0.87, compared with 41.8% at η = 0.81 and 40.2% at η = 0.93. These results show stable performance near the default, which we selected from branch similarity dynamics without downstream performance tuning.

Summary of revisions

Thank you for your constructive feedback. The additional schedule comparisons provide evidence that the choice of early residual control affects performance, and Table 3 now reports the extended threshold sweep. Together, these results clarify the empirical support for AdaptiveCap and the sensitivity of its similarity threshold.

I appreciate the author's feedback. When masked diffusion models first emerged, they quite naturally—and perhaps without much deliberation—adopted the classifier-free guidance (CFG) sampling approach from traditional continuous diffusion models. While the role of CFG in the dynamics of various sampling stages has been thoroughly explored for traditional diffusion models, relatively little is known about how CFG influences masked diffusion models at different stages. This paper investigates this issue and arrives at conclusions that differ significantly from those established for traditional diffusion models. In the context of traditional diffusion models, it is generally believed that applying CFG during the early stages of sampling is unhelpful or even detrimental; however, this paper demonstrates that such a view does not hold true for masked diffusion models. I believe this work contributes to a deeper understanding of masked diffusion models, and therefore, I maintain my positive assessment.

We sincerely thank you for your positive assessment and thoughtful comments. We appreciate your recognition of our contribution to understanding early-stage CFG in masked diffusion models.

Reviewer_hAD8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates classifier-free guidance (CFG) in masked diffusion language models, focusing on the behavioral divergence between the conditional branch and the null-prompt branch during the early stages of sampling. The authors observe that, in the initial phase of generation, the logits from these two branches exhibit low similarity alongside a relatively high guidance residual intensity. Motivated by this observation, the paper proposes two adaptive control strategies: "Adaptive on/off," which disables CFG until the branch similarity reaches a predefined threshold, and "AdaptiveCap," which dynamically limits the relative intensity of the guidance residual based on pre-sampling results for each individual prompt. The methods are evaluated on LLaDA-1.5 and Dream-7B using the MBPP, MATH-500, and MMLU benchmarks. The results show that, at an equivalent guidance scale, AdaptiveCap matches or outperforms standard CFG, with a maximum observed accuracy improvement of 7.1 percentage points. The paper also explores the relationship between branch similarity and the efficacy of early-stage guidance control through comparative experiments in the context of image diffusion models.

Critical strengths
  1. This paper analyzes the early sampling behavior of CFG in masked diffusion language models by considering branch similarity, residual strength, and cross-step stability, and validates the results on different models and tasks. These results help clarify how CFG actually behaves in these models.
  2. AdaptiveCap requires no additional training and introduces low presampling overhead, consistently outperforming or matching standard CFG at the same guidance scale. The method is simple and could be easy to adopt in practice.
Critical weaknesses
  1. Rojas et al. (2026) found that overly strong early guidance can impair generation quality, while gradually increasing guidance yields better results. This paper reaches a similar conclusion, but without comparison to these simpler non-adaptive strategies, making it difficult to determine whether the adaptive design of AdaptiveCap truly provides additional benefits.

  2. AdaptiveCap adjusts guidance based on the relative residual strength of the original logits, but this metric is affected by logit shifts that do not change the softmax probability. This means that even with the same sampling distribution, the method may produce different scaling factors, thus the rationale for this control rule is unclear.

Questions for authors
  1. Compared to Rojas et al.'s (2026) Ramp-Up and Right-Interval schedules, as well as fixed scaling strategies, does AdaptiveCap still offer a significant advantage under the same hyperparameter tuning budget and selection protocol?
  2. Since vocabulary-wide logit shift does not change the sampling distribution, why does AdaptiveCap use raw-logit residual strength, which is sensitive to this shift, to control guidance? What is the rationale behind this design?
  3. Would similar performance improvements be achieved by using fixed or random early scaling coefficients instead of estimating prompt-specific tau through pilot runs?
  4. In LLaDA block decoding, do S_t and R_t contain future blocks? Do low-similarity early regimes reappear at the start of each new block?
  5. Could you report the distribution of tηt_\eta across different models and tasks, and the proportion of samples that never reached the similarity threshold and were therefore capped throughout the generation process?
  6. Table 1 and Table 2 show numerical differences under the same w=0.3,η=0.87w=0.3, \eta=0.87 settings. Did both tables use the same seeds and evaluation protocol?
7 comments
AuthorsOpenReview ↗

Response to Reviewer hAD8 (1/2)

Thank you for recognizing the value of our analysis of early CFG dynamics and for your constructive feedback. Below, we explain our design choices and address the concerns you raised.

(W1, Q1, Q3) Comparison with other guidance schedules

We used the same branch-similarity threshold η=0.87\eta=0.87 for all early CFG controls. This threshold was selected from our diagnostics and sets a prompt-specific control period without tuning a timing parameter rr on downstream scores. Right-Interval becomes Adaptive on/off and activates guidance at the first threshold crossing on its generation trajectory. Ramp-Up uses a prompt-specific warm-up to find the first crossing and set the duration of the linear ramp. For the fixed scaling method you suggested, we set the multiplier to 0.50.5 and apply it until the threshold is reached. AdaptiveCap uses the same threshold to end capping and estimates its prompt-specific reference τ\tau from a 25-step Full CFG pilot.

The table reports scores (%) on the same evaluation items with seed 258.

DREAM-7B

Methodw=0.1w=0.1w=0.3w=0.3w=0.5w=0.5w=0.7w=0.7w=1.0w=1.0
MBPP
Ramp-Up65.467.765.065.063.4
Fixed scaling65.467.765.465.863.0
Adaptive on/off66.567.365.065.461.5
AdaptiveCap68.167.765.466.264.6
MATH-500
Ramp-Up41.642.041.839.236.2
Fixed scaling40.840.840.639.035.6
Adaptive on/off42.042.440.639.035.8
AdaptiveCap42.443.446.042.642.0
MMLU
Ramp-Up70.871.270.068.269.8
Fixed scaling70.471.270.270.669.6
Adaptive on/off70.471.271.070.069.6
AdaptiveCap71.271.871.869.470.0

LLaDA-1.5

Methodw=0.1w=0.1w=0.3w=0.3w=0.5w=0.5w=0.7w=0.7w=1.0w=1.0
MBPP
Ramp-Up52.551.451.449.447.0
Fixed scaling52.551.849.449.044.0
Adaptive on/off51.049.447.549.445.1
AdaptiveCap52.151.851.051.046.7
MATH-500
Ramp-Up38.037.437.036.434.4
Fixed scaling38.037.037.236.436.2
Adaptive on/off34.033.033.432.031.8
AdaptiveCap38.837.837.237.036.8
MMLU
Ramp-Up65.865.064.662.060.4
Fixed scaling66.464.263.461.257.8
Adaptive on/off67.665.463.461.861.2
AdaptiveCap66.664.663.662.061.4

Ramp-Up, fixed scaling, and AdaptiveCap often improved on both Full CFG and complete early suppression. AdaptiveCap’s advantage was clearest on DREAM-7B MATH-500, where Full CFG performed best at a relatively larger guidance scale. When stronger guidance is useful for a task, AdaptiveCap can preserve that benefit by reducing only early corrections that exceed the prompt-specific reference. On DREAM-7B MATH-500, it led at every tested scale and exceeded the strongest alternative by 4.2 and 5.8 percentage points at w=0.5w=0.5 and w=1.0w=1.0, respectively. Simpler controls sometimes achieved similar scores where Full CFG already performed best with weak guidance.

We added this comparison to Section 5.3, Strategies for Controlling Early CFG, in the revised manuscript.

(W2, Q2). Why use raw residual strength?

AdaptiveCap uses raw RtR_t so that vocabulary-mean differences between the conditional and null branches inform how much to reduce early guidance. This component accounts for a substantial share of DREAM’s early residual strength (Fig. 7) and reflects differences in the branches’ post-normalization hidden states through their shared output projection (Eq. 33). A uniform logit shift leaves softmax probabilities unchanged. Its effect on guided token scores is indirect, through the multiplier that AdaptiveCap applies to the full CFG residual.

Centering removes the vocabulary-mean component and changes the conditional logit norm used to normalize the residual. At w=0.1w=0.1, we compute both the generation-time strength RtR_t and the 25-step pilot reference τ\tau under each definition, keeping all other settings fixed.

ModelDatasetRaw strength (%)Centered strength (%)
DREAM-7BMBPP68.165.4
MATH-50042.441.2
MMLU71.270.2
LLaDA-1.5MBPP52.151.8
MATH-50038.837.8
MMLU66.666.2

Raw strength yields higher scores in all six settings, supporting its use to set the amount of early guidance reduction at w=0.1w=0.1. We added the rationale and ablation to Section 5.4, Raw Logits for Early Guidance Control, in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer hAD8 (2/2)

Q4. LLaDA block decoding

At each step, both branches predict logits for the full response. We measure branch similarity and relative residual strength over all still-masked response positions, including future blocks. This matches the measurement scope in DREAM and defines a single control period per response. Unmasking is restricted to the active block.

The first 80 steps include block boundaries at steps 32 and 64. At both boundaries, the response-wide measurements show no recurrence of the initial low-similarity, high-strength pattern.

Q5. When does early control end?

From the AdaptiveCap generation logs at w=0.3w=0.3, η=0.87\eta=0.87, we record the first step tηt_\eta at which branch similarity reaches η\eta. We compute quartiles over samples that reach the threshold. The last column reports the fraction of all samples that never reach it.

ModelDatasettηt_\eta (25th / 50th / 75th percentile)Did not reach η\eta
DREAM-7BMBPP2 / 2 / 22/257 (0.8%)
DREAM-7BMATH-5004 / 5 / 80/500 (0%)
DREAM-7BMMLU8 / 17 / 31109/500 (21.8%)
LLaDA-1.5MBPP8 / 13 / 258/257 (3.1%)
LLaDA-1.5MATH-5006 / 9 / 121/500 (0.2%)
LLaDA-1.5MMLU16 / 28 / 4976/500 (15.2%)

Across the five guidance scales, the fraction that never reaches the threshold is highest on MMLU, ranging from 15.0% to 22.4% for DREAM and from 6.8% to 21.8% for LLaDA. The corresponding fractions are at most 3.1% on MBPP and 0.2% on MATH-500.

Q6. Comparing Tables 1 and 3

At w=0.3w=0.3 and η=0.87\eta=0.87, Table 1 reports DREAM results as the mean and standard deviation across five sampling seeds. Table 3 reports a similarity-threshold sweep using a single seed. For example, AdaptiveCap scores 71.6±0.471.6 \pm 0.4 on DREAM MMLU in Table 1 and 71.871.8 in Table 3. The deterministic LLaDA scores agree across both tables at this setting.

Summary

We appreciate your constructive feedback. To address W1 and W2, we added a comparison of guidance schedules in Section 5.3 and a raw-versus-centered strength ablation in Section 5.4. With a shared similarity threshold chosen without downstream tuning, AdaptiveCap improves on Adaptive on/off, Ramp-Up, and fixed scaling in several settings, with the clearest gains on MATH-500 with DREAM-7B. Raw residual strength yields higher scores than centered strength in all six model and task combinations at w=0.1w=0.1.

In this response, we also clarified the scope of the LLaDA measurements and reported when early control ends across tasks, including how often generation ends before the similarity threshold is reached. We highlighted the revised text in blue so you can easily find the changes.

Thanks for the additional comparisons and the clarifications on Q4–Q6. The explanations on block decoding and when early control ends are clear to me now.

But I'm not fully convinced by the new results. The simpler controls use the same similarity threshold, and in most settings they get scores close to AdaptiveCap. The gain that really stands out is on DREAM MATH-500, so I think the evidence for the prompt-specific cap still mostly comes from this one setting. I also still have some doubt about measuring guidance strength on raw logits, since it is not clear to me how much of the high early strength reflects a real difference in what the model samples.

I'm also still not sure why changing the guidance in only the first few steps can give several points on DREAM MATH-500, but very little on the other tasks. It would help to have some explanation for this, and a comparison with fixed-step schedules under the same tuning budget.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion and Revisions

Dear Area Chair and Reviewers, we appreciate the time and effort the reviewers devoted to providing careful feedback. The discussion clarified how our work differs from existing guidance schedules, and the additional experiments strengthened, rather than changed, the core claims of the paper. We summarize the main points below.

Contribution and difference from existing guidance schedules

Our main contribution is new knowledge about how early null-prompt CFG behaves in masked diffusion language models, analyzed in terms of branch similarity and guidance strength. Masked diffusion models adopted CFG from continuous diffusion, where early guidance is commonly considered unhelpful or even harmful, and early-stage CFG has so far been studied mainly in image generation.

Across two models and three tasks, we identify an early regime with little response context, low branch alignment, and high relative residual strength. The residual in this regime still carries useful guidance, as AdaptiveCap outperformed Adaptive on/off, which removes it, in 24 of 30 settings and tied in one. Comparisons with image models further show that this behavior differs from early CFG in continuous and masked image diffusion.

Reviewers recognized the value of analyzing how CFG behaves in masked diffusion language models, while differing on whether this analysis and the resulting control provide sufficient novelty beyond existing guidance schedules.

Reviewers hAD8 and buST raised concerns about novelty relative to Rojas et al. (ICLR 2026) and about the margin over simpler controls. Based on low-dimensional theory and image generation experiments, Rojas et al. favor weaker guidance early and stronger guidance later, and they evaluated Right-Interval and Ramp-Up only on image generation. Their LLaDA experiment evaluated guidance normalization. Both schedules therefore set how much to reduce guidance from the sampling step and the end step alone, without considering the actual size of the residual.

Our observation is different. In masked diffusion language models, the relative correction applied in logit space is particularly large in the early steps even at the same guidance scale ww, and this early residual still carries useful guidance. The issue is therefore not the early timing itself, but an excessively large correction while the two branches are still weakly aligned. AdaptiveCap accordingly reduces the residual only when its observed strength exceeds a prompt-specific reference and keeps its direction, rather than reducing guidance by time. This control applies until the null branch aligns with the conditional branch.

At the reviewers' request, we compared AdaptiveCap with Right-Interval, Ramp-Up, and fixed scaling, using both prompt-specific end steps and a shared fixed end step. AdaptiveCap scored higher than Ramp-Up in 23 of 30 settings and tied in two with prompt-specific end steps. With the fixed end step, it scored higher in five of six settings and tied in one. Simpler controls remained competitive in some settings, such as fixed scaling on LLaDA MBPP. Because the benefit holds under a shared end step, it is not tied to the similarity-based stopping rule.

Against Full CFG, AdaptiveCap scored higher in 29 of 30 settings and tied in one, with the largest gains on MATH-500. At w=1.0w=1.0, DREAM-7B improved from 35.2 to 42.3 and LLaDA-1.5 from 31.4 to 36.8. These results further clarify that early-stage analyses and controls developed for image diffusion do not simply carry over to masked diffusion language models.

Raw-logit strength signal

Reviewers hAD8 and buST asked why the strength signal uses raw logits, given that a uniform logit shift leaves token probabilities unchanged. Because the CFG residual is added and scaled in raw-logit space, we measured strength directly in the same space.

Contrary to our initial expectation, raw and centered definitions computed at the same states gave average reduction factors that differed by at most 0.08 across six settings, with no consistent direction, while raw strength capped fewer steps in every setting.

We therefore removed Appendix J, which we had added during the rebuttal period to interpret the mean component, and present raw strength as an empirical choice supported by a six-setting downstream ablation. This concerns one implementation choice within AdaptiveCap and does not change the main contribution of the paper.

We thank the reviewers again. The requested comparisons and analyses helped us define the contribution more precisely and correct our earlier interpretation of the raw control signal. We will continue to respond carefully to any remaining questions for the rest of the discussion period.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.