acceptodds
Under review as a conference paper at ICLR 2027

Unmask the State: State-Adaptive Strategy Selection for Masked Diffusion Language Models

Abstract

Masked diffusion language models (MDMs) admit flexible generation orders, making the unmasking strategy an inference decision. Existing methods vary in how they prioritize positions, control parallelism, restrict selection regions, revise predictions, or plan future denoising, yet it remains unclear when these choices should change during generation. We study this question through strategy reversals, where an alternative action becomes preferable to a fixed choice. We organize MDM inference into five axes—score, cardinality, region, commitment, and planning—and define adaptation opportunity as the one-step utility advantage of the best candidate action over a validation-selected fixed action. This view shows that adaptation value depends on both the frequency and magnitude of such reversals. Across three MDMs and ten tasks, adaptation opportunities are highly heterogeneous, with some regimes exhibiting concentrated and predictable one-step gains. This motivates selective adaptation: lightweight detectors calibrated on validation prompts identify high-opportunity states, capturing, for example, 56.9% of the candidate-set oracle opportunity by adapting only the top 10% of states on LLaDA-8B constrained JSON filling. Our transition-level results suggest that state adaptation is most useful when applied selectively rather than uniformly.

Primary area:
generative models
Keywords:
Masked Diffusion Language Models, Unmasking Strategies, State-Adaptive Decoding, Training-Free Inference, Generation Order, Parallel Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fEapOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper examines how MDMs can improve from better decoding strategies, based on the current state. The authors introduce a framework to map decoding decisions into five axes. The contributions are the framework for measuring opportunities to improve decoding decisions, using an oracle at the final state. The authors also introduce an adaptation procedure that specifies when to apply an alternative action and which one.

The evaluation is done on three models and ten tasks, with interventions and then the normal continuation. Their stated finding is that opportunities and the predictability of opportunities varies significantly. The experiments show mild utility improvements of 3-6 percentage points over a static policy.

Critical strengths

Their framework separates the predictability of an alternative decoding step from the increase in utility. This helps, as it narrows the headroom provided by the oracle and allows for benchmarking alternative decoding methods.

Critical weaknesses
  1. The reported numbers do not cleanly reconcile: for LLaDA-8B on constrained JSON filling, Table 10 reports an oracle gap of 0.0287, and Table 13 reports the fixed and oracle utility of 0.9078 and 0.9875, respectively, implying a gap of 0.0797, which is impossible by Equation 4. Note that I am having trouble reconciling the numbers in general, as the result tables in the appendix are 20 pages with formatting problems. Further, Table 10 reports the 100% coverage lift at 0.0594, which exceeds the Table 10 oracle gap. It is unclear to me how the policy improves beyond the oracle gap, which should not be possible. This number substantiates the headline result in the abstract and is the basis for the selective adaptation argument, thus reducing confidence in the contribution.
  2. The aggregated gains provide limited evidence for the added value of the proposed selective gating. Dream's gain is largely driven by one dataset; GSM8k. Selective decoding improves only 3/10, 4/10, 3/10 tasks across the three models. Further, the AUC of the state-level detector is 0.54-0.57, which is slightly better than guessing. This reduces my confidence in the proposed method's contribution.
  3. The headline decoding settings gives zero end-to-end utility: LaDA-8B Constrained JSON / region is the abstract’s headline (0.854 AUROC, 56.9% capture). End-to-end, Fixed, Always and Selective all score 0.000, while the reference decoder scores 0.320.
Questions for authors
  1. Could the authors clarify the numerical consistencies stated above, and reconcile the headline numbers used in the table 5 and the abstract?
  2. Could the authors clarify whether the realized lift of 0.0594 at 100% coverage exceeds the oracle gap stated in Table 10?
  3. How much of the improvement of is attributable to gating rather than to the alternative action? Dream's GSM8k improvement contributes about 80% of the aggregate gain. Please also provide uncertainty estimates for these numbers to inspect significance.
  4. What are the validation set sizes and sources for each task? Please further clarify what role the validation data plays and also evaluate the held-out performance on completely unseen datasets.
5 comments
AuthorsOpenReview ↗

Official Rebuttal [1/5]

We thank the reviewer for checking the numerical results so closely. The inconsistent captions and difficult appendix layout made key comparisons appear invalid; we apologize for that presentation failure. We have traced the reported quantities to their evaluation definitions, audited the tables, and expanded the end-to-end tests to address the substantive concern about selective gating. We respond to each point below.

W1: The reported numbers do not cleanly reconcile: for LLaDA-8B on constrained JSON filling, Table 10 reports an oracle gap of 0.0287, and Table 13 reports the fixed and oracle utility of 0.9078 and 0.9875, respectively, implying a gap of 0.0797, which is impossible by Equation 4. Note that I am having trouble reconciling the numbers in general, as the result tables in the appendix are 20 pages with formatting problems. Further, Table 10 reports the 100% coverage lift at 0.0594, which exceeds the Table 10 oracle gap. It is unclear to me how the policy improves beyond the oracle gap, which should not be possible. This number substantiates the headline result in the abstract and is the basis for the selective adaptation argument, thus reducing confidence in the contribution.

A1: The apparent violation is caused by comparing different state sets and different fixed baselines. The table below reconciles the exact values for LLaDA-8B/Constrained JSON Fill/region; the full-coverage lift and the oracle gap are compatible when measured on the same 640 held-out states.

QuantityOriginal Table 10Tables 12–15 / main results
Evaluation states100 prompts × 8 = 800 (includes validation)80 held-out prompts × 8 = 640
Fixed actionBest fixed chosen in hindsight on 800 statesBest fixed chosen on 20 validation prompts
Oracle procedureTwo-fold rollout cross-fitted candidate-set oracleSame estimation procedure
Fixed utility0.94130.9078
Oracle utility0.97000.9875
Oracle − Fixed0.02870.0797 [0.0625, 0.0984]
Diagnostic at 100% coverageNot directly comparable0.9672 (lift +0.0594; 74.5% gap recovery)

Thus 0.9672 − 0.9078 = +0.0594 ≤ 0.9875 − 0.9078 = +0.0797 (74.5% recovery; oracle − diagnostic = 0.0203). The original 0.0287 Table 10 gap used 800 states and a hindsight-selected baseline, so it cannot be used as the upper bound for this held-out diagnostic lift. The same 640-state hindsight-fixed gap is 0.0125, making clear that validation-fixed opportunity includes calibration loss. We audited all 104 model–task–axis settings with held-out transition-level utilities (Tables 12–13; the remaining 16 settings, HumanEval for all three models and LLaDA-8B GSM8K, have no compatible held-out branch table) and found zero violations: the oracle utility is never below the Fixed, Stage, or Diagnostic utility. We have corrected two legacy Table 10 cells (GSM8K/region 0.0137 → 0.0050; HumanEval 0.0048 → 0.0482)and reorganized the appendix into separate, explicitly labeled tables (Appendix I).

AuthorsOpenReview ↗

Official Rebuttal [2/5]

W2: The aggregated gains provide limited evidence for the added value of the proposed selective gating. Dream's gain is largely driven by one dataset; GSM8k. Selective decoding improves only 3/10, 4/10, 3/10 tasks across the three models. Further, the AUC of the state-level detector is 0.54-0.57, which is slightly better than guessing. This reduces my confidence in the proposed method's contribution.

A2: The criticism of a universal improvement claim is justified. Mean AUROC 0.54–0.57 is modest, and the large Dream/GSM8K improvement at 100% coverage does not measure gating. We therefore focus on where the detector and controller yield an empirically supported benefit, including matched-coverage Random controls and disjoint final-test prompts. Low positive-opportunity prevalence can make performance difficult to estimate, but does not by itself force AUROC toward 0.5; we do not use it to dismiss the reported mean AUROC.

ModelFixedSelectiveSel − FixedTasks improvedTasks worsenedEx-GSM8K liftSel − Random macro
LLaDA-8B0.4780.523+0.0455/100/10+0.050+0.033
LLaDA-1.50.3170.391+0.0746/100/10+0.082+0.062
Dream-7B0.4280.497+0.0703/101/10+0.027+0.020

On the new final-test split, Selective improves 5/10, 6/10, and 3/10 tasks (the original submission reported 3/10, 4/10, and 3/10 on the preliminary evaluation). To separate timing from other effects, we group the 30 settings in Table 18 by coverage: nine have 0% coverage (Selective never deviates from Fixed); three have 100% coverage (LLaDA-8B/TD07 +0.055, Dream/GSM8K +0.450, Dream/Multi-Span Cloze 0.000), which reflect an alternative-rule effect rather than timing; and five have partial coverage but identical Fixed, Always, Selective, and Random utilities. Timing can therefore be evaluated in the remaining 13 settings. Excluding all three 100%-coverage settings, the macro Sel − Fixed gain is +0.044 (LLaDA-8B, 9 tasks), +0.074 (LLaDA-1.5, 10 tasks), and +0.031 (Dream, 8 tasks); the Ex-GSM8K column above still includes LLaDA-8B/TD07. We will add this categorization to Appendix I.6.

For a compact view of state-aware timing, here are the strongest and most uncertain cases, with paired 95% bootstrap CIs. These are all compared on the same prompt sets per setting.

ModelTaskAlways − Fixed (95% CI)Sel − Random (95% CI)Sel − Fixed (95% CI)
LLaDA-8BConstrained JSON Fill-0.043 [-0.075, -0.011]+0.046 [+0.033, +0.059]+0.053 [+0.025, +0.081]
LLaDA-8BUnique List Commit-0.205 [-0.233, -0.177]+0.185 [+0.154, +0.216]+0.215 [+0.184, +0.246]
LLaDA-1.5Carry RTL-0.035 [-0.064, -0.006]+0.106 [+0.063, +0.149]+0.130 [+0.100, +0.160]
LLaDA-1.5CSV Missing Cells-0.002 [-0.018, +0.014]+0.031 [-0.002, +0.064]+0.040 [+0.028, +0.051]
LLaDA-1.5JSON Mode Eval-0.039 [-0.244, +0.167]+0.184 [+0.043, +0.326]+0.222 [-0.006, +0.454]
DreamCarry RTL-0.100 [-0.116, -0.083]+0.136 [+0.120, +0.152]+0.169 [+0.152, +0.185]

In 12/30 settings, unconditional switching has a negative point estimate and Selective beats Fixed, Always, and Random. These 12 are all of the 13 timing-evaluable settings except Dream/Unique List, where Selective is −0.006 [−0.017, +0.005] relative to both Fixed and Random. The three selective contrasts have positive 95% intervals simultaneously in 10/12 settings; eight also have a negative Always − Fixed interval. The two exceptions have intervals crossing zero, and we do not claim universal significance or multiple-comparison adjustment. In LLaDA-8B/CJSON, the one-step detector AUROC is 0.854. In a distinct Dream Carry/cardinality test, AUROC is 0.802, with end-to-end Sel − Random +0.066 [0.050, 0.083]. These results support selective gains in specific regimes rather than generally strong detection.

AuthorsOpenReview ↗

Official Rebuttal [3/5]

W3: The headline decoding settings gives zero end-to-end utility: LaDA-8B Constrained JSON / region is the abstract’s headline (0.854 AUROC, 56.9% capture). End-to-end, Fixed, Always and Selective all score 0.000, while the reference decoder scores 0.320.

A3:The original 0.000 end-to-end CJSON result is a real failure, not a reporting typo. An audit showed that its preliminary prompts differed in template/constraint composition from calibration, causing the chosen region action to fail output extraction. Also, one-step utility need not translate to repeated decoding because the visited-state distribution changes. Below we report the original pilot and the corrected new, disjoint, matched-generator 500-prompt test separately; they are not a controlled before/after comparison.

LLaDA-8B / CJSON / regionnRefFixedStageAlwaysRandomSelectiveCoverage
Original mismatched pilotpreliminary0.3200.000—0.000—0.000—
New generator-matched final test5000.9200.9080.9120.8650.9150.9610.250

On the new split, Sel − Fixed = +0.053 [0.025, 0.081], Sel − Always = +0.096 [0.065, 0.127], and Sel − Random = +0.046 [0.033, 0.059] (paired 95% bootstrap). The original failure reveals sensitivity to distribution shift; the revised claim will be restricted to the matched-generator evaluation and will not infer end-to-end success from AUROC alone.

Q1: Could the authors clarify the numerical consistencies stated above, and reconcile the headline numbers used in the table 5 and the abstract?

A1:The abstract and Tables 5–6 and 12–15 use 640 held-out states with a 20-prompt validation-selected fixed action. Original Table 10 instead uses 800 states with a hindsight-selected fixed action. The exact headline quantities are:

SourceEvaluation definitionValue
Original Table 10 oracle gap800 states; hindsight fixed0.0287
Tables 12–13 oracle gap640 held-out; validation fixed0.0797 [0.0625, 0.0984]
Table 4 detector AUROC640 held-out; validation fixed0.854
Tables 4–5 capture at 10% coverage640 held-out; validation fixed56.9%
Tables 4–5 lift at 10% coverage640 held-out; validation fixed+0.0453
Tables 5 / 13 / 15 lift at 100% coverage640 held-out; validation fixed+0.0594

The related utilities are Fixed 0.9078, Oracle 0.9875, and full-coverage Diagnostic 0.9672. We will relabel the estimands in the abstract, main tables, and appendix so these figures are not treated as interchangeable.

Q2: Could the authors clarify whether the realized lift of 0.0594 at 100% coverage exceeds the oracle gap stated in Table 10?

A2: No. The applicable candidate-set oracle gap is 0.0797, not the original Table 10 value 0.0287, because the 100%-coverage diagnostic uses the same 640 held-out states and validation-selected fixed action as the former.

Quantity (640 held-out states)Value
Validation-selected Fixed0.9078
Candidate-set Oracle0.9875
Diagnostic at 100% coverage0.9672
Oracle − Fixed+0.0797 [0.0625, 0.0984]
Diagnostic − Fixed+0.0594
Oracle − Diagnostic+0.0203

Therefore +0.0594 ≤ +0.0797; no bound is violated under the consistent evaluation definition. The separate hindsight-fixed 800-state gap is not the relevant bound.

Reviewer_TWcQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focus on the issue that when a masked diffusion model should change its rule to fill masked positions. It can change which position to fill for the next step, how many positions it will fill in one time and if the filled positions can be changed later. This paper first conduct a simple test on whether changing the choice at a simple step can improve the final answer. Then it tries to identify those useful moments without trying each choice during the normal generation. Experiments on three different models and ten tasks indicate that changing the rule will be helpful in some cases while the useful moments are hard to predict. The paper also tests the method over complete generation runs

Critical strengths
  1. This paper distinguishes three useful questions: Is a useful chioce available? Will the method answer the correct moment when change is useful? Can the method choose a better choice in the end? This will be helpful to help us understand why finding possible gains isn’t enough for building a useful method.

  2. The authors give a lot of successful and unsuccessful cases. This will help the readers to understand where the method will be useful and where it will fail

Critical weaknesses
  1. For LLaDA-8B/Constrained JSON Fill/Region, the gap in Table 10 is 0.0287, while the corresponding oracle-minus-fixed gaps in Tables 12 and 13 are 0.0797. Please show us whether these analyses used different evaluation states or other different settings. Also please specify the range of the data for each table to help readers understand the relationships between them.

  2. In this paper, it isn’t clear how much improvement comes from choosing the moment to change the rules. According to Table 18, most of Dream's average gain is given by GSM8K. In this task, the method changes the rules at each step, so it receives the same score as the method that always changes. The result supports the new rules, but doesn’t indicate the benefit of choosing the right moment to use it.

Questions for authors
  1. Would you explain why Tables 10, 12, and 13 give different results for the same setting?

  2. Why this paper focuses on whether some better choice exists, rather than whether the method's own proposed choice will be helpful?

  3. Is the five-axis framework intended to cover all major unmasking decisions, or is it a practical classification of existing methods? Please clarify which decisions, if there exists any, fall outside its scope。

3 comments
AuthorsOpenReview ↗

Official Rebuttal [1/3]

We thank the reviewer for identifying the unclear oracle-gap definitions and, especially, the need to distinguish state-aware timing from simply choosing a better rule. These points led us to clarify the estimands and add the matched-coverage timing control.

**W1: For LLaDA-8B/Constrained JSON Fill/Region, the gap in Table 10 is 0.0287, while the corresponding oracle-minus-fixed gaps in Tables 12 and 13 are 0.0797. Please show us whether these analyses used different evaluation states or other different settings. Also please specify the range of the data for each table to help readers understand the relationships between them.

Q1: Would you explain why Tables 10, 12, and 13 give different results for the same setting?**

A1: The discrepancy is ours: the original caption omitted differences in the state pool and fixed-action selection rule. Both estimates use the same underlying rollout collection but not the same estimand. For LLaDA-8B/Constrained JSON Fill/region:

QuantityOriginal Table 10Tables 12–15 / main results
Evaluation states100 prompts × 8 = 800 (includes validation)80 held-out prompts × 8 = 640
Fixed actionBest fixed chosen in hindsight on 800 statesBest fixed chosen on 20 validation prompts
Oracle procedureTwo-fold rollout cross-fitted candidate-set oracleSame estimation procedure
Fixed utility0.94130.9078
Oracle utility0.97000.9875
Oracle − Fixed0.02870.0797 [0.0625, 0.0984]
Diagnostic at 100% coverageNot directly comparable0.9672 (lift +0.0594; 74.5% gap recovery)

On the same 640 held-out states, the hindsight-fixed gap is only 0.0125, while the validation-selected-fixed gap is 0.0797. Thus 0.0672 (84.3%) of the latter gap reflects the choice of fixed baseline, rather than pure state dependence. We will label these separately as hindsight state-dependence gap and held-out validation-fixed opportunity. We have corrected two imported Table 10 values: GSM8K/region 0.0137 → 0.0050 and HumanEval 0.0048 → 0.0482 (strongest axis: commitment), with split, state count, and selection rule stated for each results table.

AuthorsOpenReview ↗

Official Rebuttal [2/3]

W2: In this paper, it isn’t clear how much improvement comes from choosing the moment to change the rules. According to Table 18, most of Dream's average gain is given by GSM8K. In this task, the method changes the rules at each step, so it receives the same score as the method that always changes. The result supports the new rules, but doesn’t indicate the benefit of choosing the right moment to use it.

A2: You are right about Dream/GSM8K: its gate activates at 100% of steps, so Fixed 0.360 → Always = Selective = Random 0.810 is an alternative-rule gain (+0.450), not evidence of better timing. We added a three-seed Random gate that applies the same diagnostic action at randomly selected steps with the same per-prompt activation count as Selective. The contrasts have distinct interpretations: Always − Fixed (unconditional rule change), Selective − Always (selective withholding), and Selective − Random (state-aware placement with activation frequency controlled).

The table reports all 12/30 settings where Always is worse than Fixed at the point-estimate level and Selective exceeds Fixed, Always, and Random. Brackets are paired 95% bootstrap intervals.

ModelTaskSel. coverageAlways − Fixed (95% CI)Sel − Always (95% CI)Sel − Random (95% CI)Sel − Fixed (95% CI)
LLaDA-8BCSV Missing Cells0.300-0.006 [-0.019, +0.007]+0.060 [+0.044, +0.075]+0.041 [+0.024, +0.058]+0.054 [+0.038, +0.070]
LLaDA-8BCarry RTL0.350-0.032 [-0.052, -0.011]+0.105 [+0.083, +0.127]+0.059 [+0.034, +0.084]+0.073 [+0.052, +0.095]
LLaDA-8BConstrained JSON Fill0.250-0.043 [-0.075, -0.011]+0.096 [+0.065, +0.127]+0.046 [+0.033, +0.059]+0.053 [+0.025, +0.081]
LLaDA-8BUnique List Commit0.540-0.205 [-0.233, -0.177]+0.420 [+0.392, +0.447]+0.185 [+0.154, +0.216]+0.215 [+0.184, +0.246]
LLaDA-1.5CSV Missing Cells0.280-0.002 [-0.018, +0.014]+0.042 [+0.023, +0.061]+0.031 [-0.002, +0.064]+0.040 [+0.028, +0.051]
LLaDA-1.5Multi-Span Cloze0.280-0.060 [-0.103, -0.017]+0.170 [+0.132, +0.208]+0.090 [+0.061, +0.119]+0.110 [+0.075, +0.145]
LLaDA-1.5Carry RTL0.300-0.035 [-0.064, -0.006]+0.165 [+0.135, +0.195]+0.106 [+0.063, +0.149]+0.130 [+0.100, +0.160]
LLaDA-1.5TD07 Sparse Mask0.350-0.027 [-0.043, -0.011]+0.140 [+0.129, +0.151]+0.097 [+0.074, +0.120]+0.113 [+0.093, +0.133]
LLaDA-1.5JSON Mode Eval0.200-0.039 [-0.244, +0.167]+0.261 [+0.038, +0.484]+0.184 [+0.043, +0.326]+0.222 [-0.006, +0.454]
LLaDA-1.5Unique List Commit0.450-0.028 [-0.053, -0.003]+0.155 [+0.128, +0.182]+0.110 [+0.091, +0.129]+0.127 [+0.099, +0.154]
DreamCarry RTL0.450-0.100 [-0.116, -0.083]+0.269 [+0.253, +0.285]+0.136 [+0.120, +0.152]+0.169 [+0.152, +0.185]
DreamHTML Close Tags0.420-0.040 [-0.084, +0.004]+0.124 [+0.084, +0.163]+0.066 [+0.036, +0.096]+0.084 [+0.047, +0.121]

Uncertainty: The three selective gains have strictly positive intervals in 10/12 cases. The exceptions are LLaDA-1.5/CSV (Sel − Random [−0.002, +0.064]) and LLaDA-1.5/JSON Mode (Sel − Fixed [−0.006, +0.454], n=20). Eight of the 12 cases additionally have a strictly negative Always − Fixed interval. These per-setting intervals are not multiplicity-adjusted.

We also performed a distinct Dream Carry RTL/cardinality end-to-end experiment (n=500; 38% coverage), matching its one-step detector finding (AUROC 0.802, 64.3% opportunity capture at 10% coverage):

FixedAlwaysRandomSelectiveSel − Fixed (95% CI)Sel − Always (95% CI)Sel − Random (95% CI)
0.5430.5200.5650.631+0.088 [+0.072, +0.105]+0.111 [+0.095, +0.127]+0.066 [+0.050, +0.083]

Dream/GSM8K accounts for 64.6% of Dream’s new +0.070 macro gain (versus 80.4% of the earlier pilot gain), but for 0% of the gains on either LLaDA model. The ex-GSM8K macro gains are +0.027 (Dream), +0.050 (LLaDA-8B), and +0.082 (LLaDA-1.5).

AuthorsOpenReview ↗

Official Rebuttal [3/3]

Q2: Why this paper focuses on whether some better choice exists, rather than whether the method's own proposed choice will be helpful?

A2: We distinguish the value of a possible intervention from the usefulness of our actual controller. The candidate-set oracle measures available opportunity, whereas empirical action selection, detector calibration, and end-to-end tests ask whether our method realizes it.

QuestionEvaluationWhat it establishes
Does a better action exist?Cross-fitted candidate-set oraclePotential one-step opportunity within tested actions
Can our diagnostics choose it?Held-out one-step comparisonsAction-selection quality
Can the controller identify when to intervene?Held-out AUROC, capture, calibrated coverageOpportunity detectability
Does it help generation?Disjoint final-test Fixed / Always / Random / SelectivePractical policy value and timing beyond intervention frequency

The new end-to-end comparisons include explicit negative and abstaining regimes; we no longer infer the policy’s benefit from an oracle gap alone.

** Q3: Is the five-axis framework intended to cover all major unmasking decisions, or is it a practical classification of existing methods? Please clarify which decisions, if there exists any, fall outside its scope. **

A3: Our intention is to provide a principled decomposition of unmasking decisions, rather than merely a practical classification of existing methods.

To clarify the expressive scope of our framework, we have added a formal completeness result (Theorem 2, Appendix A.1). Specifically, we prove that any valid reveal-and-commit decision can be represented by our transition-level parameterization. For any desired reveal set, choosing that set as the candidate region and setting the reveal fraction to one reproduces exactly the same selection, while the commitment axis specifies revisability. Moreover, with an unrestricted planning mechanism, the framework can represent arbitrary history-dependent stochastic reveal-and-commit policies while preserving their generation trajectory distributions.

This completeness result holds under a fixed denoiser, token-value sampler, and revision operator, rather than covering all aspects of inference. Token-value sampling decisions, changes to revision dynamics, and computational optimizations such as KV caching fall outside its scope.

More importantly, our motivation extends beyond representational completeness. By decomposing unmasking into interpretable decision axes, we aim to systematically investigate when and why each decision should adapt to the current state. This framework provides a foundation not only for understanding existing strategies but also for analyzing adaptation opportunities and developing new inference strategies.

Reviewer_zXxXOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to view unmasking strategies through 5 axes: score, region, cardinality, commitment and planning, exploring the first 4 and holding the last one static. For each axis, it measures how much could be gained by changing the action at a given step instead of always using a fixed action chosen on validation data. It then proposes a selective adaptation strategy, where a lightweight detector decides when to deviate from the fixed action, that changes one axis at a time.

Critical strengths
  • The idea of breaking inference into multiple separate axis is interesting and can lead to better categorization of inference methods as well as research ideas.
Critical weaknesses
  • I see the value of breaking down the inference process of discrete diffusion models into several axes, but this alone is not enough, and the current results do not show the value of an adaptive inference policy, which, for me, is the main contribution of the paper.
  • Additionally, the current results are hard to interpret as "utility" means different things depending on the dataset, but it is averaged across tasks. I also do not see why only up to 100 prompts are used, since the end-to-end evaluation is not expensive. Lastly, how does it compare with other decoding strategies in the literature (e.g., Fast-dLLM), and what is the tradeoff between NFEs and performance?
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Official Rebuttal [1/4]

We thank the reviewer for the careful assessment. The reviewer’s concerns about the practical value of adaptation, the comparability of utilities, and the NFE–quality trade-off helped us sharpen the empirical claims and expand the evaluation. We address each point directly below.

W1: I see the value of breaking down the inference process of discrete diffusion models into several axes, but this alone is not enough, and the current results do not show the value of an adaptive inference policy, which, for me, is the main contribution of the paper.

A1: We agree that the taxonomy alone cannot establish practical value. The scientific question is when changing a strategy is beneficial, and our expanded end-to-end study measures whether selective gating actually realizes that benefit. On a fresh, disjoint final-test split, Selective improves the validation-selected Fixed macro in all three models:

ModelFixedSelectiveSel − FixedTasks improvedTasks worsenedEx-GSM8K lift
LLaDA-8B0.4780.523+0.0455/100/10+0.050
LLaDA-1.50.3170.391+0.0746/100/10+0.082
Dream-7B0.4280.497+0.0703/101/10+0.027

More directly, in 12/30 settings, Always − Fixed is negative while Selective exceeds Fixed, Always, and a three-seed, coverage-matched Random gate. All three selective contrasts have strictly positive paired 95% bootstrap intervals in 10/12 of these settings (not corrected for multiple comparisons). For example, LLaDA-8B/Unique List: Fixed 0.365, Always 0.160, Random 0.395, Selective 0.580; Sel − Random = +0.185 [0.154, 0.216]. A separately evaluated Dream Carry RTL/cardinality controller also reaches 0.631 versus 0.543 Fixed and 0.565 Random (Sel − Random +0.066 [0.050, 0.083], n=500). These are end-to-end gating effects, not just oracle opportunities. We do not claim that adaptation improves every task.

  • LLaDA-8B: 0.478 →\rightarrow 0.523 (+0.045; 5/10 tasks improved, 0/10 degraded)
  • LLaDA-1.5: 0.317 →\rightarrow 0.391 (+0.074; 6/10 tasks improved, 0/10 degraded)
  • Dream-7B: 0.428 →\rightarrow 0.497 (+0.070; 3/10 tasks improved, 1/10 slightly lower by -0.006)

Crucially, in 12 of the 30 model–task settings (4 on LLaDA-8B, 6 on LLaDA-1.5, 2 on Dream), applying the alternative diagnostic action at every step (Always) hurts performance relative to Fixed (Always − Fixed < 0), whereas Selective adaptation activates the alternative action on only 20%–54% of states and simultaneously outperforms Fixed, Always, and a coverage-matched Random gate (Sel − Fixed > 0, Sel − Always > 0, Sel − Rand > 0). For example, on LLaDA-8B Unique List Commit, Always drops utility from 0.365 (Fixed) to 0.160, while Selective reaches 0.580 (+0.215 over Fixed, +0.185 over Random). Furthermore, on Dream Carry RTL/cardinality—where one-step opportunity detection is strong (AUROC 0.802, 64.3% capture at 10% coverage)—end-to-end selective decoding (38% coverage, n=500n=500) reaches 0.631, outperforming Fixed (0.543), Always (0.520), and Random (0.565). These results demonstrate concrete practical value precisely when state-aware gating is needed.

AuthorsOpenReview ↗

Official Rebuttal [2/4]

W2: Additionally, the current results are hard to interpret as "utility" means different things depending on the dataset, but it is averaged across tasks. I also do not see why only up to 100 prompts are used, since the end-to-end evaluation is not expensive. Lastly, how does it compare with other decoding strategies in the literature (e.g., Fast-dLLM), and what is the tradeoff between NFEs and performance?

A2: The reviewer is correct that utilities have different meanings across tasks despite their common [0,1] range. We therefore treat within-task paired differences as primary evidence; unweighted macro utility is descriptive, not a single interchangeable task score.

TaskHorizonTerminal utility (range [0,1])
CSV Missing Cells16Exact cell match (binary)
HumanEval256Pass all official unit tests (binary)
Multi-Span Cloze64Fraction of indexed spans correct
GSM8K256Extracted final numeric answer correct (binary)
Carry RTL32Fraction of three carry digits correct
TD07 Sparse Mask48Fraction of three separated masked spans correct
JSON Mode Eval128Field-level credit for schema-valid JSON; invalid = 0
Constrained JSON Fill96Fraction of constrained components correct; exact = 1
Unique List Commit48Position-level credit for a duplicate-free, ordered list
HTML Close Tags48Exact balanced close sequence = 1; partial credit per tag position

Expanded sample and literature baseline. Final-test prompts are disjoint from calibration and transition-level held-out prompts: 500 each for seven constructed tasks and GSM8K, 64 remaining HumanEval cases, and 20 remaining JSON Mode Eval cases. We compare six 256-NFE arms (Ref, Fixed, Stage, Always, Selective, Random) plus Fast-dLLM at threshold τ=0.9 and block size 32, reporting its measured NFE. All arms use the same prompts and scorers. Below are all 30 absolute outcomes and paired differences; the tables explicitly show that Fast-dLLM is better in several settings.

LLaDA-8B — final-test terminal utilities.

TaskAxisnRefFixedStageAlwaysSelectiveRandomFast (NFE)Sel. coverage
CSV Missing Cellsregion5000.0400.0500.0520.0440.1040.0630.060 (11)0.300
HumanEvalcommitment640.1410.1410.1410.1410.1410.1410.125 (130)0.000
Multi-Span Clozeregion5000.9820.9820.9750.9820.9820.9820.910 (19)0.000
GSM8Kregion5000.6600.7600.7600.7600.7600.7600.750 (80)0.000
Carry RTLregion5000.0300.6470.6430.6150.7200.6610.420 (24)0.350
TD07 Sparse Maskregion5000.1470.1000.1000.1550.1550.1550.080 (27)1.000
JSON Mode Evalregion200.3790.6710.6710.6710.6710.6710.550 (24)0.000
Constrained JSON Fillregion5000.9200.9080.9120.8650.9610.9150.932 (23)0.250
Unique List Commitregion5000.2600.3650.3530.1600.5800.3950.390 (30)0.540
HTML Close Tagscommitment5000.1600.1600.1600.1600.1600.1600.090 (22)0.356

LLaDA-8B — per-task paired policy contrasts.

TaskSel − Random (95% CI)Sel − Always (95% CI)Sel − Fixed (95% CI)
CSV Missing Cells+0.041 [+0.024, +0.058]+0.060 [+0.044, +0.075]+0.054 [+0.038, +0.070]
HumanEval0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Multi-Span Cloze0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
GSM8K0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Carry RTL+0.059 [+0.034, +0.084]+0.105 [+0.083, +0.127]+0.073 [+0.052, +0.095]
TD07 Sparse Mask0.000 [0.000, 0.000]0.000 [0.000, 0.000]+0.055 [+0.039, +0.071]
JSON Mode Eval0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Constrained JSON Fill+0.046 [+0.033, +0.059]+0.096 [+0.065, +0.127]+0.053 [+0.025, +0.081]
Unique List Commit+0.185 [+0.154, +0.216]+0.420 [+0.392, +0.447]+0.215 [+0.184, +0.246]
HTML Close Tags0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
AuthorsOpenReview ↗

Official Rebuttal [3/4]

LLaDA-1.5 — per-task paired policy contrasts.

TaskSel − Random (95% CI)Sel − Always (95% CI)Sel − Fixed (95% CI)
CSV Missing Cells+0.031 [-0.002, +0.064]+0.042 [+0.023, +0.061]+0.040 [+0.028, +0.051]
HumanEval0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Multi-Span Cloze+0.090 [+0.061, +0.119]+0.170 [+0.132, +0.208]+0.110 [+0.075, +0.145]
GSM8K0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Carry RTL+0.106 [+0.063, +0.149]+0.165 [+0.135, +0.195]+0.130 [+0.100, +0.160]
TD07 Sparse Mask+0.097 [+0.074, +0.120]+0.140 [+0.129, +0.151]+0.113 [+0.093, +0.133]
JSON Mode Eval+0.184 [+0.043, +0.326]+0.261 [+0.038, +0.484]+0.222 [-0.006, +0.454]
Constrained JSON Fill0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Unique List Commit+0.110 [+0.091, +0.129]+0.155 [+0.128, +0.182]+0.127 [+0.099, +0.154]
HTML Close Tags0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]

Dream — final-test terminal utilities.

TaskAxisnRefFixedStageAlwaysSelectiveRandomFast (NFE)Sel. coverage
CSV Missing Cellscommitment5000.0400.0600.0600.0600.0600.0600.060 (8)0.330
HumanEvalcommitment640.0780.0000.0000.0000.0000.0000.375 (86)0.000
Multi-Span Clozecommitment5000.3200.9000.9000.9000.9000.9000.799 (13)1.000
GSM8Kregion5000.4000.3600.3300.8100.8100.8100.828 (103)1.000
Carry RTLcommitment5000.4330.4670.4670.3670.6360.5000.543 (13)0.450
TD07 Sparse Maskscore5000.0000.0000.0000.0000.0000.0000.000 (25)0.000
JSON Mode Evalcommitment200.2780.3630.3630.3630.3630.3630.400 (16)0.792
Constrained JSON Fillcommitment5000.4201.0001.0001.0001.0001.0000.792 (11)0.340
Unique List Commitcommitment5000.1630.2680.2680.2400.2620.2680.267 (29)0.490
HTML Close Tagscommitment5000.3200.8600.8600.8200.9440.8780.820 (5)0.420

Dream — per-task paired policy contrasts.

TaskSel − Random (95% CI)Sel − Always (95% CI)Sel − Fixed (95% CI)
CSV Missing Cells0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
HumanEval0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Multi-Span Cloze0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
GSM8K0.000 [0.000, 0.000]0.000 [0.000, 0.000]+0.450 [+0.402, +0.498]
Carry RTL+0.136 [+0.120, +0.152]+0.269 [+0.253, +0.285]+0.169 [+0.152, +0.185]
TD07 Sparse Mask0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
JSON Mode Eval0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Constrained JSON Fill0.000 [0.000, 0.000]0.000 [0.000, 0.000]0.000 [0.000, 0.000]
Unique List Commit-0.006 [-0.017, +0.005]+0.022 [+0.011, +0.033]-0.006 [-0.017, +0.005]
HTML Close Tags+0.066 [+0.036, +0.096]+0.124 [+0.084, +0.163]+0.084 [+0.047, +0.121]
AuthorsOpenReview ↗

Official Rebuttal [4/4]

NFE–quality comparison. The following sweep uses a separate 100-prompt GSM8K subset of the new final-test pool, so its values should not be conflated with the 500-prompt table above. All observed sweep points are listed, including Fast-dLLM thresholds.

ModelPolicy / settingNFERefFixedSelectiveFast
LLaDA-8Bsteps 16160.4100.1600.160—
LLaDA-8Bsteps 32320.4700.5600.560—
LLaDA-8Bsteps 64640.6200.7800.780—
LLaDA-8Bsteps 1281280.6400.7800.780—
LLaDA-8Bsteps 2562560.6600.7700.770—
LLaDA-8BFast τ=0.534———0.630
LLaDA-8BFast τ=0.748———0.730
LLaDA-8BFast τ=0.861———0.750
LLaDA-8BFast τ=0.980———0.750
LLaDA-8BFast τ=0.9595———0.760
LLaDA-1.5steps 16160.3600.3600.360—
LLaDA-1.5steps 32320.4400.4400.440—
LLaDA-1.5steps 64640.4700.5500.550—
LLaDA-1.5steps 1281280.5000.6000.600—
LLaDA-1.5steps 2562560.6100.6100.610—
LLaDA-1.5Fast τ=0.532———0.670
LLaDA-1.5Fast τ=0.746———0.740
LLaDA-1.5Fast τ=0.860———0.740
LLaDA-1.5Fast τ=0.977———0.780
LLaDA-1.5Fast τ=0.9592———0.760
Dreamsteps 16160.1500.1500.160—
Dreamsteps 32320.2900.1500.270—
Dreamsteps 64640.3500.2200.670—
Dreamsteps 1281280.3600.2600.800—
Dreamsteps 2562560.4000.3600.830—
DreamFast τ=0.550———0.690
DreamFast τ=0.766———0.820
DreamFast τ=0.882———0.830
DreamFast τ=0.9103———0.830
DreamFast τ=0.95116———0.840

Fast-dLLM often reaches competitive or higher accuracy using far fewer forward passes (e.g., LLaDA-1.5 GSM8K). Conversely, Selective is stronger on several constraint-sensitive tasks at its reported step budget, such as LLaDA-8B Carry (+0.300 utility versus Fast, but 256 vs 24 measured NFE). We claim more useful action selection under a chosen decoding budget, not lower latency or uniformly superior efficiency. No wall-clock speedup is established by these experiments. These comparisons and their limitations are reported in Appendix I.6 (Tables 18 and 20).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.