acceptodds
Under review as a conference paper at ICLR 2027

Reveal Policies in Diffusion Language Models

Abstract

Masked diffusion language models (DLMs) generate text by repeatedly predicting masked tokens and choosing which position to reveal next. We show that this reveal policy is part of the decoder specification: with the same conditional predictions, different causal policies induce different distributions over final answers. We call each such distribution a reachable answer law and their set the reachable answer-law polytope, characterize that polytope recursively, and determine what repeated samples under one or several policies identify. For a finite acyclic decoder with known transition kernels, Bellman and occupancy-measure programs exactly optimize terminal-answer utility under hard per-generation or expected model-evaluation budgets. We prove a sharp value-of-information result: response information beyond current per-position marginals can carry nearly half the utility range, establishing revealed history and lookahead as valuable signals for optimal reveal control. On trained models, with one token revealed after each of 256 evaluations, confidence ordering beats random ordering across four DLMs and matches or beats left-to-right decoding at the same budget. A four-path reveal-policy portfolio improves over confidence batching by 6.7 points for Dream-7B and 5.6 for LLaDA-1.5 on the GSM8K grade-school math benchmark. This demonstrates a matched-compute gain from combining reveal policies. On the Massive Multitask Language Understanding (MMLU) benchmark, 32 randomized-order continuations from a threshold-selected partial state end in its leading option in 95.6-99.2% of samples across four models. Thus threshold-selected partial states induce highly concentrated answer laws even under randomized future reveal order.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language model, token reveal order, decoding strategy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xUzxOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reveal policy in masked diffusion language models: after each model evaluation, the decoder must decide which masked position to reveal next. Because later predictions are conditioned on previously revealed tokens, different causal reveal policies can induce different distributions over completed answers even when the underlying model is fixed. The paper defines the set R(s)R(s) of answer distributions reachable from a partial state ss under admissible causal reveal policies. It proves that R(s)R(s) satisfies a recursive characterization and is a convex polytope. It then connects linear optimization over this set to Bellman recursions and occupancy-measure programs, covering both hard per-generation budgets and expected evaluation budgets. A two-position construction shows that current per-position marginals can be insufficient for selecting the optimal first reveal. Empirically, the paper reports four main findings. First, confidence-based reveal ordering outperforms random ordering on GSM8K across four diffusion language models at matched sequence-evaluation budgets. Second, a four-path portfolio using confidence, margin, entropy, and left-to-right policies outperforms four confidence paths on Dream-7B and LLaDA-1.5. Third, exact enumeration on constrained three-slot graphs reveals large policy-dependent answer-law diameters. Finally, MMLU states selected by an option-confidence threshold exhibit concentrated outcomes under randomized future reveal orders, enabling an early-stopping rule that reduces model evaluations while approximately preserving accuracy.

Critical strengths
  • The reachable answer-law formulation cleanly separates reveal-policy effects from sampling noise and answer correctness.
  • The recursion for R(s)R(s) covers randomized, history-dependent causal policies rather than only fixed reveal orders.
  • The paper clearly distinguishes exact reachable-set characterization from lower bounds obtained using a finite tested policy family.
  • The matched-budget confidence-versus-random results are consistent across four trained models and include appropriate controls at block size one.
  • The three-slot experiments exhaustively enumerate all deterministic history-dependent policies and independently verify the resulting values.
  • The portfolio gain is replicated on a second model and remains positive under parser sensitivity analyses.
  • The prospective stopping experiment uses disjoint prompts and reports both accuracy and hardware timing.
Critical weaknesses
  • The main polytope, Bellman, and occupancy-measure results largely follow from standard finite-horizon MDP/CMDP machinery after projecting terminal distributions through an answer map. The paper should distinguish the conceptual contribution from the technical novelty more explicitly.
  • The portfolio comparison omits four-path margin-only, entropy-only, and left-to-right-only ensembles. Consequently, the evidence shows that the fixed portfolio outperforms four confidence paths, but not that diversity or complementarity is responsible.
  • The exact optimization assumes a fully known finite DAG of transition kernels and terminal utilities. Its state space grows exponentially with output length, and no estimation-error or scalable-planning guarantee is given for full-length models.
  • Corollary 1 is a useful insufficiency construction, but its interpretation is somewhat broad. The two-position example establishes the value of knowing the response structure beyond current marginals; it does not directly demonstrate an online controller exploiting a nontrivial revealed history over several decisions.
  • “Matched compute” should be qualified as matched sequence-evaluation budget. The compared arms can have different physical invocation counts, batch shapes, memory use, and wall-clock behavior.
  • The MMLU concentration result concerns one randomized continuation procedure on threshold-reaching prompts, not concentration over all admissible reveal policies. It is also conditional on a model-level inclusion criterion.
  • Comparisons with stronger learned, reward-guided, or adaptive reveal-order baselines are limited.
Questions for authors
  1. Could you report four-path homogeneous ensembles for margin, entropy, and left-to-right, as well as their paired differences from the portfolio? Without these controls, how can the portfolio gain be attributed to policy complementarity rather than one stronger constituent policy?
  2. Do leave-one-policy-out portfolios retain the gain? Which policy contributes most, and how sensitive is the result to the allocation of paths among policies?
  3. The appendix refers to a “subset ablation below.” Which experiment does this refer to, and are any intended ablation results missing from the current manuscript?
  4. Can you report wall-clock time, FLOPs, peak memory, and physical model invocations for the principal portfolio comparison? Please also clarify which claims use “compute” to mean sequence-model evaluations.
  5. How were the block size, portfolio composition, and 65%65\% stopping threshold selected? Were any GSM8K or MMLU test results inspected during these choices?
  6. Can the portfolio uncertainty analysis include random seed groups, for example through a crossed or hierarchical bootstrap over prompts and seeds?
  7. What part of Theorems 1 and 2 goes beyond applying standard finite-horizon MDP/CMDP reachable-distribution and occupancy-measure results followed by projection through the answer map?
  8. Can the response-information result be extended to at least three positions, where an observed token genuinely changes a later reveal decision? How should the cost of obtaining or estimating the response structure be accounted for?
  9. What guarantees remain when the transition kernels are estimated from finitely many model evaluations rather than known exactly?
2 comments
AuthorsOpenReview ↗

Response to reviewer xUzx

Thank you for your helpful questions and for noting that the reachable-set formulation separates policy effects from sampling noise and correctness. Numbers refer to the revised paper, where changes are in red.

Q1 and W2 (single-policy ensembles). General response G2 gives every single-policy ensemble (Table 13). On Dream-7B the portfolio beats all of them. On LLaDA-1.5 and JustGRPO (Ni et al., 2026), LLaDA-8B-Instruct trained with GRPO, four left-to-right (LTR) paths are best, so §3.2 now chooses the composition on 500 train prompts. It beats the best single-policy ensemble on test by +.0447 and +.0191 (Table 15).

Q2 and Q3 (leave-one-out, allocation, the "subset ablation below"). It is the drop-one ablation on a fixed 400-prompt LLaDA-1.5 subset. Table 11 restores it, with all eight contrasts. Every drop-one variant falls below the portfolio, and four LTR paths beat it, as in G2. On Dream-7B the 70 allocations span .4393–.6680. With blocks of 32, the four best mixes contain no LTR path.

Q4 and W5 (cost). On one A100 40GB (Table 17), four-path variants make 64 calls × 4 sequences per prompt and take 9.3 s and 9.7 s, at 17.1 and 16.8 GiB. One deep confidence path makes 256 × 1 calls in 11.7 s and 11.5 s. All reveal rules read their scores from the same model pass, so FLOPs are about equal. The paper now says "matched sequence-evaluation budget".

Q5 (choices). Block size 32 is common for LLaDA on GSM8K, as in JustGRPO. The portfolio was fixed in the Dream-7B plan. Before the choice we had seen the LLaDA-1.5 subset and 252 Dream-7B test prompts. On all other prompts the gains hold. The 0.65 threshold was frozen before the test on 4,250 new MMLU prompts (Table 30).

Q6 (seed runs). Intervals now resample prompts and seed runs. Chosen mix minus portfolio is +.0235 [+.0116, +.0357] and +.0611 [+.0455, +.0766]. An earlier resampling error made them too narrow. After the fix, Dream-7B portfolio minus four LTR includes zero.

W6 (MMLU scope). The concentration result covers random-order continuations, not every policy, as the paper now says. The inclusion rule (App. C.9) excludes SDAR-8B, now named.

W7 (baselines). Margin, from Kim et al. (2025), is one of our five policies. Against tuned self-consistency the mix wins on LLaDA-1.5 (+.0198) and ties on Dream-7B. It beats greedy one-step lookahead by +.0339 and +.0432 (G4). The learned policy of Hong et al. (2026) has no public checkpoints, so we could not run it.

Q7 and W1 (beyond MDP machinery). Theorem 3 makes the earlier sufficient condition exact. For completed outputs, the policy does not matter exactly when the predictions agree with one joint distribution, which holds exactly when swapping two reveals never changes it. Theorem 4 shows a mix can strictly beat every single-policy ensemble. Jury theorems (List and Goodin, 2001; Chen et al., 2024) take voters as given. Here each voter's answer distribution is chosen through its policy.

Q8 and W4 (three positions). Corollary 1(ii) extends the example to three positions. The first token decides which copy rule the next two follow. Any rule whose second reveal ignores it scores 1/2, and the rule that uses it scores 0.9. The model calls this needs enter Theorem 2 as action costs.

Q9 and W3 (estimated predictions). By Proposition 2, an answer distribution moves by at most the prediction error times the number of reveals, and a policy planned on estimates loses at most twice that (Table 5).

References

  • Lingjiao Chen, Jared Davis, Boris Hanin, Peter Bailis, Ion Stoica, Matei Zaharia, James Zou. Are More LLM Calls All You Need? Towards the Scaling Properties of Compound AI Systems. NeurIPS 2024, vol. 37, pp. 45767–45790.
  • Chunsan Hong, Seonho An, Min-Soo Kim, Jong Chul Ye. Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies. ICLR 2026, pp. 108226–108260.
  • Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, Sitan Chen. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. ICML 2025, PMLR 267:30749–30768.
  • Christian List, Robert E. Goodin. Epistemic Democracy: Generalizing the Condorcet Jury Theorem. Journal of Political Philosophy 9(3):277–306, 2001.
  • Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. ICML 2026, PMLR 306:93000–93015.
Reviewer_xUzxOpenReview ↗

Thank you for the detailed rebuttal.

Thank you for the detailed rebuttal. The additional homogeneous-ensemble comparisons, allocation ablations, hardware measurements, and revised uncertainty analysis substantially address my main empirical concerns. I also appreciate the more qualified claims, particularly regarding when policy mixing helps and the scope of the concentration results. The theoretical extensions help clarify the contribution, although I retain some reservations about its technical novelty and the scalability of exact planning. Overall, the response strengthens the paper and provides sufficient clarification of my concerns. I have no further questions and maintain my weak-accept recommendation.

Reviewer_cLVhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how reveal order affects the answers produced by masked diffusion language models. It characterizes the set of answer distributions attainable under different reveal policies and formulates policy optimization under compute constraints. The experiments compare reveal orders in terms of task accuracy, show improvements from combining different policies in an ensemble, and evaluate early stopping.

Critical strengths

S1. Findings on how reveal policies affect answer distributions

The paper goes beyond comparing task accuracy by quantifying how much the final-answer distribution can vary across reveal policies e.g, Appendix C.2. This provides empirical evidence that not only the initial reveal choice but also subsequent reveal ordering can significantly affect the final-answer distribution.

S2. Controlled evaluation of a reveal-policy portfolio

In Sec. 3.2, the authors compare a confidence-only ensemble with a portfolio of different reveal policies, keeping the model-evaluation budget, number of generation paths, and voting procedure fixed. This comparison partially isolates the benefit of combining different reveal policies from ensembling. The portfolio improves accuracy on two models, with consistent gains under alternative answer-extraction procedures.

Critical weaknesses

W1. Weak connection between theory and experiments

The main theoretical results (Sec.2) largely rely on standard tools from MDP/convex analysis. More importantly, the experiments (Sec.3) do not follow naturally from their theoretical analysis, nor do they illustrate how the theory explains the observed empirical gains. Thus, the theoretical analysis remains largely disconnected from the empirical findings, making it difficult to understand a central claim that ties the paper together.

W2. Insufficient empirical analysis of the portfolio's effectiveness and practical efficiency

The portfolio results in Sec.3 are interesting, yet the paper lacks ablations of individual reveal policies and alternative combinations, leaving it unclear why the particular four-policy combination works well. Although the number of evaluations across arms is matched, the paper does not report FLOPs, latency, or throughput, making its practical efficiency difficult to evaluate. Moreover, in settings where training costs can be amortized, optimizing a single-path decoder during training may be more cost-effective in the long term running. The paper would benefit from comparisons with, or at least a discussion of, existing training-based approaches [1-4].

W3. Novelty and generality of the empirical findings

The impact of reveal order on text generation (Sec. 3.1) has already been demonstrated in prior work [5-6]. The early-stopping experiment in Sec. 3.3 assumes a known answer position and a fixed set of answer options, leaving its applicability to general open-ended generation unclear. Its relationship to existing work on early stopping and token stabilization [7-9] is also insufficiently discussed. At last, although the experiment is related to answer-law concentration, its connection to the central theme (in my understanding) of reveal-policy control and optimization is not clearly established, weakening the overall coherence of the paper.

References

[1] Tang et al., wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models. 2026.

[2] Oba et al., Diffusion-State Policy Optimization for Masked Diffusion Language Models. 2026.

[3] Xie et al., Step-Aware Policy Optimization for Reasoning in Diffusion Large Language Models. 2025.

[4] He et al., MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models. 2025.

[5] Kim et al., Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. 2025.

[6] Hong et al., Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies. 2026.

[7] Kim et al., KLASS: KL-Guided Fast Inference in Masked Diffusion Models. 2025.

[8] Oba et al., Stopping Computation for Converged Tokens in Masked Diffusion-LM Decoding. 2026.

[9] Wei et al., Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles. 2026.

Questions for authors

How does the theoretical analysis (Sec.2) help explain the empirical findings observed in Sec. 3? Can the authors clarify what insights from the theory are supported by these experiments?

(other main questions and concerns are already outlined in the Critical Weaknesses section above.)

3 comments
AuthorsOpenReview ↗

Response to reviewer cLVh

Thank you for your thorough review and for the references. We are glad the reviewer found the measured spread of answer distributions across reveal policies informative. Revisions are in red.

W1 and Q1 (theory and experiments). Two new theorems are specific to reveal control (general response G1). Theorem 3 says exactly when all reveal policies give the same outputs, and bounds how far apart they can be. Theorem 4 gives the exact voted accuracy of any mix of policies across paths. §2.3 (new) ties each result to a test (G1). The vote formula ranks all 70 compositions on new runs with rank correlation .936–.962.

W2 (ablations, efficiency, training-based methods). In G2 the portfolio beats every single-policy ensemble on Dream-7B. Chosen on 500 train prompts, a mix beats the best single-policy ensemble on test by +.0447 and +.0191. On one A100, four-path variants take 9.3 s and 9.7 s per prompt, and one deep confidence path 11.7 s and 11.5 s (Table 17). All reveal rules read their scores from the same model pass, so FLOPs are about equal. Choosing a mix needs no training. Reveal control also applies after training. On JustGRPO [10], LLaDA-8B-Instruct trained with GRPO, four left-to-right (LTR) paths become the best mix. §4 now discusses [1–4].

W3 (novelty, stopping, coherence). [5, 6] show that reveal order matters. We add the set of answer distributions reveal policies can reach (Theorem 1), when the policy does not matter (Theorem 3), why rules using only current predictions, as in [5], can miss the best reveal (Corollary 1), and which policy each voting path should use (Theorem 4). A learned policy [6] reaches one point in that set, and Theorem 2 finds the best point on a known graph. Without blocks LTR acts like block size 1, so part of the gain could come from mixing block sizes. With blocks of 32, mixing still beats the best single-policy ensemble, and the four best of all 70 mixes contain no LTR path (Table 20).

The stopping rule needs a fixed answer position and option list, described in details in App. C.10. KLASS [7] and SlowFast [9] decide when single tokens commit. SureLock [8] skips work on settled tokens and still decodes to the end. Our rule ends the generation once one answer is very likely, saving about 53% of MMLU model calls. Proposition 1 treats stopping as one more decoder action, whose best rule is a threshold.

References

  • [1] Xiaohang Tang, Rares Dolga, Sangwoong Yoon, Ilija Bogunovic. wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models. ICLR 2026, pp. 150357–150385.
  • [2] Daisuke Oba, Hiroki Furuta, Naoaki Okazaki. Diffusion-State Policy Optimization for Masked Diffusion Language Models. NeurIPS 2026.
  • [3] Shaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong, Guangyi Chen, Eric P. Xing, Kun Zhang. Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards. ACL 2026, pp. 42703–42720 (arXiv:2510.01544, first titled "Step-Aware Policy Optimization for Reasoning in Diffusion Large Language Models").
  • [4] Haoyu He, Katrin Renz, Yong Cao, Andreas Geiger. MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models. arXiv:2508.13148, 2025.
  • [5] Jaeyeon Kim, Kulin Shah, Vasilis Kontonis, Sham M. Kakade, Sitan Chen. Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions. ICML 2025, PMLR 267:30749–30768.
  • [6] Chunsan Hong, Seonho An, Min-Soo Kim, Jong Chul Ye. Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policies. ICLR 2026, pp. 108226–108260.
  • [7] Seo Hyun Kim, Sunwoo Hong, Hojung Jung, Youngrok Park, Se-Young Yun. KLASS: KL-Guided Fast Inference in Masked Diffusion Models. NeurIPS 2025, vol. 38, pp. 92267–92301.
  • [8] Daisuke Oba, Danushka Bollegala, Masahiro Kaneko, Naoaki Okazaki. Stopping Computation for Converged Tokens in Masked Diffusion-LM Decoding. ICLR 2026, pp. 143558–143580.
  • [9] Qingyan Wei, Yaojie Zhang, Zhiyuan Liu, Puyu Zeng, Yuxuan Wang, Biqing Qi, Dongrui Liu, Linfeng Zhang. Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles. ICLR 2026, pp. 4340–4354.
  • [10] Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. ICML 2026, PMLR 306:93000–93015.
Reviewer_cLVhOpenReview ↗

Response to rebuttal

Thank you for the detailed response.

The new results, particularly the validation of the voting formula, make the connection between the theory and experiments much clearer, regarding W1. The additional ablations and runtime measurements also address several of my concerns regarding W2.

My main concerns now relate to the practical value and generality of the empirical contributions regarding W2 and W3. The tuned reveal-policy portfolio does not significantly outperform self-consistency with independently tuned temperatures for each path. At the matched 256-evaluation budget, a single greedy path also achieves higher accuracy, although the portfolio is somewhat faster. These results leave me unconvinced that mixing reveal policies offers a clear advantage over other ways of introducing diversity.

Regarding W3, the new theoretical results help distinguish this work from prior studies, yet empirical observation that reveal order matters is already established in some literature, and the provided early-stopping experiments still rely on a predefined answer position and fixed answer options, a limited settings for DLMs. Given these remaining concerns, I will keep my original score.

Thank you for the follow-up.

Mixing versus temperature. We agree that mixing reveal policies does not beat every other source of diversity. Self-consistency with per-path temperatures gives each of its four confidence paths its own temperature, chosen on train prompts. With four full paths on the 1,319 test problems, it ties our train-chosen mix of reveal policies, as the abstract states. The mix minus this variant is +.0065 on Dream-7B and +.0076 on LLaDA-1.5.

Standard self-consistency runs four confidence paths at one temperature. With both tuned on train prompts, the mix beats it on LLaDA-1.5 in two tests. It wins by +.0201 on 1,000 new problems and by +.0198 on the test problems (adjusted p = .0026 and .0030). On 1,000 more problems, the LLaDA-1.5 mix also beats self-consistency at each of three temperatures (adjusted p ≤ .0002). Its smallest gain, +.0179, is against T = 1.2, the best of the three on those problems. On Dream-7B the two tie whenever both are tuned. The differences are +.0005 and −.0019 on two sets of 1,000 new problems and −.0005 on the test problems.

Theorem 4 explains why temperature can match mixing. A large vote is correct when the correct answer's share of paths exceeds the leading wrong answer's share (part b). Mixing raises this margin only by lowering the leading wrong answer's share (part d). Table 22 shows that a higher temperature lowers this share about as much, but costs more single-path accuracy. On one set of 1,000 new Dream-7B problems, self-consistency at its train-chosen T = 1.0 has a leading wrong share of .0965 and a single-path accuracy of .8498. The fixed portfolio at T = 0.2 (one confidence, one margin, one entropy and one left-to-right path) has .1022 and .8587. Their margins are close (.7533 and .7564), and so are their votes (.8821 and .8816). So mixing is a second source of diversity next to temperature, at the same number of model calls. Which one helps more depends on the model.

One greedy path at 256 evaluations. This result compares one deep path with four shallow ones. At 256 calls, each of four batched paths gets 64 calls, a quarter of a full decode. One greedy path with all 256 calls scores .8271 on Dream-7B and .7870 on LLaDA-1.5. All 70 four-path compositions score lower (at most .7545 and .7647). This split limits any four-path ensemble, whatever its source of diversity. When each voting path is a full decode, the train-chosen mix beats the greedy path by +.0347 and +.0330. It uses four times the evaluations and a median 3.06 and 3.36 times the greedy path's wall time. A greedy path that spends a similar budget (976 evaluations) on one-step lookahead takes 0.92 times the mix's wall time. The mix beats it by +.0339 and +.0432 (adjusted p ≤ .0003).

Prior work (W3). We agree that earlier work shows that reveal order matters. As you note, the new theoretical results help distinguish this work from prior studies. Our experiments test what these results predict, for example the vote formula's ranking of all 70 compositions (rank correlation .936–.962).

Stopping. The stopping test uses answers with a designated position and a fixed option set. This format covers multiple-choice and classification tasks. A masked diffusion model scores the answer position at every evaluation, so the rule can stop while the answer is still masked. On 4,250 new MMLU prompts per model, it cuts model evaluations by 53.1% on LLaDA-8B and 53.9% on MMaDA-8B. Both stay within a one-point accuracy tolerance set before the test. A retrospective analysis finds savings of 52%–63% for six models on MMLU and for LLaDA-8B on ARC-Challenge, and no interval shows an accuracy loss (Table 29). Proposition 1 shows that the best stopping rule is a threshold for any answer format.

Reviewer_edYVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how the reveal policies affect the output distribution of diffusion language models. Theoretically it proposes a reachable answer distributions set and formulate the budgeted decoding as a finite MDP. Validations include small decoding graphs, GSM8K and MMLU task.

Critical strengths
  1. The formulation of budgeted decoding as MDP is clear, and the distinction between policy effects, sampling, and correctness in Table 3 is clear on what repeated-sampling experiments can and cannot establish.
  2. The experiment design is careful. Comparisons are matched on model-evaluation budget, several key contrasts are preregistered, results are replicated across models, and the stopping rule is tested prospectively on a disjoint prompt set. The experiments under fixed budget is useful evidence to support the importance of decoding policy choice.
Critical weaknesses
  1. Limited connections between theories and practical methods. The reachable-set theory formulates the answer law under individual policies, and the portfolio gain in the experiment comes from majority voting across four paths. I don’t see the theory explains why voting across policies should improve accuracy. Moreover, Corollary 1 argues for the importance of response-aware control, but all four portfolio policies still comes from marginals. Also the practical portfolio and stopping rules are more like heuristic instead of from the optimal control analysis.
  2. Incomplete comparison for the portfolio gains. The portfolio combo performs better than the confidence baseline. However, it is unclear whether the gain is from the combination itself or just because a stronger decoding strategy is included, such as margin or L2R
Questions for authors

Can authors please compare the portfolio performance with all the homogeneous ensembles of each different decoding policy under the same budget? How is the practical portfolio design and the stopping rule motivated from your theoretical analysis?

2 comments
AuthorsOpenReview ↗

Response to reviewer edYV

Thank you for your careful review, and for recognizing the clarity of the MDP formulation and the matched-budget experimental design. Your request to compare the portfolio with every single-policy ensemble led us to rebuild §3.2. Numbers refer to the revised paper, where changes are in red.

W1 (why voting across policies helps). A composition says which reveal policy each voting path uses. The new Theorem 4 gives the exact voted accuracy of any composition. A vote is won when the correct answer appears more often than the most common wrong answer, so single-path accuracy alone does not decide it, and a mix can beat every single-policy ensemble. Estimated from two seed runs, the formula ranks all 70 compositions on a held-out run with rank correlation .936–.962 in four new runs. Mean single-path accuracy gives only −.743 to +.613 (general response G3). Theorem 4(d) gives the mechanism. A mix's vote margin beats the average of its policies' margins exactly when their most common wrong answers differ. In 5 of 6 run–model cells, the mixing gain is larger on prompts where this holds (App. C.7).

W1 (Corollary 1). The portfolio policies use only current predictions. Corollary 1(ii) builds a three-position example where any rule that sees only current predictions scores 1/2 and the best rule scores 0.9. On transition graphs measured from Dream-7B and LLaDA-1.5, 1,140 small MMLU graphs each, exact planning beats confidence ordering on 96.1% and 92.5% of graphs, by +.0108 and +.0329 (Table 8). The score there is agreement with the options the model scored highest before revealing anything. It uses no labels and is not accuracy. A fitted rule that sees current predictions and this score recovers 78% and 82% of the gain. Exact planning still beats it by .0024 and .0058, with intervals above zero.

W1 and Q1 (portfolio and stopping rule). The portfolio was fixed beforehand. Theorem 4 makes the best composition depend on each policy's answer distributions, which vary by model, so we now choose it on 500 train prompts. Chosen this way, it beats the best single-policy ensemble on test by +.0447 (Dream-7B) and +.0191 (LLaDA-1.5, Table 15). For stopping, Proposition 1 shows that the best rule is a threshold on the true agreement with a continued decode. Our 0.65 rule uses the leading option's score in its place. Proposition 1(v) bounds the rule's agreement from below by the threshold plus the gap between true agreement and score at stopping states. We measure that gap as .093–.118 under random reveal order (Table 27).

W2 and Q1 (combination or one strong policy?). It depends on the model (G2, Table 13). At 256 full-length model passes per prompt, the portfolio beats four margin and four entropy paths on both models (corrected p = .0003 each). On Dream-7B it beats every single-policy ensemble, including four left-to-right (LTR) paths (+.0212). On LLaDA-1.5 four LTR paths beat it by .0420. On JustGRPO (Ni et al., 2026), LLaDA-8B-Instruct trained with GRPO, four LTR paths are the best of all 70 compositions. A mix chosen on train prompts still beats four LTR on test on both models, by +.0447 and +.0191.

References

  • Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. ICML 2026, PMLR 306:93000–93015.

Thank you for your detailed rebuttal. My concerns are mostly solved. I will raise my score to 3.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response [1/2]

We thank the reviewers for their careful reading. G1 links the theory to the experiments. G2 tests whether the ensemble gain comes from mixing policies or from one strong policy. G3 checks the vote formula, the vote rule and blocked decoding. G4 compares mixing with self-consistency. G5 lists the revisions. Numbers refer to the revised paper, where changes are in red.

G1. What the theory predicts, and what we observe.

A reveal policy picks which masked position to reveal next. A composition says which policy each of the four voting paths uses. Two new theorems (App. B.1) say when the policy matters and how voting over policies works.

  • Theorem 3. All reveal policies give the same outputs exactly when the per-position predictions agree with one joint distribution. Otherwise the spread between policies is bounded by order effects, that is, by how much the result changes when two reveals swap order.
  • Theorem 4. It gives the exact voted accuracy of any composition. A vote is won when the correct answer appears more often than the most common wrong answer, so single-path accuracy alone does not decide it. A mix can strictly beat every single-policy ensemble.

Each result has a test (§2.3):

  • Block size (Theorem 3). At block size 1 all policies give identical outputs. At block size 32, confidence ordering beats random ordering by +.0604 to +.1572 (Table 1), and by +.1866 and +.0492 on MATH-500.
  • Spread (Theorem 1). On small graphs where every reachable answer distribution can be computed, the two most different policies differ by 0.449 and 0.456 in total variation, on average.
  • Planning (Theorem 2). On graphs measured from Dream-7B and LLaDA-1.5, exact planning beats confidence ordering by +.0108 and +.0329 (Table 8). The score is agreement with the options the model scored highest before revealing anything. It uses no labels and is not accuracy.
  • Voting (Theorem 4). The formula ranks all 70 compositions on new runs with rank correlation .936–.962 (G3).
  • Stopping (Proposition 1). The best stopping rule is a threshold. Our 0.65 rule saves 53.1% and 53.9% of the eight model calls on MMLU, within one point of accuracy (§3.3).

G2. Single-policy ensembles, and the composition chosen on training prompts.

We ran every four-path single-policy ensemble on all 1,319 GSM8K test prompts, each at 256 full-length model passes per prompt (Table 13). JustGRPO is LLaDA-8B-Instruct trained with GRPO (Ni et al., 2026). Voted accuracy, Dream-7B / LLaDA-1.5 / JustGRPO:

  • portfolio (one confidence, one margin, one entropy and one left-to-right path): .6445 / .5776 / .6796
  • four confidence: .5857 / .5215 / .5986
  • four margin: .5870 / .5046 / .5816
  • four entropy: .5122 / .4713 / .5919
  • four left-to-right (LTR): .6233 / .6196 / .8530

On Dream-7B the portfolio beats every single-policy ensemble. On LLaDA-1.5 and JustGRPO four LTR paths are best. No composition is best on every model, so we choose the composition on 500 GSM8K train prompts and score it once on test (Table 15).

  • Dream-7B: the chosen mix scores .6680, the best of 70. It beats four LTR by +.0447 [+.0283, +.0611] and the portfolio by +.0235 [+.0148, +.0321].
  • LLaDA-1.5: the chosen mix scores .6387. It beats four LTR by +.0191 [+.0060, +.0321] and the portfolio by +.0611 [+.0464, +.0755].

All four differences have p ≤ .003 after correction for multiple tests. They also hold on the test prompts we had never inspected before the choice.

AuthorsOpenReview ↗

General response [2/2]

G3. The vote formula, the vote rule and blocked decoding.

  • Ranking (Table 16). From two seed runs of each policy, the formula ranks all 70 compositions on the third run. On four new runs (SVAMP on three models, GSM8K on LLaDA-8B-Instruct) the rank correlation is .936–.962. Mean single-path accuracy gives only −.743 to +.613.
  • Prediction (App. C.8). Predicted from single-policy runs at the same batch size, all 21 real mixed ensembles fell within ±3 points of the prediction. An earlier test that mixed batch sizes had 13 of 14 inside.
  • Vote rule (Table 24). The vote counted each unparsed output as its own answer. We also scored failures as one shared wrong answer, and left them out. Under all three rules the chosen mix beats the portfolio. On LLaDA-1.5 its edge over the best single-policy ensemble is +.0191, +.0345 and +.0027, and the last interval includes zero.
  • Blocked decoding (Tables 20–21). With blocks of 32, the chosen mix beats the best single-policy ensemble on both models (+.0189, +.0105), and the four best mixes contain no LTR path. At the same 256 evaluations, one greedy blocked path is more accurate than any four-path ensemble (.8271, .7870). When every voting path is a full blocked decode, the mix beats that greedy path by +.0347 and +.0330, at about 3.1 and 3.4 times its wall time.

G4. Self-consistency, SVAMP and RL training.

With four full blocked paths each, and temperatures chosen on the same train prompts, the mix beats self-consistency on LLaDA-1.5 by +.0198 [+.0080, +.0315] on all 1,319 test problems (corrected p = .0030) and ties it on Dream-7B. It beats one greedy path with one-step lookahead by +.0339 and +.0432 at about the same wall time (Tables 22–23). On 1,000 SVAMP problems, the mixes chosen on GSM8K beat four LTR on Dream-7B and LLaDA-1.5 (+.1149, +.0494), so the composition is best chosen on the target task. RL training raises JustGRPO's single-path accuracy by +.3249 under LTR and +.0811 under confidence, so training changes which policy is best (Table 25).

G5. Revisions (in red, with the main text within 10 pages).

  1. The abstract, §3.2 and Table 2 report G2–G4, including the results that favor one policy and the greedy blocked path.
  2. §2.3 (new) links each theory result to its test. Corollary 1(ii) extends Corollary 1(i) to a later decision.
  3. App. B.1–B.4 (new) give Theorems 3–4 and Propositions 1–2. Table 4 lists, for each result, the standard tool it builds on and what is new.
  4. Appendix C adds MATH-500, intervals that also resample seed runs (fixing an error that had made them too narrow), blocked decoding, self-consistency, the vote-rule check, SVAMP and RL training, ensemble prediction, the excluded SDAR-8B checkpoint and how the 0.65 threshold was frozen.
  5. §4 discusses the nine works cLVh raised, and adds other work on reveal policies, compute-matched comparisons and voting.
  6. "Matched compute" is now "matched sequence-evaluation budget", the same number of full-length model passes per prompt.

References

  • Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models. ICML 2026, PMLR 306:93000–93015.
AuthorsOpenReview ↗

Summary of the discussion for the AC

Reviewer edYV raised their score from weak reject to weak accept ("My concerns are mostly solved").

Reviewer xUzx keeps weak accept. They write that the single-policy ensembles, allocation ablations, hardware measurements, and revised uncertainty analysis "substantially address my main empirical concerns". Their remaining reservations are the technical novelty of the theory and the scalability of exact planning. On novelty, Theorem 3 shows that the reveal policy cannot change completed outputs exactly when the model's predictions agree with one joint distribution. Theorem 4 shows that a mix of policies can strictly beat every single-policy ensemble. Exact planning is a check on graphs measured from the models, not a decoder for full-length generation. A rule fitted on training graphs that sees only current per-slot probabilities recovers about 80% of exact planning's gain over confidence ordering.

Reviewer cLVh writes that the validation of the voting formula makes the connection between theory and experiments "much clearer". They add that the ablations and runtime measurements address several of their W2 concerns. Two concerns remain, and we answered both in our reply under their comment.

  1. Self-consistency with per-path temperatures ties our train-chosen mix of reveal policies, as the abstract states. Against standard self-consistency, which uses one temperature, the mix wins on LLaDA-1.5 in three held-out tests and ties on Dream-7B. Theorem 4 explains why temperature can match mixing.
  2. The stopping test uses answers with a designated position and a fixed option set. A diffusion model scores that position at every evaluation, so the rule can stop early. On 4,250 new MMLU prompts per model, it cuts model evaluations by 53.1% and 53.9% within the pre-set accuracy tolerance.

A new comparison adds evidence on the first point. HEX (Lee et al., 2026) votes over paths decoded with different block sizes. We compared it with our train-chosen mix on the 1,319 GSM8K test problems, at 1,024 evaluations per problem. The HEX ensemble keeps four of HEX's five block sizes (8, 16, 32, and 64), so that four paths cost 1,024 evaluations. Its temperature was tuned for self-consistency in blocks of 32, not for HEX. On Dream-7B and LLaDA-1.5 the mix scores .8721 and .8278, against .8597 and .8228 for the HEX ensemble. Averaged over the two models, the mix is ahead by +.0082 (adjusted p = .0184). At HEX's published temperature of 0.9, the HEX ensemble scores .8659 and .8212, and the mix is still ahead by +.0061. The mix and the temperature were chosen on train prompts. The test problems were used in earlier tests, so the mix's accuracy on them was already known. Like the mix, the HEX ensemble reveals one token per call and uses our prompt, answer parser, and vote. It also differs from HEX as released in the models and in nucleus sampling at 0.95. Its sampler draws from the same distribution with a different random stream, and it decodes each prompt alone. On Dream-7B it uses Dream's one-position logit shift, which HEX does not implement. Run with HEX's own prompt, parser, and five paths on LLaDA-8B-Instruct, our implementation scores .8362, against the .8218 that Lee et al. report. We have uploaded a revised PDF with this comparison. It is the paragraph "Against HEX at the same budget" in Appendix C.6.

References

  • Jihoon Lee, Hoyeon Moon, Kevin Zhai, Arun Kumar Chithanar, Anit Kumar Sahu, Soummya Kar, Chul Lee, Souradip Chakraborty, Amrit Singh Bedi. Test-Time Scaling in Diffusion LLMs via Hidden Semi-Autoregressive Experts. ICLR 2026, pp. 46880–46899.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.