acceptodds
Under review as a conference paper at ICLR 2027

Towards Solving the Unmasking Order in dLLMs via Constrained MaxEntropy Sampling

Abstract

A masked diffusion language model (dLLM) must choose which masked position to denoise at every step. Various rules exist for this choice, such as committing the most confident position (confidence order) and sampling positions in proportion to a tempered confidence (tempered selection). They are stated as separate procedures with no common objective, and tempered selection keeps every position eligible, so it cannot defer a commitment. We present **PACE**, a *position sampling strategy with adaptive confidence eligibility*, which keeps the odds of tempered selection but defers the least confident positions under an entropy budget relative to tempered selection. PACE follows from one constrained optimization problem: find the highest-entropy distribution over positions whose expected *damage*, the cost of a commitment, stays within a budget. Its solution is a Gibbs distribution over positions, and each rule we study corresponds to one choice of damage function, eligible position set and Lagrange multiplier, which is the inverse position temperature. A damage that depends on the committed token's probability and adds over successive commitments must be the surprisal, which gives the odds of tempered selection, and we prove that, among rules with these odds that defer only the least confident positions and meet the budget, PACE is the unique one that defers the fewest. We also show that selecting positions by the tempered confidence of their drawn tokens sharpens the distribution of written tokens toward an effective temperature that combines the token and position temperatures. On two model families and four benchmarks at token temperature , PACE improves single-sample accuracy on all eight model–benchmark pairs at the same compute, by to points over confidence order and to points over tempered selection, and leads at every number of samples up to ; its single-sample gain over tempered selection holds at a second token temperature.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, masked diffusion models, decoding, sampling, inference-time methods

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eKBEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors formulated the next token to predict in masked diffusion models as a constrained optimization problem, i.e., maximize sampling randomness while retaining high confidences. The derived solution is to sample from the longest prefix that fulfills the confidence budget. The solution appears to be theoretically sound.

Critical strengths
  1. simple solution derived from well formulated problem.
  2. the decoding strategy is algorithmically adaptive to the decoding status.
Critical weaknesses
  1. missed comparison with naive but straightforward alternatives, e.g., sampling among top-m, position level top-p, tempered selection. While these methods have fixed scheme during the decoding, the proposed method seems to have some adaptiveness. It would be interesting to see the benefits. Also, each method requires some hyperparameter setting. so it is important to make sure they are properly selected and sensitivity analysis across different datasets.
  2. related to the point 1, it would be interesting to analyze the effectiveness at different decoding stages, showcase how such adaption would change.
  3. the idea is highly similar to top-H decoding, where the extension is to select positions for decoding rather than token vocabularies.
Questions for authors
  1. tempered selection was compared and PACE showed better performance. However, from the same optimization objective, tempered selection was also an optimum, whereas PACE added additional heuristics. Please explain where the gains from. also pay attention if the hyperparameters were chosen in a fair way.
  2. additionally, can we extend this to multi-position predictions per decoding?
Reviewer_59j4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In discrete diffusion, generation during inference mainly involves two steps: token sampling and position sampling. The paper introduces a new heuristic for position sampling called PACE, which uses an adaptive entropy cutoff to exclude low-confidence positions before sampling based on confidence. Other existing heuristics include high confidence, margin, entropy, random. And these heuristics support both deterministic and stochastic sampling, allowing variation from a peaked distribution to a uniform distribution by adjusting the temperature. PACE adds a minor restriction to confidence-based position sampling, but its benefits over well-tuned existing heuristics (mentioned above) remain unclear.

Critical strengths

None

Critical weaknesses
  1. Novelty and comparison with existing heuristics:
  • PACE adds a minor restriction to existing confidence-based sampling without providing a fair and substantial comparison with existing heuristics. The authors put a lot of emphasis on theory, which obscures the lack of empirical evidence that this small modification offers more than better tuning of existing methods.
  1. Limited evaluation and readability:
  • The paper tests LLaDA-8B and Dream-7B on subsets of four downstream benchmarks. These experiments do not clearly show where PACE's ordering helps. The authors should test settings where ordering matters, i.e. tasks where ordering matters and better heuristics shine. e.g. Sodoku.

  • Stochastic confidence sampling: The comparisons with stochastic versions of existing heuristics are limited. The authors should compare PACE with for example high confidence and margin heuristics with stochastic sampling enabled instead of deterministic. This would help establish whether PACE adds a benefit or whether adding stochastic to existing heuristics already achieve similar results.

  • Efficiency evaluation is also limited: The choice of a heuristic doesnt only come from better performance but does it provide better generation efficiency under matched compute (few step generation). This is also very limited.

  • The paper is hard to read for a fairly simple idea. Paper needs refactoring, reorder of section and clear explanation of the heuristic supported by good results.

Questions for authors

No questions, see above for the weaknesses.

Also my final recommendation: The authors should test PACE in the existing codebases, setups and tasks used in earlier works that introduce other heuristics, particulary where heuristic differences matter. Since it requires only a few lines of code and no retraining, comparisons should be straightforward where checkpoints are available. Where checkpoints are unavailable, the authors could train the smaller models. In my experience, most of those results are reproducible, and testing PACE in these setups should be straightforward.

Reviewer_83WFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the unmasking order problem in Masked Diffusion Language Models and proposes PACE, a training-free position sampling strategy. The method first constructs a tempered position distribution based on the confidence of the candidate tokens sampled at each position, then adaptively truncates low-confidence positions according to a relative-entropy budget, and finally randomly selects the position to be determined from within the retained candidate set. The authors interpret this strategy through a maximum-entropy optimization framework and analyze the connection between position selection and token temperature. Experiments on LLaDA and Dream show that PACE achieves noticeable single-sample accuracy improvements over Confidence Order and Tempered Selection.

Critical strengths
  1. The research problem is of practical significance. Unmasking order is an important degree of freedom that distinguishes dLLMs from autoregressive language models and has a direct impact on generation quality and the inference process.

  2. The method is simple and the experimental results are relatively good. PACE requires no additional training or model forward calls and achieves relatively clear Pass@1 improvements on multiple mathematical reasoning and code generation tasks.

Critical weaknesses

W1. The comparison with existing adaptive decoding methods is insufficient. In recent years, several works have studied adaptive unmasking for dLLMs, such as information-gain-based sampling [1], learned unmasking policies [2], Frontier Gating [3], and Entropy-Bounded Unmasking [4]. These works are directly related to the research objective of PACE, but this paper mainly compares with Confidence Order, Tempered Selection, and fixed AR Order. Although different methods may have different training and inference overheads, the authors are still advised to add several representative strong baselines under unified models, sampling settings, and computational budgets. In addition, Top-k/Top-p position sampling and the Adaptive Position Temperature discussed in the theoretical part of the paper are also worth direct comparison. These experiments would help more fully demonstrate the advantages of PACE over existing adaptive decoding strategies.

W2. The connection between the core algorithm and existing Top-H Decoding deserves further clarification. The main operation of PACE is to sort position probabilities and then select the largest prefix set satisfying H(qS)≤αH(q)H(q_S)\leq\alpha H(q), followed by renormalized sampling within the retained set. This is very close to the core mechanism of Top-H Decoding [5], which performs truncation based on a relative-entropy budget; the main difference is that the truncation target is shifted from the token probability distribution to the unmasking position distribution. Although this transfer may be applicable to the decoding-order problem specific to dLLMs, the authors are advised to further explain what new technical challenges position selection introduces compared with ordinary token sampling, and what the independent methodological contribution of PACE is relative to existing Top-H mechanisms.

W3. The current experiments still have limitations in evaluating practical inference efficiency and generation diversity. The main experiments adopt a decoding setting in which only one token is determined per step, so generating a sequence of length 256 requires 256 model forward passes. The appendix adopts a confidence-thresholding method that dynamically commits multiple tokens and already achieves comparable accuracy at a notably lower NFE; therefore, the accuracy–compute advantage of PACE still needs further verification. In addition, in the supplementary experiments with Ttok=0.6T_{\mathrm{tok}}=0.6, Tempered Selection outperforms PACE under larger sampling budgets. For example, on LLaDA/MBPP, their Pass@64 values are 73.3% and 60.0%, respectively. This suggests that PACE may focus more on improving single-sample accuracy and may have some impact on sampling coverage. The authors are advised to further discuss this trade-off and to supplement parallel decoding and more complete Pass@k–compute comparisons.

References

[1] Yang, K., et al. (2026). Improving Sampling for Masked Diffusion Models via Information Gain. ICML 2026.

[2] Jazbec, M., et al. (2026). Learning Unmasking Policies for Diffusion Language Models. ICML 2026.

[3] Yeom, J., et al. (2026). Answer First, Reason Later: When Commitment Order Costs Accuracy in Diffusion Language Models. arXiv:2608.05687.

[4] Ben-Hamu, H., et al. (2025). Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking. NeurIPS 2025.

[5] Baghaei Potraghloo, E., Azizi, S., Kundu, S., & Pedram, M. (2025). Top-H Decoding: Adapting the Creativity and Coherence with Bounded Entropy in Text Generation. NeurIPS 2025.

Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.