acceptodds
Withdrawn from ICLR 2027

Better Selector, Better Decoder? Evaluating Token Selection in Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parallel diffusion decoding can accelerate text generation by filling several masked positions per forward pass. Its accuracy depends on both which positions are selected and how many are filled at once. Adaptive decoders couple these decisions, because earlier selections shape the context used to determine later counts. Changing a selector can therefore also change its schedule of per-step counts, while fixing the schedule leaves unclear whether the measured advantage generalizes. We record the per-step counts that adaptive decoders choose and let different selectors fill exactly these counts. As a control, we spread the same positions evenly over the same number of forward passes. On GSM8K in both LLaDA and Dream, DAPD’s selector is more accurate than confidence when both use DAPD’s counts. By contrast, confidence is more accurate when both use counts from a confidence-threshold decoder. We also let a different selector fill the first half of the output positions. The original decoder then completes the answer more accurately by choosing new counts than by reusing its recorded counts. These findings motivate the Decision-Value Protocol. It recommends evaluating a selector together with the rule that sets its counts, choosing counts from the current text, and choosing complete decoders by accuracy and measured latency. In a later study on ASDiv, following the protocol selects a decoder that is 9.38 points more accurate than the one a selector comparison favors.

Primary area:
generative models
Keywords:
diffusion language models, parallel decoding, decoding schedules

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Lg42OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the evaluation of token selection in DLM. It identifies that prior adaptive decoders determines which token to commit and how many to commit at each decoding step, which complicates the comparison of different selectors. The authors conduct controlled experiments by fixing and balancing the number of committed tokens across different selectors and find that their rankings can change under different decoding schedules. Based on these observations, the paper proposes a Decision-Value protocal, which suggests evaluating selectors w/ their count selection rules.

Critical strengths
  1. The paper controls token selection under varying decoding schedules. This helps illustrate how the evaluation results can be affected by the choice of decoding budget etc.
Critical weaknesses
  1. The main finding is that changing the number of committed tokens across decoding steps can change the performance ranking of different selectors. However, is this somewhat expected and known? For example, in Fast-dLLM, changing the confidence threshold affects how many tokens are committed at each step, and can naturally lead to different decoding trajectories and accuracy. This seems similar to the effect of hyperparameter tuning in existing decoding methods. I think this effect has been already considered in all prior work that people will always compare performance and efficiency together. The proposed DVP mainly summarizes some reasonable evaluation practices, and I don't see much new insight beyond these observations.
  2. The paper lacks understanding of why the observed ranking reversals happen. Most formulations are definitions of the experimental setup and comparison metrics. The paper shows that changing schedules can affect selector performance, but lacks what properties of the selector lead to these reversals. I would expect more insights beyond reporting empirical differences.
  3. Im concerned about the practical significance of the findings. The paper shows that changing the decoding schedule can change the ranking of token selectors. However, in practice, we usually compare complete decoding methods, each with its own schedule and hyperparameters. It is unclear how often the reported ranking changes would actually lead to selecting a worse decoder. In Section 5.2, the authors do not find such cases on GSM8K, and the cases on ASDiv involve the additional entropy-based selector. This makes me question how serious this issue is for existing decoding methods.
Questions for authors
  1. What is the main technical contribution of DVP beyond standard evaluation practices? Comparing selectors under matched budgets and evaluating complete decoders using accuracy and latency seem to be quite natural.
  2. How often do the observed ranking reversals actually lead to choosing a different complete decoder among existing published methods? The strongest examples in Section 5.2 appear to involve the additional entropy-based decoder.
  3. Since the number of committed tokens can already be controlled by hyperparameters in existing methods, what new insights does DVP provide beyond showing that changing these configurations can affect selector rankings?
Reviewer_ybNdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how to compare and choose decoding strategies for masked diffusion language models. It focuses on two coupled decisions: the count rule (how many masked positions to fill at each step) and the selector (which positions to fill). The authors report three empirical findings: (1) selector rankings can reverse depending on which decoder supplied the counts; (2) after the generated text changes, choosing new counts improves continuation over reusing recorded counts; and (3) a selector’s advantage under shared counts need not carry over to complete decoders. Based on these findings, the authors propose the Decision-Value Protocol (DVP), a three-practice protocol for evaluating selectors and choosing complete decoders.

Critical strengths
  1. The controlled four-run design—recorded counts versus balanced counts under the same forward-pass budget—is a reasonable attempt to isolate count-source effects.

  2. The paper evaluates two model families and several decoders, and it includes multiple sensitivity checks.

  3. The authors are relatively transparent about several limitations, including differing budgets across count sources and the absence of a comparison between DVP and alternative decision rules.

Critical weaknesses
  1. Limited novelty. The paper proposes no new decoding algorithm or theory. DVP is essentially a checklist—compare under multiple count sources, recompute counts from current text, and compare complete decoders by accuracy and latency. The contribution is closer to a careful analysis note than an ICLR research paper.

  2. The core reversal evidence is confounded. The main DAPD-vs-C95 reversal uses very different forward-pass budgets, and balancing substantially shrinks the gaps. The “within 10% budget” subset is not random and does not isolate the count source. Under C95’s counts, DAPD also fills many positions outside its own proposal (31% on LLaDA, 15% on Dream), so the comparison measures an adapted selector, not the complete DAPD decoder.

  3. The four-selector study does not robustly support the average-lead claim. On Dream GSM8K, the breadth criterion fails: entropy has a negative own-selector lead, and only two of four leads shrink under balancing. Without entropy, Dream’s shrinkage is only 0.52 points with interval [ − 3.13 , 4.30 ] [−3.13,4.30]. After correction over 165 main tests, Dream’s A_R − A_ U

interval includes zero (Table 15). The ASDiv replication uses only 256 problems.

  1. DVP’s motivation is not strong Table 21 reports only two reversed choices out of 240 pairwise comparisons, both on Dream ASDiv and both involving the entropy rule. The entropy rule is an ad hoc threshold decoder added for the four-selector study, not a released decoder. Among confidence, DAPD, and KLASS, there are no reversed choices. DVP therefore addresses a problem that appears almost exclusively for a non-standard decoder.

  2. Experimental scope is too narrow. The four-selector study is confined to 256 GSM8K and 256 ASDiv problems, two model families, batch size one, 256 generated positions, and arithmetic tasks. There is no end-to-end comparison with strong adaptive decoders such as Fast-dLLM on full benchmarks, no throughput/memory analysis, and no realistic serving setting. Generality beyond the tested implementations and block structures is not established.

  3. Presentation quality is poor. The provided manuscript has a numeric dump on the first page, a truncated Figure 1 caption, mangled tables, and repetitive prose. Figure 1 is essential but unreadable in the provided file, and Eq. (1) is under-explained. This makes verification difficult and falls below ICLR presentation standards

Questions for authors

NAN

Reviewer_Ln3dOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the coupling between token selection and the schedule of per-step counts in parallel diffusion decoding, and revisits how token selectors are evaluated. Through controlled experiments, the authors report three findings: (1) selector rankings depend on the count source; (2) counts chosen anew beat recorded counts after the text changes; (3) a selector’s advantage under the same counts need not carry over to complete decoders. Based on these findings, this paper proposes the Decision-Value Protocol (DVP), which recommends evaluating selectors together with their count rules, choosing counts from the current context, and comparing complete decoders using accuracy and latency.

Critical strengths
  1. This study offers an insightful examination of the coupling between token selection and per-step commitment counts in parallel diffusion decoding. Its recorded-count replay experiments and balanced controls, which preserve per-block forward-pass budgets, provide concrete evidence that selector rankings depend on the count schedule. The analyses of block structure, EOS handling, and measured latency further offer practical guidance for interpreting component-level improvements and evaluating complete decoders.

  2. The clear organization and presentation make the paper easy to follow.

Critical weaknesses
  1. The first DVP recommendation provides useful diagnostics of selector–count-rule interactions. However, it remains unclear how comparisons across count sources and balanced schedules translate into actionable guidance for evaluating or designing selectors. Beyond establishing schedule dependence, the practical implications of these analyses are insufficiently specified.

  2. The practical applicability of the second DVP recommendation is insufficiently established. Native adaptive decoders that already determine counts from the current state naturally satisfy this recommendation. The paper does not clearly identify realistic scenarios where reusing counts from another trajectory is necessary or common, leaving unclear when this recommendation would change existing practice.

Questions for authors

Please see Weaknesses 1 and 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.