- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies early commitment selection in masked diffusion LMs (mainly LLaDA, with some Dream results). Using shared-state branching at the first non-padding commitment, it reports that changing that single action can alter a substantial fraction of final answers on a diagnostic set, and highlights frequent early EOT commitments on short RAG canvases.
The proposed EndFirst method subtracts a uniform offset from non-EOS logits at that first step only, then resumes the base decoder with no extra forward pass. Headline numbers are short-answer RAG QA EM lifts on NQ / HotpotQA / TriviaQA / T-REx under a small canvas and pass budget.
Critical strengths
Shared-state branching is a sensible way to isolate commitment choice from prediction quality, and the EOT/EOS token-identity controls give a concrete failure mode on LLaDA’s short RAG setup.
Critical weaknesses
-
Overstated Generality and Tokenizer-Specific Scope: The empirical narrative is narrowly tied to short-answer RAG tasks evaluated on a fixed 32-token canvas using LLaDA’s distinct EOT and EOS tokenization scheme. Gains fail to transfer to models with different commitment schedules, with Dream showing negligible movement in Table 1 (approximately 0 to 2 EM points). Broader evaluation regimes such as open-ended generation, multi-sentence completion, and complex reasoning are entirely absent. Consequently, the claim of a foundational commitment bottleneck appears primarily to reflect an artifact of LLaDA’s premature terminal-token emission on short retrieval canvases.
-
Marginal Algorithmic Novelty: The proposed method, EndFirst, reduces to a single static logit offset modulated by a scalar hyperparameter . While the underlying diagnostic analysis is insightful, the operational technique is incremental and overlaps significantly with existing deferred-commitment and boundary-control heuristics. As an algorithmic contribution to a top-tier venue, a tuned scalar penalty represents relatively thin methodological depth.
Questions for authors
-
Outside short-answer RAG, does first-commitment lock-in still appear (e.g., open-ended instruction following, longer canvases)? If EOT-first rates collapse, how should EndFirst be reinterpreted?
-
Can EndFirst be stated for models that use a single terminal/pad token (or Dream’s padding regime) without relying on EOT≠EOS?
-
Please revise the abstract/intro to state the LLaDA-EOT/RAG scope as a primary limitation rather than implying a general commitment-selection advance.
-
Have you tried combining EndFirst with remasking / revocable decoding, or learning / instead of a fixed schedule?
Thank you for the thorough experiments. While the intervention remains simple and strictly bounded to concise-answer tasks (as confirmed on GSM8K), the upfront scope revisions make the claims fair and well-supported, so I am raising my score by 1 point.
Good luck!