acceptodds
Under review as a conference paper at ICLR 2027

Answer First, Reason Later: When Commitment Order Costs Accuracy in Diffusion Language Models

Abstract

Masked diffusion language models write a response by filling a fixed-length canvas of masked positions in any order, so the final answer can be fixed before the reasoning printed above it exists. It is not known when this freedom of order costs accuracy, or whether the early answer is the reason. We record when every position of a response commits and compare unrestricted confidence-based decoding with frontier gating, which lets only positions near the left-most unfinished one commit, on GSM8K and MATH-500 with LLaDA and Dream models. In LLaDA models the cost lies entirely in end-anchored trajectories, in which the end of the response, including the answer, commits before its middle: gating adds +32.0 points of accuracy on these GSM8K responses and +10.7 on MATH-500, and nothing elsewhere, where unrestricted decoding already proceeds from left to right. Every answer-first response is end-anchored. End-anchoring can be predicted before decoding from the share of the canvas that the first forward pass assigns to end-of-sequence tokens (AUC 0.83 to 0.94), and it becomes more frequent with step-by-step prompting and harder problems and less frequent with a longer canvas, which accounts for the prompt and canvas dependence of the gain. Two pre-specified interventions test this account. Holding only the last 32 positions of the response back until step 448 of 512 removes end-anchoring and recovers the whole gain of gating on end-anchored GSM8K questions (+31.4 against +32.0 points), whereas holding an equally long span in the middle changes nothing; conversely, widening the gate until the end of the canvas becomes available brings end-anchoring back and removes the gain (-8.1 points). Delaying only the answer leaves 90% of end-anchored responses end-anchored and recovers little. Evaluations of diffusion language models should report canvas length, termination rule, and commitment order alongside accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, masked diffusion, decoding order, commitment order, chain-of-thought reasoning, LLaDA, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uU6jOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the question whether any-order decoding reduces mathematical reasoning performance in MDMs. The authors identify "end-anchored" trajectories, where they notice that the response commits before its reasoning. The proposed method restricts decoding in the end window and notice improved results on GSM8K and MATH500. Further, they study the interplay of this effect over different decoding windows and with EOS prediction.

Critical strengths
  1. The experiments provide evidence for two datasets that end-anchoring reduces performance. The authors distinguish end-anchoring from answer-first commitment and canvas filling.
  2. The diagnostic tail-delay intervention is insightful and provides substantiation for the argument, since it distinguishes between blockwise diffusion and keeping the tail-end masked.
Critical weaknesses
  1. The widely supported semi-autoregressive block-diffusion is not evaluated thoroughly. Blockwise diffusion, as supported in the code and paper of LLaDA-8B, shows superior performance in Table 7. Table 2 omits this on the full GSM8k evaluation and Table 5-6 do not show it. This is critical, as this established method exceeds the author's performance on their 247-sample MATH500 subset. The authors present their method as a diagnostic rather than a new methodology what practical benefit one gains as blockwise decoding is an established way to adress their problem.
  2. The method is not tested against similar approaches in the literature in terms of performance and computational cost. Holding a large window while restricting the end may improve performance beyond unrestricted decoding, but has downsides in terms of computational cost. For example, Fast-dLLM, sliding-window decoding (AdaBlock-dLLM) provide both the gain in performance and a lower computational footprint. These approaches are widely used, and therefore undermine the argument that the stated performance gain over unrestricted decoding is significant.
  3. The "Gain lies entirely in end-anchored trajectories" seems true by construction. The authors note a position-order correlation of 0.995, finding no gains in restricting decoding at other positions seems trivial.
  4. "AUC of 0.83-0.94" combined with "87.5%" accuracy needs context; on GSM8K the majority class baseline is 81.9%, so 87.5% is 5.6 pp above the trivial solution. On MATH at L=512, accuracy is 85.2% against a base rate of 84.8%. This is, however, still being conveyed as a contribution in the abstract.
Questions for authors
  1. Why is semi-AR omitted from the full GSM8k and MATH500 comparisons?
  2. Do you have results for established approaches such as FastdLLM or AdaBlock-dLLM? This would improve my diagnosis of how generic the contribution is.
  3. How do the diagnosis and method apply to long-form reasoning? When the window is either very large or of unknown size.
  4. Figure 8 describes the difficulty groups for GSM8k, using equal thirds and the reported 6%, 20% and 41%, this implies about 22.3% overall. The main result uses 194/1069=18.1%, please provide the group sizes and counts, since this implies about 239 end-anchored responses instead of 194.
  5. Could you give context to question (5) and explain why the detector still forms a substantial contribution?
Reviewer_axk9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates end-anchoring, a phenomenon in which the end of a response is committed before the intermediate reasoning during unrestricted confidence-based decoding of masked diffusion LMs. Using LLaDA, the authors show that the accuracy gains from frontier gating arise almost entirely from avoiding end-anchoring, and that simply delaying commitment of the response tail yields comparable gains. They further show that the prevalence of end-anchoring varies with canvas length, prompting strategy, and problem difficulty. Based on these findings, the authors attribute the accuracy loss under unrestricted decoding to end-anchoring and recommend reporting canvas length, termination rules, and commitment order alongside accuracy.

Critical strengths

The finding that frontier gating's accuracy gains are concentrated on problems exhibiting end-anchoring under unrestricted decoding, while other trajectories already proceed almost left-to-right, is particularly interesting. Moreover, the observation that delaying only the final 32 positions of the response achieves nearly the same gains as frontier gating provides compelling evidence that avoiding end-anchoring may explain the benefits of order-constrained decoding methods, including AR and semi-AR generation.

Critical weaknesses

Despite the paper's central empirical finding, the mechanistic analysis of why end-anchoring occurs and why it degrades accuracy remains limited. While Table 4 suggests that answer predictions are less stable when committed early, it remains unclear why such premature commitments occur and how left-to-right generation resolves the resulting errors. Moreover, the intuition that reasoning is backfilled before an already fixed ending does not fully explain the substantial accuracy gains observed even for end-anchored trajectories in which the answer itself is not committed early.

Questions for authors

Can the authors explain why end-anchoring occurs and the mechanism through which it degrades reasoning accuracy? In particular, why is an answer more likely to be incorrect when committed before the reasoning, and when the answer itself is not committed early, how does prematurely fixing the end of the response constrain the preceding reasoning?

Reviewer_R6PEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Diffusion LLMs like LLaDA can generate tokens in any order. Normally they decode with a confidence measure, committing positions from high confidence to low confidence. This paper studies two decoding methods. The first is unrestricted decoding, which is based only on confidence. The second is gating, where only the 16 positions after the leftmost unfinished position can be decoded. This is similar to the semi-autoregressive (blockwise) decoding that LLaDA originally had. The paper is not trying to propose this decoding. It is trying to analyze why unrestricted decoding loses accuracy on math tasks like GSM8K. It finds that the accuracy loss is in the end-anchored trajectories, where the end of the response, including the answer, is committed before the middle part of the reasoning. These account for about 18% of GSM8K responses, and gating or semi-autoregressive block decoding improves their accuracy a lot. The rest are already decoded from left to right, so gating makes no difference there. The paper analyzes two LLaDA models and one Dream model, and finds that this pattern mostly occurs in the LLaDA series. It also finds that the shorter the canvas, the more end-anchored trajectories there are. It also looks at problem difficulty in this analysis.

Critical strengths
  1. It tries to analyze why unrestricted decoding sometimes performs worse than semi-AR decoding in diffusion LLMs like the LLaDA models. Prior work has not fully answered this, so the paper sheds some light on it.
  2. The paper shows that fixing the end of the response early hurts the intermediate reasoning, and that disclosing the answer first by itself is not the cause. It does ablations such as holding the last 32 positions back until late in decoding (step 448 of 512), and accuracy improves to the level of gating. Delaying only the answer helps little.
  3. It calls on future work to report canvas length, termination rule, and generation order alongside accuracy, which are good practices that should be adopted.
Critical weaknesses
  1. Semi-AR decoding is already the recommended decoder released with LLaDA. Also, your result mostly applies to LLaDA models, and Dream does not have this problem. So the impact is limited.
  2. Also I think it is limited to math tasks, so the paper’s impact or insight might be limited. For example, after reading this paper, the practical suggestion seems to be decoding from left to right or in a block-autoregressive way, but this already seems standard in the field. So I am not sure what useful takeaway this paper brings.
  3. Overall, I think this is an analysis paper, but it does not give a very deep insight into why the model produces end-anchored trajectories on some problems. The paper shows they are more frequent on harder problems and can be predicted from the step-0 EOS share, but it has not answered fully why the model behaves this way on these 18% of problems.
Questions for authors
  1. When the canvas is longer, end-anchoring decreases, but then more compute is wasted on the canvas. How do you think about this trade-off?
  2. why do you think this happens to LLaDA series but not Dream models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.