Answer First, Reason Later: When Commitment Order Costs Accuracy in Diffusion Language Models
Abstract
Masked diffusion language models write a response by filling a fixed-length canvas of masked positions in any order, so the final answer can be fixed before the reasoning printed above it exists. It is not known when this freedom of order costs accuracy, or whether the early answer is the reason. We record when every position of a response commits and compare unrestricted confidence-based decoding with frontier gating, which lets only positions near the left-most unfinished one commit, on GSM8K and MATH-500 with LLaDA and Dream models. In LLaDA models the cost lies entirely in end-anchored trajectories, in which the end of the response, including the answer, commits before its middle: gating adds +32.0 points of accuracy on these GSM8K responses and +10.7 on MATH-500, and nothing elsewhere, where unrestricted decoding already proceeds from left to right. Every answer-first response is end-anchored. End-anchoring can be predicted before decoding from the share of the canvas that the first forward pass assigns to end-of-sequence tokens (AUC 0.83 to 0.94), and it becomes more frequent with step-by-step prompting and harder problems and less frequent with a longer canvas, which accounts for the prompt and canvas dependence of the gain. Two pre-specified interventions test this account. Holding only the last 32 positions of the response back until step 448 of 512 removes end-anchoring and recovers the whole gain of gating on end-anchored GSM8K questions (+31.4 against +32.0 points), whereas holding an equally long span in the middle changes nothing; conversely, widening the gate until the end of the canvas becomes available brings end-anchoring back and removes the gain (-8.1 points). Delaying only the answer leaves 90% of end-anchored responses end-anchored and recovers little. Evaluations of diffusion language models should report canvas length, termination rule, and commitment order alongside accuracy.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.