- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines the question whether any-order decoding reduces mathematical reasoning performance in MDMs. The authors identify "end-anchored" trajectories, where they notice that the response commits before its reasoning. The proposed method restricts decoding in the end window and notice improved results on GSM8K and MATH500. Further, they study the interplay of this effect over different decoding windows and with EOS prediction.
Critical strengths
- The experiments provide evidence for two datasets that end-anchoring reduces performance. The authors distinguish end-anchoring from answer-first commitment and canvas filling.
- The diagnostic tail-delay intervention is insightful and provides substantiation for the argument, since it distinguishes between blockwise diffusion and keeping the tail-end masked.
Critical weaknesses
- The widely supported semi-autoregressive block-diffusion is not evaluated thoroughly. Blockwise diffusion, as supported in the code and paper of LLaDA-8B, shows superior performance in Table 7. Table 2 omits this on the full GSM8k evaluation and Table 5-6 do not show it. This is critical, as this established method exceeds the author's performance on their 247-sample MATH500 subset. The authors present their method as a diagnostic rather than a new methodology what practical benefit one gains as blockwise decoding is an established way to adress their problem.
- The method is not tested against similar approaches in the literature in terms of performance and computational cost. Holding a large window while restricting the end may improve performance beyond unrestricted decoding, but has downsides in terms of computational cost. For example, Fast-dLLM, sliding-window decoding (AdaBlock-dLLM) provide both the gain in performance and a lower computational footprint. These approaches are widely used, and therefore undermine the argument that the stated performance gain over unrestricted decoding is significant.
- The "Gain lies entirely in end-anchored trajectories" seems true by construction. The authors note a position-order correlation of 0.995, finding no gains in restricting decoding at other positions seems trivial.
- "AUC of 0.83-0.94" combined with "87.5%" accuracy needs context; on GSM8K the majority class baseline is 81.9%, so 87.5% is 5.6 pp above the trivial solution. On MATH at L=512, accuracy is 85.2% against a base rate of 84.8%. This is, however, still being conveyed as a contribution in the abstract.
Questions for authors
- Why is semi-AR omitted from the full GSM8k and MATH500 comparisons?
- Do you have results for established approaches such as FastdLLM or AdaBlock-dLLM? This would improve my diagnosis of how generic the contribution is.
- How do the diagnosis and method apply to long-form reasoning? When the window is either very large or of unknown size.
- Figure 8 describes the difficulty groups for GSM8k, using equal thirds and the reported 6%, 20% and 41%, this implies about 22.3% overall. The main result uses 194/1069=18.1%, please provide the group sizes and counts, since this implies about 239 end-anchored responses instead of 194.
- Could you give context to question (5) and explain why the detector still forms a substantial contribution?