- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper attributes much of arbitrary-order decoding's diversity loss to LCR's proposal filtering rather than confidence-first ordering. Toy analyses and LLaDA diagnostics support this explanation. TPP restores rollout coverage, while entropy-guided initialization further improves coverage and policy optimization.
Critical strengths
- The sampler distinction is precise and consequential. Exponential suppression in the toy models is supported by proposal-versus-commitment diagnostics, correcting an overly broad interpretation of the flexibility trap (Sections 3-5).
- The explanation leads to a simple intervention: TPP recovers coverage and EGI improves it further. The AR temperature control and gains across three RL objectives strengthen the empirical case (Figure 3, Table 4, Appendix C.5).
Critical weaknesses
- Evaluation uses one-token-per-step decoding and two related LLaDA checkpoints. Whether the mechanism and EGI gains extend to practical parallel decoding or another model family remains untested.
- RL results select the best of ten test-set checkpoints, without reporting variability across training seeds (Appendix D). This weakens confidence in the smaller downstream gains.
Questions for authors
- Do TPP and EGI retain their coverage advantage when multiple tokens are committed per step?
- Do the RL gains persist across seeds using final-budget or validation-selected checkpoints?
Rebuttal
We thank the reviewer for the constructive feedback and helpful suggestions for strengthening the empirical evaluation.
W1. Generalization beyond the LLaDA family
We evaluated Dream 7B Instruct [1] on GSM8K, MATH-500, HumanEval, and MBPP with 32 rollouts per prompt; GSM8K uses a random 500-problem subset. Across all four benchmarks, TPP consistently improves over LCR and recovers rollout coverage close to AR. On the math benchmarks, AR retains a small advantage, which may partly reflect Dream's initialization from the autoregressive Qwen2.5-7B model [1] and the sequential nature of mathematical reasoning. Despite this residual gap, TPP remains close to AR, reproducing the key LCR-TPP distinction beyond the LLaDA family. EGI also remains competitive with AR. Full results are provided in App. D.1 (Fig. 11) of the revised manuscript.
Q1. Coverage under parallel decoding
We evaluated parallel decoding using Fast-dLLM [2] variants on LLaDA-8B-Instruct with a common confidence threshold of 0.9. We denote Fast-dLLM with LCR, TPP, and EGI as FD-LCR, FD-TPP, and FD-EGI, respectively. FD-LCR applies LCR's proposal filtering, FD-TPP selects positions using top-token probabilities before sampling, and FD-EGI applies EGI at the first step followed by FD-TPP.
FD-TPP improves coverage over FD-LCR, and FD-EGI improves it further with only a modest increase in NFE. Thus, the same diversity behavior underlying LCR, TPP, and EGI persists under parallel decoding. Pass@K and Potential@K are percentages, and NFE is the mean number of model evaluations. Full results are provided in Sec. 6.2 (Fig. 4) and App. D.2 (Fig. 12) of the revised manuscript.
W2 & Q2. RL robustness across seeds and checkpoints
We additionally ran JustGRPO with three training seeds per sampler and evaluated the final-budget checkpoints, while also reporting accuracy averaged over all ten evaluation checkpoints. Both summaries report mean ± sample standard deviation across training seeds. EGI achieves higher mean accuracy than AR on all four benchmarks under both summaries, without relying on best-checkpoint selection. All values are percentages. Full results are provided in App. E.1 (Fig. 15 and Tab. 9) of the revised manuscript.
References
[1] Ye et al., Dream 7B: Diffusion Large Language Models, arXiv, 2025.
[2] Wu et al., Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding, ICLR, 2026.