- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper shows that self-consistency, which generates multiple answers and takes a majority vote, can be ineffective for dLLMs because the samples are often nearly identical. As a result, the authors propose Order-Space Exploration (OSE), which creates diversity by perturbing the order in which tokens are revealed rather than changing how tokens are sampled, and then uses majority voting to select answers. They provide a theoretical analysis of this approach and evaluate it on Dream and LLaDA across six tasks. OSE increases answer diversity by 2–5× without hurting single-sample accuracy, outperforms standard self-consistency, and achieves competitive results at lower cost than alternative decoding ensembles. They additionally show that using OSE samples for GRPO training improves learning.
Critical strengths
- Novel test-time scaling approach: using the unmasking order as a source of diversity is specific to dLLMs and the proposed method requires only one hyperparameter, no verifier, and no retraining. Each sample also costs the same as one from the default sampler.
- Useful connection to RL training: the authors show that the same order-perturbed samples can serve as GRPO rollouts. This results in improvements in the trained policy (+7.6 on MATH500 and +8.5 on Countdown).
- Theoretical motivation: the core results seem correct and they seem to help explain why perturbing the order should be cheaper than raising the temperature.
- Well-executed experiments: within its scope, the evaluation is thorough, and it covers two models and six tasks. It also includes results over three seeds and CIs, temperature sweeps, a temperature-matched HEX comparison, ablations and wall-clock costs.
Critical weaknesses
- Related work and citations: the introduction states that confidence-based ranking is the most widely adopted approach without citing the relevant methods, and related work is discussed only in Appendix. Confidence-based decoding methods and prior work on sampler equivalence could be better referenced in the introduction. In addition, related test-time scaling methods are also not discussed or compared empirically in the main text.
- Limited evidence for the decoder analysis: the decoder analysis focuses on Dream’s default settings, so it is unclear whether OSE provides general benefits or mainly fixes a degenerate sampler configuration. The results on LLaDA suggest the effect may depend on the model.
- Theoretical claims could be better supported: the interpretation of OSE’s cost bound is somewhat unclear, and the bound would benefit from empirical validation. Some claims about on-policy sampling also require further justification.
- Dense presentation and inconsistent notation: while the paper is generally easy to follow, sometimes the main text feels a bit too dense. The notation and terminology used is sometimes inconsistent (for example, T denotes temperature and the number of steps, and OSE is called both "Order-Space Exploration" and "Order-Ensemble Decoding"), which makes the claims harder to verify.
- Limitations are not reported: they authors do not openly report limitations which include issues about voting, and model-dependent gains.
Questions for authors
-
Would it be possible to separate the effects of token sampling and unmasking-order perturbations? This would help clarify whether the gains come specifically from the different orders.
-
Does OSE remain effective across different samplers, decoding configurations, and dLLMs, including cases where the default sampler is already stochastic?
-
Would it be possible to empirically validate the proposed cost bound? Could the authors also clarify the impact of the assumptions, particularly the dependence of the ranking on the model parameters?
-
How does OSE compare with alternative test-time scaling and diversity methods under matched compute budgets? Similarly, how does Order-Diverse GRPO compare with existing approaches to generating diverse training rollouts?
-
Figure 2 is hard to parse, please consider improving its clarity. More generally, the main text is quite dense, and some notation is used inconsistently (as mentioned in Weaknesses), so a revision for clarity would strengthen the manuscript.
4 comments
Q4. Matched compute; diverse RL rollouts. Test time: an OSE draw costs the same forward passes as a default draw, so N OSE draws and N default draws have equal compute; HEX is 3–3.5× slower per draw in our implementation.
RL: with gold rewards, policies trained on OSE rollouts beat those trained on default rollouts under the default decoder: +8.5 on Countdown [+3.6, +13.4], +7.6 on MATH500 [+3.8, +11.4], +2.7 on GSM8K (Sec. 5.5). New runs (App. T) add MBPP+ and Zebra, a block-32 rollout baseline, and label-free rewards (plurality vote of the group, no labels). Δ = accuracy minus base, default decoder:
| rollouts | Countdown label-free | MATH500 label-free | MBPP+ gold | MBPP+ label-free | Zebra gold | Zebra label-free |
|---|---|---|---|---|---|---|
| default (λ = 0) | −0.9 | −2.6 | 0.0 | +1.7 | 0.0 | 0.0 |
| block-32 | +6.3 | −1.9 | +2.7 | +4.0 | −2.2 | +0.3 |
| OSE λ = 1 | +12.1 | +0.8 | +5.8 | +5.2 | +11.2 | +14.2 |
Default rollouts almost never give mixed groups (1–9% with gold rewards, 0.0–0.3% label-free), so label-free RL cannot start. OSE gains most in every column and, without labels, comes close to its gold-reward gain, because its vote is a far better teacher than its single draw (teacher lift +10.8 to +25.7, against +0.1 to +4.3 for block-32). Under DAPO at a matched budget (9,600 rollouts), the default order makes zero updates; OSE makes 141 and moves Countdown from 15.2 to 18.8. No importance weights are needed (Prop. 4.6).
W1. Related work and citations. The introduction now cites confidence-based decoders, sampler-equivalence results and dLLM test-time scaling methods (HEX, OTS, prefix-denoising and temporal self-consistency). Sec. 5.4 compares OSE with HEX and a tuned block decoder.
W4 / Q5. Notation and clarity. We now use K for the number of steps and T only for temperature, one name (OSE) throughout.
W5. Limitations. A new Limitations paragraph states that OSE helps most when the default sampler is near-deterministic, as deployed (Dream, LLaDA, Stable-DiffCoder, all at λ = 1); with a stochastic, near-ceiling default (LLaDA 2.0-mini), OSE and temperature are substitutes and a smaller λ is needed; the vote leaves a gap to pass@N; and the cost bound is qualitative.
We hope these answers address the reviewer's questions and would be glad to clarify further.
Response to authors
I thank the authors for the detailed response and new experiments, which address most of my concerns. Just one concern remains: the decoupled experiment (answer to Q1) shows that the collapse comes from temperature in the ranking, and that token temperature with a protected ranking is competitive with OSE on short-answer tasks. This is in line with W2, as the motivation relies on a specific sampler configuration. I encourage the authors to revise the introduction to better clarify this and OSE's role. I thank the authors for their effort and will increase my score.
We thank the reviewer for the constructive discussion and for raising the score. We agree and have revised the introduction accordingly (in blue). It now states that the collapse at high temperature is a property of the default sampler, where the same temperature also ranks the positions, rather than of dLLMs in general, and that with a protected ranking (App. P, Q) token temperature is competitive on short-answer tasks, at the price of per-task tuning and a lower single draw on long generation. It also clarifies OSE’s role. The abstract and conclusion are scoped in the same way. The revised PDF is uploaded.
We thank the reviewer for the careful review. We've addressed the concerns below.
Q1. Separate the effect of token sampling from the order. We built a decoupled sampler that computes the ranking at the default
T_rank = 0.1but draws tokens atT_tok. WithT_rankfixed, the unmasking order remains entirely unchanged at everyT_tok, meaning any performance shifts come exclusively from the token draw (a full study ofT_tokfrom 0.1 to 1.0 in App.) Below is thevote@8on Dream:Three findings. (i) The collapse comes from temperature in the ranking: coupled T = 0.7 writes 8–23 tokens, decoupled 170–380, as at
T = 0.1. (ii) With the ranking protected, token temperature is a strong baseline on short answers. (iii) But it costs single-draw accuracy and needs tuning. On MATH500 the single draw is 23.5 at T_tok = 0.7 and 4.9 at 1.0, against 38.7 for OSE (base 35.6), and the bestT_tokvaries by task (0.3 to 1.0), whileλ = 1is fixed. Adding OSE on top ofT_tokdoes not help, so the two are substitutes (App. Q); our order ablation (App. S) also shows that local perturbations help and global ones collapse.Q2. Other samplers and dLLMs, including stochastic default samplers. (a) Block decoding with raised temperature: we ran the full Dream grid (
block 32/64 × T, six tasks; App. P). With the best block size and temperature for each task, block decoding lands within −0.7 to +3.0 vote points of OSE on five of six tasks (+8.5 on Countdown, where blocks also fix the answer format), but it needs that per-task search, and a highTbreaks long generation (MATH500 atT = 0.9: single draw 19.1, against 38.7 for OSE). (b) Two new dLLMs, withλ = 1as everywhere else unless stated (App. R):vote@16 = pass@1(MBPP 63.8). Its temperature is decoupled by construction, so the temperature arm is the decoupled baseline of Q1: | Stable-DiffCoder | MBPP pass@1 | MBPP vote@16 | MBPP+ pass@1 | MBPP+ vote@16 | HumanEval pass@1 | HumanEval pass@16 (oracle) | |---|---|---|---|---|---|---| | default | 63.8 | 63.8 | 69.5 | 67.0 | 75.0 | 75.0 | | raised temperature | 63.0 | 70.4 | 70.1 | 74.2 | 75.8 | 91.3 | | OSEλ = 1| 65.1 | 70.6 | 71.8 | 74.1 | 76.8 | 91.0 |OSE matches the decoupled temperature's vote gain (MBPP +6.8 vs +6.6; MBPP+ +7.1 vs +7.2) with fewer distinct answers (1.48 vs 1.85), and is the only method that raises pass@1 everywhere.
λ = 0.5, OSE raises the vote on GSM8K, MATH500 and Countdown (+0.5 to +1.3) and keeps the single draw within 0.3 points on GSM8K and MATH500, where temperature costs 4.5 points on MATH500. Gains are small because the default is already stochastic and near the ceiling (vote ≈ 93% on GSM8K, ≈ 98% on Countdown), leaving little for any method.Q3. Cost bound; assumptions. We agree the bound is loose (an 8–1–1 split already exceeds 1) and now present Prop. 4.4 as a qualitative statement. The direct check is the single draw, which moves by only −1.0 to +2.3 points at
λ = 1on all 12 settings (Table 12). Ranking depends onθ: Prop. 4.6 treats the order as part of the environment, as d1 does with its confidence ranking, and so leaves out the score term of the order. Every RL gain is measured with the default decoder, not the training sampler.