- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Motivated by the observations that BDLMs are biased towards AR generation, which is both shown empirically and justified theoretically in the paper, the authors propose a token sampling method for block DLMs where the decoding order is biased towards autoregressive generation order. For a selection criteria of choice (e.g. confidence threshold) the proposed sampling technique, called PARD, selects consecutive tokens for which the selection is fulfilled until the first token that does not meet the criteria. Experiments show promising results for the proposed technique.
Critical strengths
The critical strength is the simplicity of the proposed technique, which does not seem to introduce any new hyperparameters, nor requires specific post-training, which make it very easily deployable in practice.
Critical weaknesses
Authors themselves make an observation that the proposed sampling technique has largest edge on coding benchmarks (l. 442). The entire motivation for the proposed method rests on the results obtained on the coding benchmark and MDMs are never benchmarked after section 3.1. Hence, I am not 100% convinced that AR order bias in BDLMs is as general as the paper claims. From Table 2, we see that AR outperforms other sequential methods most strongly on code, with code also being the domain where PARD has largest edge against other baselines. On the other benchmarks, PARAD does outperforming the rest in majority of cases, but the gains are very small in many cases (same for AR) and can potentially be attributed to noice effects?
Questions for authors
- I would like to see whether and to which extent the AR bias is present on other domains than code (e.g. add other domains to section 3.1), this could better justify the more general claim the paper makes about AR bias in BDLMs
- the paper is missing important baselines of learned unmasking policies like e.g. Jazbec et. al (ICML Oral) ,"Learning unmasking policies for diffusion language models". Also I wonder whether and how the observed AR bias can be used to improve these policies?
- Table 2 average seem to be biased to coding domains, which are the ones on which proposed methods performs the best
- Table 2 presents performance numbers with standard hyperparamters applied. Could authors also present throughout numbers at those hyperparameters for methods other than just AR and PARD (Table 3)? I would like to see numbers for e.g. confidence parallel and possibly some other parallel methods there as well (I would expect confidence parallel to have similar throughout given that it has similar excepted tokens pattern from Fig. 15).
- regarding technical details: could authors also share which inference backend is being used for the evail as well as what batch size? I understand that the default batch size for DLM evals is usually 1, so I am assuming batch size here is also 1.
- DMax (Chen, Zigeng, et al. "Dmax: Aggressive parallel decoding for dllms.") relies on a similar heuristic of keeping the decoded regions contiguous left-to-right based on a confidence threshold. I would appreciate a discussion on how the proposed method relates to that of DMax (in its motivation and mechanics)
4 comments
W2. Small gains on non-coding benchmarks and noise effects.
We agree that the gains are larger on coding tasks. Code generation is particularly sensitive to token ordering, as earlier tokens often determine subsequent code structure. Appendix D.3 illustrates this with an example.
However, we would like to emphasize that the benefit of PARD is not limited to code. The table below reports the average performance on the non-coding benchmarks (GSM8K, MATH, IFEval, and BBH):
| Method | Fast-dLLM v2 7B | SDAR 8B | LLaDA2.1-Mini 16B |
|---|---|---|---|
| KLASS | 64.3 | 70.0 | 85.9 |
| EB-Sampler | 63.4 | 71.1 | 85.7 |
| Confidence Parallel | 63.5 | 69.6 | 84.8 |
| APD | 62.1 | 69.4 | -- |
| Hierarchy | 62.3 | 68.3 | 84.9 |
| PARD | 66.0 | 72.3 | 86.2 |
| AR (sequential) | 66.6 | 73.0 | 86.4 |
As in Table 2, APD is not reported for LLaDA2.1-Mini because no small AR verifier is available.
PARD achieves the highest non-coding average among all parallel samplers on all three models. Compared with Confidence Parallel, the default parallel sampler of these BDLMs, PARD improves the non-coding average by 2.5, 2.7, and 1.4 points on Fast-dLLM v2, SDAR, and LLaDA2.1-Mini, respectively.
We used greedy decoding in the main experiments, so the generation process is deterministic. For a fixed model, sampler, and prompt, repeated runs produce essentially identical pass@1 scores. Thus, the reported differences cannot be attributed to stochastic token sampling. In addition, we evaluate every benchmark on its full test set rather than a randomly sampled subset. For BBH on LLaDA2.1-Mini, compute constraints limit us to 8 of the 27 tasks (discussed in Section 5.2), but each of these tasks is still evaluated in full. The reported scores are therefore not affected by noise from sampling test problems. Appendix D.1 (Figures 11–14) further shows that PARD achieves stronger speed-quality trade-offs on GSM8K compared with other parallel samplers. Consistent improvements across models, tasks, and thresholds support the benefit of PARD.
Q2. Comparison with learned unmasking policies (Jazbec et al.).
Thank you for the suggestion on comparison with learned unmasking policies. We compare PARD with the learned unmasking policy of Jazbec et al. on SDAR 8B. We adapt the official code to SDAR and follow the original training configuration. As in the original paper, we train a code policy on the KodCode-RL-10K dataset. The reward weight controls the speed–quality trade-off of the policy. The original paper sweeps rather than fixing a default. Since each requires a separate training run (65.9 GPU-h), we train one policy and compare PARD against it at matched speed below. We use , which the authors commonly use in their experiments that report a single policy rather than a Pareto front. We evaluate the trained unmasking policy on HumanEval and MBPP:
| Method | Training | HE pass@1 | HE tok/s | MBPP pass@1 | MBPP tok/s |
|---|---|---|---|---|---|
| Learned policy () | 65.9 GPU-h | 61.6 | 95.6 | 64.5 | 90.6 |
| PARD (0.9) | None | 81.1 | 77.6 | 77.0 | 72.6 |
| PARD (0.7) | None | 72.6 | 98.8 | 73.8 | 90.7 |
Notably, the learned policy uses domain-specific training data, whereas PARD requires neither training nor task-specific data. Training the policy with GRPO takes 16.5 hours on 4 RTX A6000 GPUs (65.9 GPU-h). With the default confidence threshold of 0.9, PARD achieves substantially higher accuracy than the learned policy, while the learned policy is faster. With a lower threshold of 0.7, PARD is both faster and more accurate than the learned policy. It improves pass@1 by 11.0 and 9.3 points on HumanEval and MBPP, at equal or higher throughput. Thus, PARD outperforms the learned policy in speed and generation quality.
Regarding how the AR bias can improve learned policies, Jazbec et al. use expert steering to help exploration during GRPO training. They add rollouts from a confidence-based sampler to each group as expert trajectories. We suggest using PARD as the expert sampler for BDLMs. Since PARD exploits the AR bias of BDLMs, it may provide higher-reward trajectories that guide the learned policy toward better unmasking orders. We leave this direction for future work.
Q3. Table 2 average biased toward coding.
Thank you for pointing this out. We agree that the original Avg column in Table 2 could be misleading. We have updated Table 2 of the revised paper with a macro-average that weights the six benchmarks equally: we first average Base and Plus within HumanEval and MBPP, and then average the six benchmark scores.
PARD still achieves the highest macro-average among all parallel samplers on all three models. It improves over Confidence Parallel by 3.8, 3.2, and 2.0 points on Fast-dLLM v2, SDAR, and LLaDA2.1-Mini, respectively, compared with 4.5, 3.4, and 2.4 points under the original averages.
Q4. Throughput of other parallel samplers at Table 2 settings.
The table below reports the throughput (tokens/sec) of all parallel samplers under the hyperparameters of Table 2. Numbers in parentheses indicate speedup relative to sequential AR decoding.
Fast-dLLM v2 7B
| Method | HumanEval | MBPP | GSM8K | MATH | IFEval | BBH |
|---|---|---|---|---|---|---|
| AR | 85.8 | 82.7 | 96.1 | 69.6 | 64.8 | 77.6 |
| KLASS | 90.4 (1.05×) | 84.7 (1.02×) | 95.8 (1.00×) | 82.5 (1.19×) | 76.1 (1.17×) | 78.0 (1.01×) |
| EB-Sampler | 131.8 (1.54×) | 113.4 (1.37×) | 183.7 (1.91×) | 151.6 (2.18×) | 111.9 (1.73×) | 119.5 (1.54×) |
| APD | 187.3 (2.18×) | 150.0 (1.81×) | 209.4 (2.18×) | 180.4 (2.59×) | 129.1 (1.99×) | 131.5 (1.69×) |
| Hierarchy | 146.1 (1.70×) | 131.9 (1.59×) | 188.3 (1.96×) | 166.1 (2.39×) | 120.4 (1.86×) | 124.0 (1.60×) |
| Confidence Parallel | 137.5 (1.60×) | 120.1 (1.45×) | 179.8 (1.87×) | 156.8 (2.25×) | 110.6 (1.71×) | 114.4 (1.47×) |
| PARD | 134.9 (1.57×) | 112.5 (1.36×) | 179.9 (1.87×) | 152.0 (2.18×) | 111.2 (1.72×) | 113.5 (1.46×) |
SDAR 8B
| Method | HumanEval | MBPP | GSM8K | MATH | IFEval | BBH |
|---|---|---|---|---|---|---|
| AR | 46.5 | 42.4 | 46.2 | 40.7 | 32.0 | 40.1 |
| KLASS | 50.3 (1.08×) | 49.5 (1.17×) | 57.1 (1.24×) | 60.7 (1.49×) | 42.9 (1.34×) | 40.8 (1.02×) |
| EB-Sampler | 70.1 (1.51×) | 64.5 (1.52×) | 99.2 (2.15×) | 114.1 (2.80×) | 50.3 (1.57×) | 54.8 (1.37×) |
| APD | 120.0 (2.58×) | 96.8 (2.28×) | 133.2 (2.88×) | 141.6 (3.48×) | 70.3 (2.20×) | 81.3 (2.03×) |
| Hierarchy | 87.0 (1.87×) | 88.6 (2.09×) | 124.3 (2.69×) | 137.6 (3.38×) | 68.5 (2.14×) | 66.0 (1.65×) |
| Confidence Parallel | 75.5 (1.62×) | 74.2 (1.75×) | 106.6 (2.31×) | 123.8 (3.04×) | 58.9 (1.84×) | 57.3 (1.43×) |
| PARD | 77.6 (1.67×) | 72.6 (1.71×) | 108.2 (2.34×) | 124.9 (3.07×) | 59.4 (1.86×) | 60.5 (1.51×) |
LLaDA2.1-Mini
| Method | HumanEval | MBPP | GSM8K | MATH | IFEval | BBH |
|---|---|---|---|---|---|---|
| AR | 12.1 | 10.6 | 10.0 | 10.9 | 9.1 | 11.6 |
| KLASS | 19.2 (1.59×) | 15.0 (1.42×) | 13.0 (1.30×) | 14.3 (1.31×) | 8.5 (0.93×) | 13.4 (1.16×) |
| EB-Sampler | 21.7 (1.79×) | 18.7 (1.76×) | 19.1 (1.91×) | 18.6 (1.71×) | 10.0 (1.10×) | 17.1 (1.47×) |
| Hierarchy | 45.3 (3.74×) | 38.1 (3.59×) | 26.9 (2.69×) | 31.5 (2.89×) | 11.5 (1.26×) | 27.7 (2.39×) |
| Confidence Parallel | 43.6 (3.60×) | 36.0 (3.40×) | 25.3 (2.53×) | 30.0 (2.75×) | 10.6 (1.16×) | 24.3 (2.09×) |
| PARD | 44.1 (3.64×) | 36.8 (3.47×) | 29.4 (2.94×) | 28.9 (2.65×) | 11.7 (1.29×) | 25.2 (2.17×) |
As expected, Confidence Parallel and PARD achieve similar throughput on most models and tasks, consistent with their similar tokens/step patterns in Figure 15. PARD thus matches the speed of Confidence Parallel while improving its average accuracy by 3.8, 3.2, and 2.0 points on Fast-dLLM v2, SDAR, and LLaDA2.1-Mini, respectively (Table 2 in Section 5.2). KLASS runs slow due to its high precision operation (as discussed in Section 5.1), and EB-Sampler is similar to or slower than PARD.
Note that on Fast-dLLM v2 and SDAR, Hierarchy and APD are faster than PARD on HumanEval and MBPP, by – and –, respectively. However, on these tasks their base pass@1 is lower than that of PARD, by 3.5–12.2 points for Hierarchy and 4.5–10.4 points for APD (Table 2 in Section 5.2). We also showed that PARD achieves a better speed–quality trade-off than APD and Hierarchy in Section 5.1.
Q5. Inference backend and batch size.
We used HuggingFace Transformers (without model compilation) as the inference backend, on a single RTX A6000 GPU. Running inference with batch size 1 under-utilizes the A6000 GPU. Given the sufficient GPU memory, we therefore set the batch size to 4 to speed up our experiments. We used the same batch size for all methods for a fair comparison.
To confirm that PARD's speedup also holds at batch size 1, we measure the throughput (tok/s) of SDAR 8B at batch size 1 on HumanEval and GSM8K. Speedup over AR is shown in parentheses.
| Method | HumanEval | GSM8K |
|---|---|---|
| AR | 16.2 | 15.8 |
| Confidence Parallel | 37.0 (2.29$\times$) | 45.0 (2.85$\times$) |
| PARD | 37.6 (2.32$\times$) | 43.3 (2.75$\times$) |
PARD's speedup over AR is larger at batch size 1 than at batch size 4 (Table 3): vs. on HumanEval, and vs. on GSM8K. The speedup decreases as the batch size grows because the sequences in a batch move to the next block together. All sequences need to wait for the slowest sequence in each block. The speedups we report at batch size 4 are therefore conservative relative to the standard batch size of 1. The throughput of PARD is also comparable to that of Confidence Parallel at both batch sizes.
Q6. Relation to DMax.
Thank you for raising this related work. We have cited and discussed DMax in Sections 2.2 and 4 of the revised paper.
Mechanics. PARD-Confidence uses the same prefix selection rule as lines 5–8 of DMax's Algorithm 1. Both select the longest contiguous prefix of masked positions whose confidence exceeds the threshold, and fall back to the leftmost masked position if the prefix is empty. However, the two decoding procedures differ in three ways. First, in DMax the selected prefix positions are not finalized as tokens. Each one is fed back as a soft embedding, i.e., an interpolation between the embedding of its predicted token and the mask embedding, weighted by confidence, and its predicted token can change at every later step. In PARD, the selected positions are unmasked as discrete tokens. Second, DMax requires On-Policy Uniform Training (OPUT) to handle soft embeddings. When Algorithm 1 is run on the original LLaDA-2.0-mini without OPUT, GSM8K accuracy drops to 0.0% (DMax, Table 3). PARD operates in the discrete token space, so it applies directly to off-the-shelf BDLMs without training. Third, DMax uses confidence both as the selection score and as the interpolation weight. PARD is criterion-agnostic and also applies the prefix constraint to Margin and Entropy samplers.
Motivation. DMax and PARD adopt the prefix selection rule for different reasons. In DMax, the prefix selection rule is introduced heuristically to mitigate error accumulation during aggressive parallel decoding. PARD follows from our analysis of the stronger AR bias of BDLMs compared with MDMs (Section 3, Propositions 1–2). A prefix constraint is the simplest way to follow this bias while keeping parallel decoding.
W1 & Q1. AR bias beyond coding benchmarks.
Thank you for the question. We conducted the AR bias analysis of Section 3.1 on GSM8K (math reasoning). We first compare the accuracy obtained from pure AR sampling with the arbitrary-order samplers (confidence/entropy/margin) across SDAR, Fast-dLLM v2, LLaDA, and Dream:
For Fast-dLLM v2, its accuracies differ from Table 2 because we disable sub-block inference here to use the same block size of 32 as other models (discussed in Appendix C.1). We have also corrected Table 1 accordingly for Fast-dLLM v2; the conclusions remain the same.
The results are similar to those of HumanEval. Pure AR achieves the highest accuracy on both BDLMs. On both MDMs, pure AR falls slightly below Confidence, by 0.7 points on LLaDA and 0.5 points on Dream. In comparison, pure AR achieves higher accuracy than Confidence by 2.0 on Fast-dLLM v2 and 1.3 on SDAR.
As in Figure 1 of the submission, we record the within-block position unmasked at each step under Confidence. The average probability mass on the diagonal is 0.47 and 0.37 for Fast-dLLM v2 and SDAR, versus 0.27 and 0.30 for LLaDA and Dream. Within ±2 positions of the diagonal, the mass is 0.81 and 0.74 for the BDLMs, versus 0.70 and 0.67 for the MDMs. The decoding orders of the BDLMs are again more concentrated along the diagonal, indicating stronger AR bias.
Then we compare the local and global AR-ness at different k under the Confidence sampler for the four models:
Local AR-ness@k on GSM8K
Global AR-ness@k on GSM8K
Both BDLMs have substantially higher local and global AR-ness than both MDMs at every k. This is consistent with Figures 1 and 2 in the submission. We have added these GSM8K results to Appendix D.5 of the revised paper and referred to them in Section 3.1.
W1. MDMs are never benchmarked after Section 3.1.
Thank you for raising this point. We conduct a preliminary evaluation to compare PARD with Confidence Parallel at τc=0.9 on LLaDA 8B on GSM8K and HumanEval Base:
On GSM8K, PARD only slightly improves over Confidence Parallel, by 0.6 points. On HumanEval, PARD degrades performance by 3.0 points. Thus, PARD does not consistently outperform Confidence Parallel on LLaDA. In contrast, on all three BDLMs in Table 2, PARD outperforms Confidence Parallel on both benchmarks, by 1.5 to 6.7 points. This is consistent with our analysis in Section 3.1 that BDLMs have a stronger AR bias than MDMs. Hence, we focus our evaluation on BDLMs, which are also the more common choice among recent high-performing DLMs (as discussed in Section 2.1).