- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper identifies a limitation of commonly used decoding methods: marginal-only unmasking and dependency-aware unmasking cannot jointly do content and position selection. This could mean bad parallel decoding or more NFEs. The paper proposes an enhancement that leverages an external value-aware consistency model. By solving the quadratic unconstrained binary optimization problem, the method expands the pareto frontier in several experiment settings.
Critical strengths
- The method is intuitive and theoretically sound. The empirical benefits are clear.
- The value-aware consistency model can be instantiated based on the downstream tasks. The improvement is most noticeable when a strong consistency model is available in tasks like sudoku.
Critical weaknesses
- It seems to me that using bigram as the consistency model is a simplification of DFlash 2. So there might be limited novelty in this part.
Questions for authors
- Can you evaluate more models like Dream-v0-Instruct-7B and iLLaDA-8B-Instruct for the code generation experiments? It would be more assuring to know that ConGO works generally across models.
4 comments
In summary, ConGO now transfers to Dream-7B and LLaDA2.1-mini, with consistent gains across code and math benchmarks without per-benchmark tuning. We also clarify that its contribution lies in the value-aware unmasking objective rather than in a specific consistency model. We hope these results address the reviewer's concerns and have convinced them to (convincingly) recommend acceptance.
[1] Anonymous. (2026). Improving parallel decoding of block diffusion language models with autoregressive generation order [under review at ICLR 2027]
Thank you for the detailed and insightful rebuttal! Most of my concerns are addressed. I will keep the current score of 3.
We thank the reviewer for engaging with our rebuttal and for the constructive feedback throughout the review process. We are glad that the additional experiments on Dream-v0-Instruct-7B and LLaDA2.1-mini, as well as the clarification regarding DFlash 2, addressed most of the concerns. The suggestions have helped strengthen the paper, and we appreciate the reviewer’s continued support of our work.
Transfer to Dream-7B and LLaDA2.1-mini
We thank the reviewer for the careful review, and for recognizing ConGO's intuitiveness and theoretical soundness, as well as its clear empirical benefits. We address the questions below and have updated the paper accordingly.
Q1: Extended evaluation to more models
We evaluated ConGO on Dream-v0-Instruct-7B, as requested, and on LLaDA2.1-mini, a recent LLaDA model. Besides the static bigram, we now use a learned conditional PMI scorer (App. C.2; response to
ZG2j, W2) with one hyperparameter setting shared across all backbones and benchmarks. We also added BigCodeBench as a benchmark. Full results are in Table 2 and Figs. 2, 4, 6, 7, 8 (Dream) and Figs. 3, 9, 10 (LLaDA2.1-mini).Dream-v0-Instruct-7B: On both benchmarks, the thresholding-based ConGO variants beat thresholding in accuracy, NFE, and time. On MBPP, ConGO-COND-PMI-DAPD nearly matches DAPD's accuracy (49.6% vs. 50.0%) at fewer NFEs and lower latency. On HumanEval, DAPD itself does not improve on the thresholding front for Dream (please see updated Fig. 2), so extending it with ConGO brings limited benefit, while the thresholding-based ConGO variants extend the thresholding front. On BigCodeBench, ConGO-COND-PMI establishes the accuracy–latency front (Fig. 6): for example, it reaches 13.0% at 81 NFE and 3.6 s, while thresholding reaches 10.4% at 97 NFE and 4.6 s.
ConGO with the new conditional scorer also transfers to math (Figs. 7, 8). Gains on GSM8K are modest, while on MATH-500 with LLaDA-8B, ConGO-COND-PMI reaches thresholding’s best accuracy with ~20 fewer NFEs and exceeds the best accuracy of both baselines.
New coding results with Dream-v0-Instruct-7B. See Table 2 in the revised manuscript.
LLaDA2.1-mini. LLaDA2.1 is initialized from an AR model, hence PARD [1] (a concurrent ICLR submission) shows benefits from committing consecutive tokens. ConGO-PARD keeps the commit set consecutive and selects it exactly by dynamic programming. We compare ConGO-PARD without editing against the native decoder and PARD, each with and without token editing, with shared hyperparameters. Throughput is measured in tokens per forward pass (TPF). ConGO-PARD matches the strongest decoders in the high-accuracy regime. In the fast regime, it sustains ≥70% pass@1 at 1.26× / 1.42× the throughput of PARD on HumanEval / MBPP, without token editing, and the gains carry over to wall-clock time (Fig. 9). HumanEval+ and MBPP+ show the same trend (Fig. 10).
New coding results with LLaDA2.1-mini. See Figure 3 in the revised manuscript.
W1: Relation to DFlash 2
While we agree that both methods score candidate values at neighboring positions, we would like to clarify that the specific consistency model is not our claimed novelty. Instead, our contribution is the unmasking objective: a joint optimization over which positions to commit and at which values.
Different problem. DFlash 2 is a speculative-decoding drafter for an autoregressive target: its path selector fills every draft position, and the AR verifier decides how much to accept. In diffusion unmasking, there is no verifier and commitments are irreversible (token editing aside), so the central decision is which positions to commit and which to defer. In terms of ConGO's objective, DFlash 2's selector is a special case with adjacent couplings where every position must be committed, so it only chooses values. ConGO adds deferral, arbitrary couplings and hard constraints (Sudoku, DAPD integration), and exact solvers.
Consistency model. The conditional scorer added in the revision is inspired by DFlash 2's path selector, which we already credit in Sec. 3.2.2, but uses our own design and training (App. C.2). It is one plug-in for J, like the static bigram or any other existing pairwise consistency scorer.
Finally, we want to note that DFlash 2 appeared as a non-peer-reviewed blog post less than a month before the submission deadline, and we hence treat it as concurrent work as per the ICLR guidelines.