acceptodds
Under review as a conference paper at ICLR 2027

ConGO: Consistency-guided Optimization for Parallel Decoding in Discrete Diffusion

Abstract

Discrete diffusion is an emerging paradigm for generative AI, in which a sequence of tokens is generated through iterative updates using a learned denoising model. While the standard formulation randomly selects token positions to update in each denoising iteration, common heuristics select the positions with highest marginal confidence or minimum inter-token dependency to increase generation quality. These approaches, however, strictly isolate the position selection from the content selection. Such *value-blind* position selection hinders the commitment of jointly consistent tokens, which can be indicated by explicit constraints or learned priors. In this work, we propose consistency-guided optimization (ConGO) for parallel decoding in discrete diffusion models, which formulates the update selection as a quadratic unconstrained binary optimization problem. Guided by a lightweight, *value-aware consistency model*, ConGO offers a flexible spectrum of integration depending on the reliability of the available knowledge. When strong, explicit domain knowledge is available, such as in Sudoku, ConGO *jointly* selects both the positions and their content, achieving significantly higher accuracy on hard problems (up to a 42.4 pp improvement). When explicit constraints are absent, ConGO restricts its intervention purely to position selection based on learned pairwise priors (e.g., bigrams). Integrated into both marginal and dependency-aware decoding of LLaDA-8B-Instruct in coding, ConGO improves each method's accuracy-NFE trade-off towards fewer function evaluations, with the dependency-aware integration establishing a novel Pareto front.

Primary area:
generative models
Keywords:
Discrete diffusion models, masked diffusion models, uniform diffusion models, parallel decoding, non-autoregressive generation, quadratic unconstrained binary optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9X3rOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a limitation of commonly used decoding methods: marginal-only unmasking and dependency-aware unmasking cannot jointly do content and position selection. This could mean bad parallel decoding or more NFEs. The paper proposes an enhancement that leverages an external value-aware consistency model. By solving the quadratic unconstrained binary optimization problem, the method expands the pareto frontier in several experiment settings.

Critical strengths
  1. The method is intuitive and theoretically sound. The empirical benefits are clear.
  2. The value-aware consistency model can be instantiated based on the downstream tasks. The improvement is most noticeable when a strong consistency model is available in tasks like sudoku.
Critical weaknesses
  1. It seems to me that using bigram as the consistency model is a simplification of DFlash 2. So there might be limited novelty in this part.
Questions for authors
  1. Can you evaluate more models like Dream-v0-Instruct-7B and iLLaDA-8B-Instruct for the code generation experiments? It would be more assuring to know that ConGO works generally across models.
4 comments
AuthorsOpenReview ↗

Transfer to Dream-7B and LLaDA2.1-mini

We thank the reviewer for the careful review, and for recognizing ConGO's intuitiveness and theoretical soundness, as well as its clear empirical benefits. We address the questions below and have updated the paper accordingly.

Q1: Extended evaluation to more models

We evaluated ConGO on Dream-v0-Instruct-7B, as requested, and on LLaDA2.1-mini, a recent LLaDA model. Besides the static bigram, we now use a learned conditional PMI scorer (App. C.2; response to ZG2j, W2) with one hyperparameter setting shared across all backbones and benchmarks. We also added BigCodeBench as a benchmark. Full results are in Table 2 and Figs. 2, 4, 6, 7, 8 (Dream) and Figs. 3, 9, 10 (LLaDA2.1-mini).

Dream-v0-Instruct-7B: On both benchmarks, the thresholding-based ConGO variants beat thresholding in accuracy, NFE, and time. On MBPP, ConGO-COND-PMI-DAPD nearly matches DAPD's accuracy (49.6% vs. 50.0%) at fewer NFEs and lower latency. On HumanEval, DAPD itself does not improve on the thresholding front for Dream (please see updated Fig. 2), so extending it with ConGO brings limited benefit, while the thresholding-based ConGO variants extend the thresholding front. On BigCodeBench, ConGO-COND-PMI establishes the accuracy–latency front (Fig. 6): for example, it reaches 13.0% at 81 NFE and 3.6 s, while thresholding reaches 10.4% at 97 NFE and 4.6 s.

ConGO with the new conditional scorer also transfers to math (Figs. 7, 8). Gains on GSM8K are modest, while on MATH-500 with LLaDA-8B, ConGO-COND-PMI reaches thresholding’s best accuracy with ~20 fewer NFEs and exceeds the best accuracy of both baselines.

HE Acc (%) ↑HE NFE ↓HE Time (s) ↓MBPP Acc (%) ↑MBPP NFE ↓MBPP Time (s) ↓
Fast-dLLM (0.7)41.5591.725.0331.2
DAPD47.6833.350.0241.3
ConGO-PMI43.3501.329.4270.9
ConGO-COND-PMI43.3461.329.8190.7
ConGO-PMI-DAPD43.9662.545.8221.2
ConGO-COND-PMI-DAPD37.2311.349.6211.2

New coding results with Dream-v0-Instruct-7B. See Table 2 in the revised manuscript.

LLaDA2.1-mini. LLaDA2.1 is initialized from an AR model, hence PARD [1] (a concurrent ICLR submission) shows benefits from committing consecutive tokens. ConGO-PARD keeps the commit set consecutive and selects it exactly by dynamic programming. We compare ConGO-PARD without editing against the native decoder and PARD, each with and without token editing, with shared hyperparameters. Throughput is measured in tokens per forward pass (TPF). ConGO-PARD matches the strongest decoders in the high-accuracy regime. In the fast regime, it sustains ≥70% pass@1 at 1.26× / 1.42× the throughput of PARD on HumanEval / MBPP, without token editing, and the gains carry over to wall-clock time (Fig. 9). HumanEval+ and MBPP+ show the same trend (Fig. 10).

HE Best Acc (%)/TPFHE max TPF at ≥70%MBPP Best Acc (%)/TPFMBPP max TPF at ≥70%
Native87.2 / 5.78.784.8 / 4.16.2
Native + editing85.4 / 4.57.884.8 / 4.26.1
PARD89.0 / 5.79.686.2 / 6.17.2
PARD + editing87.8 / 4.57.786.8 / 4.86.3
ConGO-PARD90.9 / 6.012.186.4 / 4.910.3

New coding results with LLaDA2.1-mini. See Figure 3 in the revised manuscript.

W1: Relation to DFlash 2

While we agree that both methods score candidate values at neighboring positions, we would like to clarify that the specific consistency model is not our claimed novelty. Instead, our contribution is the unmasking objective: a joint optimization over which positions to commit and at which values.

Different problem. DFlash 2 is a speculative-decoding drafter for an autoregressive target: its path selector fills every draft position, and the AR verifier decides how much to accept. In diffusion unmasking, there is no verifier and commitments are irreversible (token editing aside), so the central decision is which positions to commit and which to defer. In terms of ConGO's objective, DFlash 2's selector is a special case with adjacent couplings where every position must be committed, so it only chooses values. ConGO adds deferral, arbitrary couplings and hard constraints (Sudoku, DAPD integration), and exact solvers.

Consistency model. The conditional scorer added in the revision is inspired by DFlash 2's path selector, which we already credit in Sec. 3.2.2, but uses our own design and training (App. C.2). It is one plug-in for JJ, like the static bigram or any other existing pairwise consistency scorer.

Finally, we want to note that DFlash 2 appeared as a non-peer-reviewed blog post less than a month before the submission deadline, and we hence treat it as concurrent work as per the ICLR guidelines.

In summary, ConGO now transfers to Dream-7B and LLaDA2.1-mini, with consistent gains across code and math benchmarks without per-benchmark tuning. We also clarify that its contribution lies in the value-aware unmasking objective rather than in a specific consistency model. We hope these results address the reviewer's concerns and have convinced them to (convincingly) recommend acceptance.

[1] Anonymous. (2026). Improving parallel decoding of block diffusion language models with autoregressive generation order [under review at ICLR 2027]

Thank you for the detailed and insightful rebuttal! Most of my concerns are addressed. I will keep the current score of 3.

We thank the reviewer for engaging with our rebuttal and for the constructive feedback throughout the review process. We are glad that the additional experiments on Dream-v0-Instruct-7B and LLaDA2.1-mini, as well as the clarification regarding DFlash 2, addressed most of the concerns. The suggestions have helped strengthen the paper, and we appreciate the reviewer’s continued support of our work.

Reviewer_22yrOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes ConGO for parallel decoding in discrete diffusion models, based on the observation that dependent tokens can still be committed together if their values are compatible, addressing the limitations of existing value-blind position selection methods.

ConGO formulates unmasking as a QUBO problem by combining unary scores for position-value assignments with pairwise consistency scores. For Sudoku, it jointly selects positions and values using explicit constraints, while for code generation, it improves position selection using corpus-based bigram statistics.

Experiments show substantial accuracy gains on Sudoku and significant NFE reductions over DAPD in code generation.

Critical strengths
  1. While existing parallel decoding methods primarily select positions based on marginal confidence or low inter-token dependency, this paper highlights that dependent positions can also be committed together when their predicted token values are compatible.

  2. ConGO achieves significant NFE reductions over DAPD in code generation using only simple bigram statistics, demonstrating the potential of value-aware parallel decoding.

Critical weaknesses
  1. Despite its general motivation, ConGO relies heavily on external information, such as known Sudoku rules or static bigram statistics, and does not provide a general method for estimating token compatibility across arbitrary positions. Moreover, the theoretical justification for its PMI-based QUBO objective relies on a first-order Markov assumption and conditional PMI, and pairwise compatibility does not generally guarantee joint consistency.

  2. Although ConGO-PMI-DAPD reduces NFEs, the overhead of combinatorial optimization offsets these gains, yielding no improvement in wall-clock latency.

Questions for authors
  1. Could a more efficient approximate QUBO solver preserve the NFE gains while reducing inference latency?

  2. Please highlight any important contributions that this review may have overlooked or underappreciated.

2 comments
AuthorsOpenReview ↗

Reduced solver cost leads to wall-clock gains, a new learned conditional scorer generalizes to new benchmarks and backbones, and a newly derived error bound justifies the Markovian approximation

We thank the reviewer for their feedback and appreciate the recognition of the significant NFE reductions of ConGO over DAPD and acknowledgement of our core insight, i.e., that dependent positions can be committed together when their values are compatible. Given these strengths, and a soundness assessment of "minor or fixable concerns," we were surprised by the low score. We took all the raised points seriously and addressed them as follows.

Q1, W2: Reduced solver cost and wall-clock time

We agree that our initial Gurobi implementation was general but suboptimal. However, we find that approximate optimization is not even needed for acceleration: ConGO's optimization problems are small and highly structured, allowing us to replace Gurobi with exact, problem-specific solvers (dynamic programming for the adjacent single-candidate objective and branch-and-bound for the DAPD integration). As detailed in App. D, both solvers recover Gurobi's optimal solutions, thus preserving the same accuracy–NFE trade-offs. Compared to the Gurobi implementation, these exact solvers further reduce average generation time per sweep by 1.52× / 1.1× for ConGO-PMI and 1.7× / 1.4× for ConGO-PMI-DAPD on HumanEval / MBPP, respectively. Thus, the NFE gains now carry over to wall-clock time (updated Fig. 2): ConGO-PMI-DAPD matches DAPD accuracy on HumanEval at 0.9 s (previously 1.4 s) instead of 1.1 s for DAPD, and on MBPP, it needs 85% of DAPD's time at matched accuracy. For details and the selected rows from the updated Table 2, please see our response to Q1 of ZG2j.

W1: External knowledge, pairwise consistency, and the Markovianity assumption

Use of external knowledge → learned conditional PMI scorer

We use Sudoku's exact rules deliberately: a perfect consistency model isolates how much consistency can improve parallel decoding. In the revision, a lightweight head on the denoiser's hidden states (0.04% of its parameters) learns the conditional PMI of Prop. 1 from natural language and code data (App. C.2). With one shared hyperparameter setting, it extends the accuracy–NFE and accuracy–latency fronts on five code and math benchmarks (HumanEval, MBPP, and new BigCodeBench, GSM8K, MATH-500) and three backbones (LLaDA-8B, and newly added Dream-7B, LLaDA2.1-mini; Figs. 2-4, 6–10). On LLaDA2.1-mini, it matches the best baseline accuracy and sustains >70% pass@1 at 26–42% higher throughput than the strongest baseline (see responses to ZG2j W2/Q3-Conditional scorer and 9X3r Q1).

No compatibility across arbitrary positions → Demonstrations on Sudoku and DAPD

The ConGO objective is defined over arbitrary pairs; restricting JJ to adjacent ones is an instantiation choice - Sudoku already couples non-adjacent cells using reliable external knowledge, and the DAPD integration adds value-blind couplings between arbitrary pairs. For ConGO's learned joint scores on language, we chose adjacent pairs as dependencies are strongest locally. Our new conditional scorer can be trained the same way for non-adjacent pairs (stated as future work in the revision), and any pairwise estimator plugs into JJ unchanged.

Markovian assumption → New approximation error bound

We agree that pairwise compatibility does not guarantee joint consistency in general (now discussed in App. B.2). ConGO handles both known and learned consistency constraints: known constraints are enforced exactly whenever they decompose into pairs (as in Sudoku, where pairwise checks capture every rule, so no committed set can violate a constraint); otherwise, Prop. 1 provides a tractable first-order approximation that generalizes marginal-only sampling rules (recovered at J=0J=0). However, this approximation relies on exact Markovianity, a strong assumption that does not hold in general. We therefore derive a new Prop. 2 (App. B.2), showing that the QUBO solution's optimality gap grows linearly with deviations from exact Markovianity and marginal estimation errors, bounding the cost of the approximation and justifying its use.

Q2: Underappreciated contributions

A unifying objective. Existing samplers differ mainly in what they assume about token interactions: marginal-only rules assume none (J=0J=0), and dependency-aware rules forbid them. ConGO makes this explicit and adds the missing case: compatible interactions. It therefore upgrades existing rules rather than competing with them: any existing score enters through the unary term, and the value-aware coupling decides what can be safely committed together.

Controlled evidence for the value-aware term. Because of this nesting, our code comparisons are clean ablations: ConGO-PMI shares Fast-dLLM's linear term, and ConGO-PMI-DAPD shares DAPD-direct's, so the shifted Pareto fronts are attributable solely to the pairwise coupling.

Presentation

We took the low presentation score to heart and aimed to improve the clarity of the manuscript. Since no specific issues were pointed out, we revised the parts we found densest: Sec. 3.2.2 now derives the objective before discussing how the coupling is estimated, related work positions the closest methods as special cases of the ConGO objective, and latency plots are provided for all language settings. We welcome pointers to any remaining issues and are happy to revise them accordingly.


In summary, the NFE gains now translate to wall-clock improvements, a learned conditional PMI scorer replaces static bigrams as a de facto scorer and generalizes to two new models and three new benchmarks, and a new theoretical result derives an error bound when first-order assumption holds only approximately. We would be grateful if the reviewer reconsidered the score in light of these changes, and are happy to clarify anything further.

Reviewer_ZG2jOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ConGO formulates parallel updates as joint selection of positions and candidate values. A quadratic objective combines marginal evidence with value-aware consistency. Sudoku uses exact constraints and multiple candidates. Code generation uses a static bigram prior with one candidate per position. Experiments cover Sudoku and LLaDA code generation.

Critical strengths
  1. The distinction between dependency and incompatibility is useful. Strongly dependent values can still be jointly valid. Multi-candidate Sudoku tests clearly demonstrate this benefit.

  2. The framework spans hard constraints and learned priors. Baseline sweeps and a sequential Sudoku control help isolate its effects. The appendix openly reports solver costs.

Critical weaknesses
  1. The efficiency evidence is mixed. In Table 2, ConGO-PMI-DAPD takes 1.4 seconds on HumanEval versus 1.1 for DAPD. Accuracy is essentially unchanged. Appendix D reports large solver overhead for the small Sudoku model. Fewer forward evaluations therefore do not establish faster inference.

  2. The general-language evidence is narrow. Code experiments use only LLaDA 1.0, two benchmarks, and unconditional adjacent-token PMI. They never exercise multi-candidate value selection. The conditional-PMI derivation is replaced by a corpus prior. DiffusionGemma results broaden Sudoku validation, but do not establish transfer of the code method. General-language applicability to newer backbones remains unclear.

Questions for authors
  1. Which operating points improve the accuracy versus wall-clock frontier after all solver costs? Please include latency sweeps for the DAPD integration.

  2. Can the adjacent single-candidate objective be solved directly by dynamic programming? How does that implementation compare with Gurobi in quality and total time?

  3. Does the code method improve quality and total latency on a newer backbone? Please compare its native decoder under matched settings. Does the PMI prior transfer without benchmark-specific retuning?

  4. The comparison and discussion of dependency modeling and error correction need more depth. Relevant approaches include DEMASK [1], Mean-Field Parallel Decoding [2], and CoDD [3]. WINO [4] and ReMix [5] instead use revision and continuous refinement. Which gains specifically require value-aware joint optimization? These comparisons concern code generation. They do not replace evaluation of exact Sudoku constraints.

[1] Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models

[2] Mean-Field Parallel Decoding for Discrete Diffusion Language Models

[3] Breaking the Factorization Barrier in Diffusion Language Models

[4] Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs

[5] Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference

4 comments
AuthorsOpenReview ↗

Exact problem-specific solvers turn NFE savings into wall-clock gains across new backbones and benchmarks

We thank the reviewer for the constructive review and for recognizing our core idea that dependence is not incompatibility, as well as the framework's breadth, controlled ablations, and transparent solver costs. We address the questions below and have updated the paper accordingly.

W1, Q1, Q2: Reduced solver cost and wall-clock time

Following the reviewer's suggestion, we replaced Gurobi with two exact, problem-specific solvers (App. D). Both return Gurobi's optimal solutions, so accuracy and NFE are unchanged.

Q2. With one candidate per position and only adjacent couplings, the objective of Prop. 1 is a chain with explicit constraints, that can be solved exactly by a dynamic program (DP). Per-step selection time drops from 11.34 ms to 0.59 ms (HumanEval) and from 4.89 ms to 0.70 ms (MBPP), making generation 1.52× and 1.1× faster on average. For the DAPD integration, whose conflict graph is arbitrary, a branch-and-bound solver with a clique-cover bound makes generation 1.7× and 1.4× faster on average.

Q1/W1. Fig. 2 now plots accuracy against generation time, including all solver costs and the requested DAPD sweeps. ConGO-PMI-DAPD took 1.4 s on HumanEval with Gurobi and now takes 0.9 s, 18% less than DAPD at matched accuracy; on MBPP, it needs 85% of DAPD's time at matched accuracy.

HE Acc (%) ↑HE NFE ↓HE Time (s) ↓MBPP Acc (%) ↑MBPP NFE ↓MBPP Time (s) ↓
WINO37.8651.523.6431.9
ReMix37.2641.327.0401.6
Fast-dLLM (0.7)37.2521.123.4220.9
DAPD42.1491.135.2281.3
ConGO-PMI36.6400.830.0220.9
ConGO-COND-PMI36.0320.723.4150.6
ConGO-PMI-DAPD42.0420.935.0231.1
ConGO-COND-PMI-DAPD40.9410.935.8241.1

Updated results with LLaDa-8B-Instruct. See Table 2 in the revised manuscript.

W2, Q3: Extended language evaluation

We thank the reviewer for pushing us on the breadth of our empirical evaluation, which led us to extend it substantially.

Conditional PMI. We add a lightweight scorer that estimates the conditional PMI from the model's hidden states (App. C.2), directly instantiating Prop. 1 and removing the theory–implementation mismatch of the initial submission. Unlike the bigram, it uses one hyperparameter setting shared across all benchmarks and backbones, yet yields the fastest points and the best MBPP accuracy (see ConGO-COND-PMI(-DAPD) in the table above).

Backbones and benchmarks. Paired with the new conditional scorer, ConGO transfers to Dream-7B (Fig. 2, Table 2; see also response to Q1 of 9X3r) and to BigCodeBench, GSM8K, and MATH-500 (Figs. 6–8). E.g., on MATH-500 with LLaDA-8B, ConGO-COND-PMI matches the accuracy of thresholding and DAPD (29.0%) in 20 fewer steps.

No multi-candidate values in coding. With the learned scorer, top-5 value selection improves over top-1 on MBPP (25.2% vs. 23.4% at ~15 NFE) and moderately on HumanEval (37.2% vs. ~36.0% at ~45 NFE). However, these improvements are moderate and, as a strong scorer is crucial for this setting, we plan to investigate this aspect further as part of our future work.

Q3: Newer backbone. On the AR-initialized LLaDA2.1-mini, ConGO-PARD commits consecutive runs like PARD [1] (a concurrent ICLR 2027 submission), but selects them with the ConGO objective and the learned scorer, solved exactly by DP. Against the native threshold decoder and PARD, each with and without token editing and with shared hyperparameters, it matches the strongest baseline at the highest accuracy (Fig. 3) and significantly improves the accuracy of baselines in the high throughput regime (see also response to Q1 of9X3r). Importantly, the gains carry over to wall-clock time (Figs. 9, 10).

Q4: Positioning

WINO and ReMix. Thank you for pointing us to these related works. They generally correct errors after commitment, while ConGO aims to prevent them from occurring. In the table above, every ConGO variant needs fewer NFEs and less time than both, and the DAPD-based variants are also more accurate. Both mechanisms could be combined.

DEMASK, Mean-Field PD. Like DAPD, both model the dependency for each pair of positions and can only suppress co-commitment of dependent positions, never recognize a consistent assignment or commit a non-argmax value. Mean-Field PD, the closest to ConGO, is an instance of Eq. 1 with ∣Ci∣=1|C^i|=1 and a value-blind J≤0J \le 0, solved approximately. Neither has public code for a comparison.

CoDD is complementary: it leaves position selection to standard heuristics, and its joint could serve as a pairwise consistency JJ in ConGO.

We expanded Sec. 5 and App. A to include these clarifications.


We hope the new results address the reviewer's concerns, and to have convinced them to (convincingly) recommend acceptance.

[1] Anonymous. (2026). Improving parallel decoding of block diffusion language models with autoregressive generation order [under review at ICLR 2027]

We greatly appreciate the time and effort dedicated by the reviewer to improve our work. As the discussion period is coming to an end, we kindly ask the reviewer to acknowledge whether we have successfully addressed their outstanding concerns. We remain available to further discuss any remaining question regarding our work.

Thanks for the response. I’ll keep my score.

We thank the reviewer for engaging with our rebuttal and for the constructive feedback throughout the review process. We appreciate the reviewer recognizing our core idea that dependence is not incompatibility, as well as the framework’s breadth, controlled ablations, and transparent solver costs. The reviewer’s questions and suggestions helped strengthen the revision: the dynamic-programming solver turned ConGO’s NFE savings into wall-clock gains, and the requests for latency sweeps and newer backbones led us to substantially broaden the evaluation. We appreciate the reviewer’s support of our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response and summary of updates

We thank all reviewers for their constructive feedback. Reviewers value ConGO's core idea of casting each parallel-decoding step of a discrete diffusion model as a joint selection of positions and consistent values (ZG2j, 22yr). They find the method intuitive, sound (9X3r), and flexible across hard constraints and learned priors (ZG2j, 9X3r), and highlight the Sudoku gains (42.4 pp) from multi-candidate value selection (ZG2j, 9X3r), the NFE reductions over DAPD (22yr), the controlled ablations, and the transparent solver-cost reporting (ZG2j).

Our response addresses the raised concerns with methodological enhancements, supported by substantial additional experiments. Details are in the individual responses, and changes are highlighted in blue in the revised PDF. The main changes can be summarized as follows:

1. Reduced solver overhead turns NFE savings into wall-clock gains (ZG2j, 22yr)

With the Gurobi solver, NFE savings have already translated into latency gains for ConGO-PMI and for DiffusionGemma at matched accuracy on Sudoku, but not for ConGO-PMI-DAPD on HumanEval. To address this, we exploit the underlying mathematical structure of the optimization problems, which makes a general-purpose solver like Gurobi unnecessarily expensive. Specifically, for the language instantiations, we replaced it with two exact, problem-specific solvers, whose solutions match the ones obtained via Gurobi:

  • a dynamic program for the adjacent single-candidate objective, as suggested by ZG2j. On HumanEval, average selection time per step drops from 11.34 ms to 0.59 ms and from 4.89 ms to 0.70 ms on MBPP. The average generation in the revision is hence 1.52x and 1.1x faster on HumanEval and MBPP, respectively, compared to the initial submission.
  • a dedicated branch-and-bound solver for integration with DAPD, resulting in 1.7 and 1.4 times faster average generation on HumanEval and MBPP, respectively, compared to the initial submission.

Our new implementation of QUBO optimization hence eliminates the initially reported solver overhead on coding tasks. With all solver costs included, ConGO extends the accuracy–wall-clock fronts of its base rules to faster operating points on both HumanEval and MBPP. We added latency sweeps for every language backbone and benchmark (updated Fig. 2; new Figs. 6, 8–10), including the DAPD-integration one requested by ZG2j.

2. New learned conditional PMI estimator extends the static bigram prior (ZG2j, 22yr, 9X3r)

Our initial submission approximated the conditional PMI of Prop. 1 with an unconditional corpus statistic. We now also train a lightweight conditional PMI estimator that directly instantiates Prop. 1, and is interchangeable with the bigram model. We further note that our contribution is the value-aware joint optimization, and any signed, value-indexed scorer can be plugged into it.

3. Transfer to two additional backbones (ZG2j, 9X3r)

With the new conditional estimator, ConGO transfers to Dream-v0-Instruct-7B and LLaDA2.1-mini, improving the accuracy–NFE and accuracy–latency trade-off over the respective baseline decoders under matched settings, without benchmark-specific hyperparameter tuning.

4. Extended breadth of evaluation (ZG2j)

We added BigCodeBench and two math reasoning benchmarks (GSM8K, MATH-500). ConGO improves or matches its base rule in most settings, with the clearest gains on MATH-500 and, for the confidence-based integration, on BigCodeBench.

5. Added near-optimality guarantee under approximate first-order dependence (22yr)

22yr argued that pairwise compatibility does not guarantee joint consistency in general and we agree. However, it is exact for Sudoku, whose rules decompose into pairwise all-different constraints. In language, it matches the pairwise granularity of dependency-aware baselines (e.g., DAPD, DEMASK). ConGO adds the value information those methods lack, and the empirical gains show this pays off. We extend Prop. 1 to an approximation guarantee (App. B.2) that justifies its use when the first-order assumption does not hold exactly: if higher-order dependencies and marginal errors are bounded, the QUBO solution is near-optimal for the true objective, with a gap that grows linearly in these errors and in the number of candidate positions.


In summary, exact problem-specific solvers turn ConGO's NFE savings into wall-clock gains, and a learned conditional PMI scorer instantiates the coupling of Prop. 1 and generalizes across backbones and benchmarks with a single, untuned hyperparameter setting. A new guarantee bounds the cost of the first-order approximation. We hope these changes address the reviewers' concerns and are happy to clarify anything further.

AuthorsOpenReview ↗

The scope of ConGO

We thank the reviewer for the careful reassessment and for raising their score. We are glad the revision resolved the latency concern, and that the reviewer finds the core idea compelling and practically useful. Regarding the remaining concern on scope, we agree that the implemented learned scorers only couple adjacent tokens. When combined with a dependency estimator, however, the resulting decoder extends to global interactions (as in ConGO-DAPD). We did not make this argument explicit in the manuscript and will add it to Sec. 3.2.3 in the revision.

1. Every candidate pair is either excluded, checked for consistency, or independent and hence safe to co-commit. Non-adjacent pairs are either dependent, in which case DAPD's exclusion constraints prevent them from being co-committed, or independent, in which case their PMI, and hence the coupling prescribed by Prop. 1, is zero, so no consistency check is needed. The remaining pairs are adjacent and checked by the value-aware term.

2. DAPD decides where to commit in parallel; ConGO decides how far each subsequence can grow. The tokens committed in one step form several contiguous subsequences, separated by masked positions. DAPD only co-commits tokens it judges independent, so each subsequence is a single token. ConGO-DAPD keeps this independence between subsequences, but lets each subsequence extend over adjacent tokens as long as their values are compatible. This decision is not purely local: each token is scored by the denoiser's marginal, which conditions on the full context xtx_t, and the pairwise term only corrects for the neighbour committed in the same step; with the learned conditional scorer, this correction is also conditioned on xtx_t. The combined decoder therefore generalizes the assumption dependency-aware decoders already make.

3. ConGO adds no error between subsequences, and little within them. Between subsequences, the error is of the same kind as for DAPD alone, i.e., it stems from dependencies its estimator misses. Within a subsequence, every adjacent pair is checked, so two-token subsequences add no error. Longer ones may miss a residual dependence between non-neighbouring tokens, but our results indicate this is small in practice: in particular, on LLaDA2.1-mini, whose AR initialization favours long contiguous commitments, ConGO reaches 1.26× / 1.42× the throughput of the baselines on HumanEval / MBPP at ≥70% accuracy (Sec. 4.2, Fig. 3).

4. The approach holds across tasks, diffusion families, and scales. ConGO uses both external knowledge and learned priors, and we evaluate it on logic puzzles, code (HumanEval(+), MBPP(+), BigCodeBench), and math (GSM8K, MATH-500), with masked and uniform diffusion and five denoisers from 5.4M to 26B parameters. Additionally, the conditional scorer requires no per-task hyperparameter tuning.


To summarize, ConGO complements any dependency estimator with the ability to commit dependent tokens whenever their values are compatible, while confining the additional approximation error to contiguous subsequences. The learned value-aware scorers couple only adjacent tokens, since these dependencies are the most informative in code and language, but they score these pairs in context. Combined with a dependency estimator, the resulting decoder extends to global interactions. Scorers for non-adjacent pairs would extend the value-aware checks further, acting as a value-aware generalization of the dependency score, and plug into the same objective without modification. As a result, we respectfully disagree that the scope of the proposed method is narrow. ConGO is more general than most existing unmasking strategies, generalizing both marginal-only and dependency-aware families of methods and introducing a novel consistency-based family of unmasking strategies. We would be grateful if the reviewer took this into account in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.