acceptodds
Under review as a conference paper at ICLR 2027

Contextual Commitment for Parallel Masked Diffusion Decoding

Abstract

Masked diffusion decoders can use unresolved predictions as context before committing them to discrete tokens. Choosing which predictions to commit and how strongly to update this provisional context requires accounting for dependence between the two groups. Yet the model provides only per-position distributions, which do not determine their joint dependence. To address this problem, we propose Contextual Commitment (CoCom). Under one dependence budget, the selected commitments constrain how strongly pending predictions can update the next input. We couple these decisions by bounding total correlation in a reference observation of committed variables and randomly retained pending variables. The bound uses only current marginals and holds for every compatible joint distribution. Both decisions therefore reuse the current model predictions, with zero additional forward passes, no additional training, and no separately tuned update strength. Across four backbones and four benchmarks, CoCom improves the accuracy–computation tradeoff over the compared accelerated decoders. Budgets selected on LLaDA-1.5 transfer directly to LLaDA-Ins and Dream, where CoCom achieves the highest accuracy among accelerated decoders in seven of eight settings. Ablations support the benefit of selecting commitments and pending updates jointly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Masked diffusion language models, Parallel decoding, Contextual commitment, Total correlation, Training-free inference

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GwZcOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents a training-free decoding framework called CoCom for masked diffusion language models. The main idea is to jointly determine which predicted tokens should be permanently committed and how strongly the remaining predictions should be used as revisable context for subsequent denoising steps. The authors further derive a computable upper bound on total correlation using only the model's marginal distributions. Based on this bound, CoCom selects the largest feasible commitment prefix and the corresponding retention probability. The resulting retention probability is then used as the interpolation strength for updating the pending token embeddings.

Critical strengths

Pros

  1. The mathematical formulation and the proposed method are coherent. I found the mathematical derivations clear, and the algorithmic design is well motivated by the theoretical analysis.
  2. The experimental results are promising. CoCom outperforms many of the compared accelerated decoding methods across different models and tasks. The reduction in mean NFE is substantial, and the reported wall-clock speedups over Native decoding are also encouraging.
  3. The ablation studies are extensive and useful.
Critical weaknesses

Cons

  1. The experimental setup for the Native baselines is not entirely clear. CoCom uses block-wise diffusion with a block length of 64. However, the authors do not clarify whether the Native baselines use the same block-wise setting or full-sequence diffusion. Since the block length can significantly affect decoding performance, this difference in configuration could make the comparison less fair.
Questions for authors
  1. My only question is related to the weakness. Could the authors clarify the specific decoding setup used for the Native DLM baselines in the Main Results Table? As mentioned in Table 6, CoCom uses block-wise decoding with a block length of 64, but the block length for Native is not specified. I also noticed that the reported GSM8K accuracy for LLaDA-1.5-8B is substantially lower than the officially reported results under block-wise decoding. Does the Native baseline use the same block-wise setting as CoCom, or full-sequence diffusion?
3 comments

We sincerely thank the reviewer for the positive assessment and for identifying an important omission in our experimental description. We clarify the Native settings below and provide an additional comparison with matched block lengths.

Critical Weaknesses

The experimental setup for the Native baselines is not entirely clear. CoCom uses block-wise diffusion with a block length of 64. However, the authors do not clarify whether the Native baselines use the same block-wise setting or full-sequence diffusion. Since the block length can significantly affect decoding performance, this difference in configuration could make the comparison less fair.

Question 1

My only question is related to the weakness. Could the authors clarify the specific decoding setup used for the Native DLLM baselines in the Main Results Tables? As mentioned in Table 6, CoCom uses block-wise decoding with a block length of 64, but the block length for Native is not specified. I also noticed that the reported GSM8K accuracy for LLaDA-1.5-8B is substantially lower than the officially reported results under block-wise decoding. Does the Native baseline use the same block-wise setting as CoCom, or full-sequence diffusion?

Native used full-sequence diffusion over a 256-token generation canvas, with 256 forward evaluations. CoCom used block length 64. We agree that this difference should have been explicit, since block length can affect accuracy independently of the acceleration method.

Our evaluation fixes the generation length at 256 tokens across backbones and methods to provide a common output budget. Full-sequence diffusion is consistent with the sampling regime in the original LLaDA-Instruct main results [1], as well as the Dream and DiffuCoder models [2, 3]. For LLaDA-1.5 specifically, the official evaluation searches sampling strategies, generation lengths, and block lengths, and reports the best results [4, Appendix D.4].

To address your concern about the fairness of this comparison, we have additionally evaluated Native with the same block length of 64 as CoCom on all four LLaDA-1.5 benchmarks, keeping its original prompts, generation length, prediction rule, and fixed 256-forward budget. The results are shown below. In the revised manuscript, we specify Native's block length in the decoder-protocol table, add this matched-block comparison to the appendix, and summarize its findings in the main results.

DatasetOriginal Native ACC (%)Native block64 ACC (%)CoCom block64 ACC (%)Native / CoCom NFE
HumanEval40.2439.6343.90256.00 / 64.76
MBPP36.8039.0040.40256.00 / 51.58
GSM8K57.2482.0380.97256.00 / 61.39
MATH50024.0038.4038.60256.00 / 96.37

Native block64 reaches 82.03% on GSM8K, close to the officially reported 83.3%. With matched block lengths, CoCom achieves higher accuracy on three tasks. On GSM8K, its accuracy is 1.06 percentage points lower with 76.0% fewer forward evaluations (61.39 vs. 256). These results support CoCom's favorable accuracy-efficiency tradeoff.

References

[1] Nie, S., et al. Large Language Diffusion Models. NeurIPS, 2025.

[2] Ye, J., et al. Dream 7B: Diffusion Large Language Models. arXiv preprint arXiv:2508.15487, 2025.

[3] Gong, S., et al. DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation. ICLR, 2026.

[4] Zhu, F., et al. LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Models. arXiv preprint arXiv:2505.19223v2, 2025.

Thank you for the clarification and the additional experiments. These are good results showing CoCom gets comparable quality results to native baseline at the same time produces significant speedup. I have increased my score to clear accept.

We sincerely appreciate your positive feedback and updated assessment, and are glad that the additional experiments addressed your concerns. Thank you again for your constructive review, and we will make sure to polish the manuscript accordingly.

Reviewer_3FADOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CoCom jointly selects permanent token commitments and provisional context updates under a shared dependence budget. A reference observation combines committed variables and randomly retained pending variables, yielding computable bounds and a coupled decoding rule. Experiments cover four backbones and four benchmarks without auxiliary forward passes. The formulation is promising, but its connection to executed updates, budget selection, and practical constraint adherence requires clarification.

Critical strengths
  1. Coupling permanent commitments and provisional context under one dependence constraint is a meaningful conceptual contribution that captures interactions between the two decisions.
  2. Computable bounds turn the formulation into a practical decoder using current marginals, without additional training or auxiliary forward passes.
  3. Evaluation across four backbones and four benchmarks provides useful breadth. Mostly unchanged budget transfer offers evidence beyond independently tuned configurations.
  4. Mechanism analyses and ablations examine separable bounds, context persistence, and alternative commitment rules, helping distinguish the roles of the proposed components.
Critical weaknesses
  1. The guarantee concerns a current-step reference observation, whereas execution interpolates fixed argmax embeddings. The expectation correspondence does not establish control over accumulated inputs or subsequent predictions, leaving the theory’s explanation of actual decoding indirect.
  2. Budget selection is not fully isolated from evaluation. Appendix B.1 selects the MBPP budget using its first 100 evaluation tasks; the separate 200-question DiffuCoder GSM8K scan also lacks a clearly documented disjoint selection set.
  3. Practical constraint adherence remains unresolved. Figure 8 reports hard-set costs exceeding the budget on 279/10,047 HumanEval and 815/76,317 GSM8K pending steps. Exceedance magnitudes and consequences need quantification, ideally alongside a consistently feasible implementation. These observations do not establish that the theorem is incorrect.
Questions for authors

See Weaknesses

3 comments

We sincerely thank the reviewer for the careful reading and recognition of CoCom's joint formulation and experiments. Your questions helped us clarify how the theory guides decoding and test the robustness of our comparisons. Both are addressed in the revised manuscript.

W1

How does the current-step reference guarantee explain the executed embedding updates?

The retention probability selected under the reference bound is used directly as the embedding interpolation coefficient. The joint rule uses current marginals to select commitments and retention probability α under a shared budget (Appendix A.5). This selection accounts for how the information added by a pending prediction depends on the committed set.

Each pending input moves a fraction α toward its argmax embedding. For fixed candidates, this equals the expected input under replacement by the candidate embedding with probability α (Appendix A.6). This correspondence holds at the input level. A guarantee for later predictions would require assumptions on the model's response to input changes.

At each forward pass, we recompute marginals from the updated context and apply the bound again. Earlier updates are already reflected in these marginals, so the current-state bound remains applicable. At fixed states, commitments, candidates and budgets, joint α exceeds separable α on over 95% of pending steps in every setting. Ranges cover four tasks per model.

BackbonePending stepsSteps with larger α (%)
LLaDA-1.5157,39696.49–98.62
LLaDA-Instruct160,42395.75–98.76
Dream135,13498.57–99.43
DiffuCoder158,05897.07–99.20

Every such step has a nonzero embedding update. The joint/separable ablation tests the value of this additional strength. Mask-only and reset-context ablations test using and preserving pending context (Section 4.3).

W2

Budget selection overlaps the MBPP and DiffuCoder GSM8K evaluation sets.

We follow the parameter-selection practice of all five accelerated baselines, as not all evaluation benchmarks provide a separate validation split. CoCom has only one control parameter, the budget β. Dataset-specific budgets selected on LLaDA-1.5 are reused unchanged on LLaDA-Instruct and Dream, giving the best accelerated accuracy in 7/8 settings. DiffuCoder, which is trained specifically for code generation, uses the same budgets except on GSM8K, where a separate scan selects a smaller budget (Appendix B.1).

To address your concern, we recomputed the comparisons after excluding the same selection questions for every method, keeping all budgets and settings fixed. The remaining sets contain 400 MBPP tasks (LLaDA-1.5) and 1,119 GSM8K questions (DiffuCoder). Cells give ACC (%)/NFE.

MethodMBPPGSM8K
Native35.75/256.0048.61/256.00
DAPD31.25/50.6823.32/83.31
ReMix30.50/67.5730.12/74.37
Saber36.50/99.6342.27/120.09
WINO37.00/73.5342.18/118.72
COVER36.00/97.8445.04/142.65
CoCom39.00/52.0947.99/60.46

CoCom still exceeds all five accelerated baselines in accuracy on both sets, with lower NFE than all except DAPD on MBPP.

W3

How large are the budget exceedances, and what happens under strict enforcement?

The screening approximation has an explicit error bound, and strict enforcement gives comparable accuracy and NFE. To reduce full meet evaluations, candidates with entropy at most 0.01 nats receive zero screening cost. Omitting this entropy can make the full commitment cost exceed β, in which case we set pending strength to zero (Appendix B.2).

The cost error is bounded by the total omitted entropy. For candidates A ⊆ C retained in screening, let r(C) = ∑ H(pᵢ), summing over i ∈ C ∖ A. Then 0 ≤ B(C) − B(A) ≤ r(C), so B(A) ≤ β implies B(C) ≤ β + r(C) in exact arithmetic. Across all 611,011 saved pending steps, the computed hard and joint bounds stay within the budget plus this allowance, recorded numerical slack and a 10⁻⁶ tolerance. On LLaDA-1.5, exceedances occur on 2.78% of HumanEval and 1.07% of GSM8K pending steps. Among these steps, mean/max excesses are 0.0314/0.1753 nats for HumanEval and 0.0134/0.1033 for GSM8K.

To address your concern about strict feasibility, we enforce B(C) ≤ β with an untruncated FP64 check, keeping other settings fixed. This yields zero violations over 10,598 HumanEval and 81,201 GSM8K steps, with comparable accuracy and NFE. Cells give ACC (%)/NFE.

LLaDA-1.5OriginalStrict
HumanEval43.90/64.7643.29/64.62
GSM8K80.97/61.3980.89/61.56

Thank you for the detailed response. The comparisons excluding the budget-selection examples substantially reduce my concern about evaluation overlap. The strict-feasibility results are particularly helpful: they suggest that the reported performance does not depend on budget violations.

The input-level correspondence also clarifies how the bound guides the implemented updates, although it does not guarantee the behavior of subsequent predictions. Please keep this distinction explicit and clearly document the screening approximation and its error allowance.

Based on these additional results, I am raising my score from 2 (Weak reject) to 3 (Weak accept). The joint formulation and empirical evidence now support a borderline acceptance recommendation.

Dear Reviewer 3FAD, thank you for your reply and for raising your score. In the final version, we will state explicitly that the correspondence holds at the input level and does not guarantee subsequent predictions, and we will clearly document the screening approximation and its error allowance.

Reviewer_k1SpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CoCom, a training-free decoding method for masked diffusion language models. The key motivation is that predictions at different positions can be dependent, and separately controlling token commitment and provisional context updates may overlook the dependencies between them. To address this, the authors derive a computable upper bound on total correlation in a reference observation using only marginal predictions, allowing both decisions to be coordinated under a shared dependence budget without additional model forward passes.

Critical strengths

One aspect I find particularly valuable is that the paper considers dependencies both within and across committed and pending tokens, rather than treating token commitment and context updates as two separate decisions. The authors use Total Correlation to derive a computable dependence bound that jointly controls these decisions. I think this is the main contribution of the paper, supported by a reasonable motivation and mathematical analysis.

The paper evaluates CoCom on four models and four benchmarks. It outperforms other accelerated decoding methods in most settings while using fewer forward passes than native decoding. The ablation studies also provide some evidence for the benefit of jointly controlling commitment and pending updates. Although the improvements vary across tasks, the overall results suggest that the method has practical value.

Critical weaknesses

The benefit of the joint dependence bound is not consistently demonstrated across tasks. While the ablation results show improvements over the simpler separable bound, the gains are relatively small on GSM8K and MATH500. Moreover, these ablations are only conducted on LLaDA-1.5. It would be helpful to see whether the joint bound provides consistent improvements across other backbones, as this is one of the main contributions of the paper.

Questions for authors

1.The ablation studies are only conducted on LLaDA-1.5. Could the authors clarify whether the joint dependence bound provides consistent improvements over the separable bound across other backbones?

2.The experiments use a fixed left-to-right token commitment order. Have the authors evaluated alternative strategies, such as confidence-based ordering, and how do they affect decoding performance?

1 comment

We sincerely thank the reviewer for the time and care devoted to reviewing our work and for the thoughtful, constructive feedback. We extended the joint/separable ablation to all four backbones and tested confidence-based commitment ordering.

Weakness & Q1

The joint-bound gains are small on GSM8K and MATH500, and the ablations cover only LLaDA-1.5. Does the joint bound provide consistent improvements over the separable bound across other backbones?

The joint correction certifies the commitment-pending interaction that the separable bound must treat as worst case. The separable variant keeps CoCom's shared budget, commitment rule and persistent pending context, allocating strength from β − B(C) (Appendix D.4). By adding the two costs, it assumes pending predictions may add no information beyond the commitments. From marginals alone, the correction certifies the new information each pending prediction must add given the committed set. The ablation isolates this certification, while bound-based commitment and persistent pending context are tested separately. For example, fixed-quota commitment loses 20.55 GSM8K points at comparable NFE, and mask-only or reset pending inputs lower accuracy on HumanEval, MBPP and MATH500 (Section 4.3).

The correction is nonnegative, so at a fixed state and commitment set it preserves every strength admitted by the separable bound (Eq. (16)). In cases where the joint bound is tight, it permits update strengths of order β¹ᐟ², whereas the separable bound permits only order β for small budgets. The two can also coincide (Appendix A.4). This gap cannot be reproduced by a fixed rescaling of the separable budget.

Both bounds retain slack, including the meet approximation gap Δ(S) = B(S) − TC*(S) and the gap between the worst compatible coupling and a particular reference coupling (Appendix A.4). Since both remain conservative, the ablation below measures the decoding value of the additional update strength admitted by the joint reference bound.

Beyond LLaDA-1.5, we compared both bounds on all four backbones and benchmarks. Each cell gives Joint/Separable. ACC is in percent. Bold marks lower Joint accuracy.

Backbone / metricHumanEvalMBPPGSM8KMATH500
LLaDA-1.5 / ACC43.90/40.2440.40/38.2080.97/80.7438.60/38.20
LLaDA-1.5 / NFE64.76/64.4851.58/51.1461.39/61.5296.37/96.25
LLaDA-Instruct / ACC41.46/39.6340.20/39.6080.52/80.0638.00/38.00
LLaDA-Instruct / NFE64.02/64.9752.08/51.7562.39/62.6199.45/100.29
Dream / ACC60.37/59.7658.20/58.0076.12/76.1941.80/41.60
Dream / NFE49.87/50.8830.86/30.9749.45/50.59101.92/103.19
DiffuCoder / ACC68.90/68.2960.20/60.2048.45/49.739.40/7.80
DiffuCoder / NFE55.14/56.2041.84/42.0460.36/61.48107.30/107.53

Joint accuracy is higher in 12 of 16 settings at similar NFE, most consistently on code generation. The extra context per step remains small: at the same states, mean joint strength is 1.3–2.4 times the separable strength but stays below 0.04 in every setting. Across settings, 87–97% of tasks keep the same outcome. On GSM8K, Dream and DiffuCoder are 0.07 and 1.28 points lower.

Q2

How does confidence-based commitment ordering compare with left-to-right ordering?

CoCom does not depend on a particular commitment order. The order determines the accuracy-NFE tradeoff. The bound and prefix rule apply to any candidate order, and monotonicity of the set cost permits adaptive parallel commitment of the largest budget-feasible prefix (Appendix A.5). We use positional ordering by default, which keeps later predictions available as revisable pending context while earlier positions are resolved.

To test your proposed alternative, we evaluated confidence ordering on all four LLaDA-1.5 benchmarks, ranking unresolved positions within the active block by maximum predicted token probability with positional tie-breaking. Budgets, the joint bound, argmax writer, pending-context update and stopping criteria remain fixed. Confidence ordering reduces NFE on all four benchmarks and improves accuracy on GSM8K, while positional ordering yields higher accuracy on the two code benchmarks. NFE diff. = Confidence NFE - Positional NFE.

DatasetPositional ACC (%)Confidence ACC (%)NFE diff.
HumanEval43.9035.98-5.49
MBPP40.4037.60-1.59
GSM8K80.9783.02-4.13
MATH50038.6038.20-4.65

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.