acceptodds
Under review as a conference paper at ICLR 2027

Do We Need Dependency-Ready Decoding for Masked Diffusion Language Models ?

Abstract

Masked diffusion models (MDMs) have shown strong performance on structured reasoning tasks such as Sudoku, Trip Planning, and Countdown. MDMs introduce a freedom in the order in which tokens may be unmasked, and to exercise this freedom for good accuracy, early decoding strategies relied on signals of confidence or the downstream impact of decoding a token. However, simple examples seem to illustrate that resolving a token's dependencies before unmasking it should be important. Therefore, we examine whether this principle, which we call dependency-ready decoding, would contribute to better accuracy, from a practical and a theoretical perspective. We examine a recent decoder that implements dependency-readiness through attention, and generally find that when it performs better than confidence-based or future utility baselines, its decoding order is collapsing to left-to-right decoding. For theoretical analysis, we establish a definition of dependency without resorting to attention, measured by an information-theoretic score of how much revealing one masked token would shift the model's prediction for another. We prove that when the margin is large enough, no single revelation can change the top prediction, so in this regime max-margin decoding already commits dependency-ready tokens. These findings leave only a narrow scope for future dependency-ready decoders: their potential benefit is limited to positions with small margins, and reliance on attention should be questioned.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Discrete Diffusion models, Masked Diffusion Models, Diffusion Models, Dependency, Decoding, Reasoning, Inference-Time Strategy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V7bDOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether dependency-ready decoding, which reveals tokens in masked diffusion language models only after their unresolved dependencies have been resolved, offers advantages over conventional confidence-based decoding. Empirically, the authors analyze the attention-based groundedness measure used in DOS and find that, under the evaluated settings, first-layer DOS behaves almost identically to left-to-right decoding. They also show that EOS handling substantially affects performance comparisons with confidence-based baselines. Theoretically, the paper introduces a Pre-Commit score, μt(v)\mu_t(v), defined using the KL divergence between predictive distributions before and after revealing the ground-truth value of another masked token. The authors prove that the top-1 prediction at a given position remains unchanged when its prediction margin exceeds 2μt(v)\sqrt{2\mu_t(v)}. Based on this result, they argue that the potential additional benefits of dependency-ready decoding are limited to positions with small prediction margins.

Critical strengths
  1. Re-examining not only the performance of existing methods but also their underlying mechanisms is an important research direction. The finding that DOS(1) without the EOS fix achieves exactly the same reported accuracy as left-to-right decoding across all evaluated tasks for both models raises questions about whether the performance improvements of DOS can genuinely be attributed to dependency modeling.

  2. The proposition itself is sound and verifiable. The stability condition for a single token revelation, formulated in terms of the prediction margin and KL divergence, follows from Pinsker's inequality and is mathematically valid within this limited scope.

Critical weaknesses
  1. Overgeneralization of the Theoretical Result The proposition provides a sufficient condition based on the relative magnitude of the prediction margin and unresolved sensitivity, rather than the margin alone. Even with a large margin, the prediction may change if the sensitivity is sufficiently high. Therefore, the conclusion that the additional benefits of dependency-ready decoding are limited to small-margin positions is not theoretically justified.

  2. Missing Connection Between Prediction Stability and Decoding Order The proposition only guarantees that the top-1 prediction at a given position remains unchanged. It does not establish that max-margin decoding selects positions similar to those chosen by an oracle dependency-ready policy. Therefore, the claim that max-margin decoding already captures dependency readiness lacks a sufficient logical foundation.

  3. Insufficient Justification of Pre-Commit as an Ideal Readiness Measure Pre-Commit measures changes in predictions following the revelation of a single token, but it does not account for the joint effects of revealing multiple tokens or guarantee prediction correctness. In particular, Pre-Commit can be small even when an incorrect prediction remains unchanged after additional information is revealed. Thus, the justification for treating Pre-Commit as an ideal criterion for dependency readiness is insufficient.

Questions for authors

See the weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer V7bD

We thank the reviewer for the careful review.

W1. The relative margin condition.

We have worded our conclusions more carefully to emphasize the role of the Pre-Commit score and not just the margin in the criterion, and we added experiments to test how frequently the condition was satisfied at various ranges for the margin. Figure 6b in newly added Section 6.3 gives the full picture. The share of maximum-margin commits that a single reveal can flip falls from 45.6% at margin 0.3–0.5 to 36.7% at 0.5–0.7, 25.9% at 0.7–0.9 and 16.8% at 0.9–1.0. A larger margin makes a flip less likely but never rules it out, so the revision keeps only the condition relative to 2μ\sqrt{2\mu}.

W2. Maximum-margin decoding and stable positions.

We have removed the suggestion of an "oracle dependency-ready policy" and instead focused on predicting positions whose margin is sufficient to guarantee stability with respect to single token revelations. Our new empirical analysis of 9×99 \times 9 Sudoku in Section 6.3 shows that predicting such stable tokens (which is more likely when the criterion is satisfied) leads to higher accuracy, and also that maximum margin decoding tends to choose positions satisfying the criterion more often than DOS.

W3. Joint effects, Pre-Commit and correctness.

Joint effects can be addressed by the same argument, and we add this comment after the Proposition in Section 6.2. A set of kk revealed tokens defines a joint score μtk(v)\mu^k_t(v), and larger kk gives a stricter criterion on the margin.

Although stability with respect to single token revelations does not guarantee correctness, stable (including those satisfying the Pre-commit criterion) commits are far more often correct on 9×99 \times 9 Sudoku: 56–77%, against 8–21% of unstable ones (c.f. newly added Section 6.3), and unstable commits account for more than half of each decoder's errors.

Thank you for the detailed response and additional experiments. Many of my concerns have been substantially addressed, although the theoretical justification for the central claims remains limited. Nevertheless, I appreciate the improvements and have decided to raise my score to 2.

AuthorsOpenReview ↗

Response to Reviewer V7bD

We sincerely appreciate your thoughtful consideration and raising your score.  In the revision, we have taken more care to avoid overclaiming, and perhaps our revisions will be easier to find with the newly uploaded version marking many of them in red.  If you could point out with a line number which specific central claim you believe is still insufficiently justified, we will try to address the problem.

Reviewer_rPxyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes dependency-ready decoding for masked diffusion language models, focusing on DOS. It shows that DOS often follows a nearly left-to-right decoding order and that its reported advantages over confidence-based methods shrink after correcting EOS handling. The paper also provides a theoretical condition under which a token’s top prediction remains unchanged after revealing any single masked token.

Critical strengths

The comparisons across GSM8K-CoT, MATH500, HumanEval, MBPP, Trip Planning, Sudoku, and Countdown are informative. They highlight differences between tasks that benefit from left-to-right decoding and those that benefit from more flexible generation orders, with no single sampler consistently performing best across all tasks. Including a left-to-right baseline and comparing results with and without the EOS fix help clarify the sources of DOS’s apparent advantages.

Critical weaknesses

1.The scope of the conclusions is broader than the empirical evidence. Although the paper explains why DOS implements dependency-ready decoding, it is less clear whether DOS is sufficiently representative to support conclusions about the general principle. Its limitations may reflect the use of attention as a dependency proxy rather than a limitation of dependency-ready decoding itself. The paper should distinguish these two possibilities more clearly.

  1. While the paper focuses on unmasking order, its broader conclusions would benefit from a discussion of revocable decoding. Methods such as WINO [1] and DARD [2] revisit earlier predictions as more context becomes available, which is not covered by the single-token stability result. Could the authors clarify the scope of their conclusions relative to these methods?

[1] Hong et al. Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs. 2025. [2] Park et al. Dependency-Aware Revocable Decoding for Efficient Diffusion Large Language Model Inference. 2026.

Questions for authors

Could early token commitments constrain the placement of later content?

The paper discusses this issue for EOS tokens. Could a similar problem arise with other formatting tokens?

Similarly, formatting tokens may be easy to predict, but they may not be the tokens we want to prioritize. Fixing them early could limit the positions available for later content.

Does the proposed notion of dependency-readiness capture this potential drawback of confidence- or margin-based decoding?

3 comments
AuthorsOpenReview ↗

Response to Reviewer rPxy

We thank the reviewer for finding the cross-task comparison informative. We are also glad that the left-to-right baseline and the EOS ablation helped clarify where DOS's apparent gains come from. We answer each point below.

W1. Is DOS representative enough to support conclusions about the principle?

Please see point 1 in our general response to all reviewers.

W2. How do the conclusions relate to revocable decoding (WINO, DARD)?

Thank you for these references, which we have added to the Related Work section. We did not study revocable decoding strategies such as WINO and DARD because revocable decisions make it less important to select dependency-ready tokens. Our stability result still bears on them: a prediction that satisfies the criterion of Section 6.2 keeps its top-1 prediction after any single further revelation, so it is a prediction that a revocable decoder would not need to revise. Conversely, revocation can repair the unstable predictions that Section 6.3 finds for max-margin decoding.

Q1. Can early commitments of formatting tokens constrain later content?

We think this problem applies primarily to EOS or padding tokens. Such tokens can be predicted repeatedly to fill the entire tail of a response, leaving no remaining room for meaningful tokens. Newlines, indentation and answer markers (e.g., "####" in GSM8K) each occupy single positions. Fixing them early constrains where content goes, but does not remove space for it. This is why we studied EOS only. As far as we are aware, only the EOS token has a significant effect: the EOS fix alone moves min-entropy decoding by 24 points on LLaDA (MBPP) and by 47 points on Dream (GSM8K).

Q2. Does readiness capture this drawback?

Yes, at the level of the oracle definition. Suppose a confidence decoder considers committing EOS at position vv while the true response continues past vv. Then some later position uu has a non-EOS ground-truth token. For a model trained with end-of-sequence padding, revealing that token makes EOS at vv inconsistent, so Δt(v←u)\Delta_t(v \leftarrow u) and μt(v)\mu_t(v) are large. An oracle readiness decoder would therefore defer vv. If the response truly ends before vv, EOS at vv is correct, μt(v)\mu_t(v) is small, and committing it early is harmless. Confidence and margin cannot tell these two cases apart, because the EOS prediction can be confident in both. The same reasoning applies to any formatting token whose correct position depends on content that has not been revealed. The limitation is that μ\mu is an oracle quantity and does not define a practical readiness estimator.

Thank you for the detailed response and for highlighting that even DOS, a dependency-aware decoder, often generates tokens in a nearly left-to-right order. I would always expect a dependency-aware decoder to perform well on both reasoning tasks that favor left-to-right generation and tasks such as Sudoku that benefit from a more flexible order, and this study shows no method consistently performs best across both ,which is an interesting message to the community.

However, the EOS correction is already part of LLaDA’s default evaluation procedure, so I see limited novelty in the correction itself. And I am still concerned for the presentation for this paper.

So together, I value the finding about DOS and am raising my score from 2 to 2.5.

AuthorsOpenReview ↗

Response to Reviewer rPxy

We thank the reviewer for the follow-up and for reconsidering the score. We take the concern regarding the presentation seriously. We have just uploaded a revised PDF in which all major changes made following the rebuttal are highlighted in red for ease of reference. If any particular sections remain unclear or difficult to follow, we would greatly appreciate specific feedback so that we can prioritize further improvements.

Reviewer_UAfeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether dependency-aware decoding is necessary for masked diffusion language models (MDMs). The authors evaluate an existing attention based method, DOS, and find that its decoding order often closely follows left-to-right generation. They also show that applying an EOS correction improves confidence based baselines, reducing DOS’S previously reported advantages. The paper further introduces an information theoretic definition of token dependency and provides a theoretical condition where maximum margin decoding preserves the top prediction after revealing another token. I have several concerns on the paper’s motivation, evaluations, and the overall contributions.

Critical strengths
  1. This paper studies a very timely topic about the masked diffusion modelling for language.
  2. The paper investigates a relevant question about the necessity of dependency-aware decoding in MDMs.
  3. The paper conducts experiment to verify the effectiveness of the model.
Critical weaknesses
  1. Most of the empirical findings come from comparing a single existing work, DOS, with different decoding strategies, adding a left-to-right baseline, and correcting EOS handling. Given that the contribution is largely specific to an existing method, it is recommended for the authors to better justify what general knowledge about dependency-aware decoding is gained beyond identifying limitations in DOS.
  2. Additionally, since the paper uses DOS as the main practical implementation of dependency-ready decoding, and that it relies on attention scores to estimate dependencies. Showing that DOS performs poorly or behaves like left-to-right decoding primarily demonstrates a limitation of attention-based dependency estimation, not necessarily the dependency-ready decoding principle itself. Although the paper mentions specific conclusions on “reconsider reliance on attention”, the authors should be careful with the phrases (e.g., the title) to avoid overclaiming.
  3. The proposition in Section 6.2 shows that a sufficiently large prediction margin guarantees stability when any other masked token is revealed. However, this does not provide any insight that the maximum-margin decoding strategy selects dependency-ready tokens. Particularly, the stability guarantee depends on the unobserved Pre-Commit score (Line 388) and does not cover the combined effects of revealing multiple tokens, which makes this proposition relatively weak.
  4. Moreover, the theoretical and empirical contributions in section 5 and 6 appears to be disconnected. Where section 5 begins with investigating an attention-based DOS, then section 6 introduces a dependency measure based on KL divergence. However, the paper does not examine whether the proposed measure explains DOS's failures, or whether confidence-based methods better approximate dependency readiness, or if the measure predicts decoding accuracy.
  5. The experimental setting is a bit narrow to support the conclusions on the usefulness of dependency-ready decoding. Specifically, the empirical validation discussion in Lines 451-455 states that the decoding strategies was evaluated by revealing one token per step. However, dependencies between tokens are generally important when several positions are decoded simultaneously, which makes the evaluation less practical and raises questions on how the observation generalizes.
Questions for authors
  1. Beyond identifying limitations in DOS, what new general insight does this work establish about dependency-ready decoding in MDMs?
  2. The paper only evaluates two MDM (dLLM) backbones (LLaDA and Dream), which is insufficient to draw a concrete conclusion, does the behaviour of dependency-ready decoding change for more recent/better performing models like LLaDA 2.x series?
4 comments
AuthorsOpenReview ↗

Response to Reviewer UAfe [1/2]

We thank the reviewer for the detailed review. We agree with the core of W2. As described in the general response, we have narrowed the paper's claims and connected Sections 5 and 6. Our answers follow point by point.

W1 / Q1. What general insight goes beyond DOS?

To our knowledge, DOS is the only published decoder that ranks positions by readiness rather than by confidence or future utility. The drawbacks we identify in DOS pertain generally to its use of attention as a proxy for dependency to prioritize positions to be decoded.

  1. We characterize dependency readiness in terms of prediction stability instead of attention scores. The notion of Pre-Commit score, and the study of prediction flips, provide an alternative that we can use for analysis.
  2. We link stability with respect to single token revelations to the prediction margin. We prove (Proposition, Section 6.2) that if the top-1 margin exceeds a function of the Pre-Commit score, mt(v)>2μt(v)m_t(v) > \sqrt{2\mu_t(v)}, then revealing any single masked position at its ground-truth value cannot change the top-1 prediction at vv. The same argument extends to joint revelations of any kk positions, with μt(v)\mu_t(v) replaced by μtk(v)\mu_t^k(v)(cf our generalization of the Proposition in Section 6.2). The certificate identifies predictions that are provably ready. Empirically, no certified prediction flipped (0 of 3,777 revelations on 4×44 \times 4 Sudoku and 0 of 467 on 9×99 \times 9 Sudoku with LLaDA). After revision, in the added Section 6.3, we show that maximum-margin decoding commits certified tokens more often than DOS: 62.9% versus 47.4% of commits on 9×99 \times 9 Sudoku with Dream (Table 3).
  3. We untangle some confounding effects in previous analyses of decoding order. The EOS fix moves confidence baselines by up to 24 points on LLaDA (MBPP, min-entropy: 16.8→40.616.8 \to 40.6) and by 47 points on Dream (GSM8K, min-entropy: 35.6→82.635.6 \to 82.6). Any claim that a decoder benefits from order flexibility should also report a left-to-right baseline and the Kendall τ\tau distance to it. Without them, DOS(1)'s results would read as gains from dependency modeling. In fact, they equal left-to-right exactly on every task, for both models.

W2. DOS tests attention, not the principle.

Our results circumscribe the potential for dependency-ready decoding strategies rather than claim that they are unnecessary, and we have revised our wording, including the title, to avoid confusion. The analysis of Section 5 is meant to criticize the attention-based implementation of dependency-ready decoding. Section 6 introduces another sense of dependency-readiness in terms of prediction stability which may apply more broadly.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer UAfe [2/2]

(continued)

W3. About the Proposition

We address the reviewer's three points in turn.

  • It does not show that maximum-margin selects dependency-ready tokens. We add an empirical analysis of this question. 62.9% of maximum-margin commits are certified to be stable with respect to single token revelations, and 76.2% cannot be flipped by any single revelation. For left-to-right/DOS the fractions are only 47.4% and 59.5%. Max-margin also avoids the margins where flips concentrate. Only 0.4% of its commits have margin below 0.3, vs. 25.7% for left-to-right/DOS, whose commits in that range are flippable 84–98% of the time.
  • It depends on the unobserved Pre-Commit score. We acknowledge that our criterion cannot be checked at inference time without knowing the ground truth tokens. However, it is a useful theoretical tool for analyzing decoding strategies.
  • About covering joint revelations. Section 6.2 now extends the result to any set UU of kk revealed tokens: with μtk(v)\mu^k_t(v) the largest KL shift that revealing kk tokens causes at vv, mt(v)>2μtk(v)m_t(v) > \sqrt{2\mu^k_t(v)} guarantees that no such joint reveal changes the top-1 prediction.

W4. Sections 5 and 6 connection.

We now added experiments in Section 6.3 to audit Section 5's decoders with Section 6's quantities (general response, point 3). This answers the reviewer's three questions.

  • Does μ\mu explain DOS's failures? In part. DOS(1) commits left to right whatever the readiness of the next cell. 40.5% of its commits are flippable, vs. 23.9% for maximum-margin. It also commits many low-margin cells: 25.7% of its commits have margin below 0.3, and 84–98% of those are flippable. For maximum-margin, only 0.4% of commits fall below 0.3.
  • Do confidence methods approximate readiness better? Yes, on this measure. Their certified share is 59.7–62.9%, vs. 47.4% for DOS.
  • Does μ\mu predict accuracy? Accuracy is 76.8% for certified maximum-margin commits and 7.5% for flippable ones. At equal margin, left-to-right/DOS commits are no more fragile than maximum-margin's (margin 0.7–0.9: 26.3% vs. 25.9% flippable). DOS fails because its fixed order forces it to commit low-margin cells.

W5. Only one token is revealed per step.

Conflicts among simultaneously decoded tokens is an orthogonal question to readiness. If multiple tokens are decoded, all of them should first be ready for decoding. Particularly, none of their predictions should flip if other masked tokens were first revealed. We also extended the Proposition from conditioning on a single revealed token to conditioning on any set of revealed tokens.

Q2. Newer backbones.

The latest models in the LLaDA series are LLaDA 2.0, 2.1 and 2.2, but LLaDA 2.0 was not considered because of its dependence on block diffusion, and LlaDA 2.1 and 2.2 may revoke previous decoding decisions, making the notion of dependency readiness less relevant.

Thanks to the authors for the detailed response and analyses, and clarifying their contributions. However, I do not think my concerns have been adequately addressed to lean towards acceptance.

  1. While the authors have narrowed their claims, this also reduced the paper's contributions. The main empirical findings still focus on identifying limitations in DOS. While the author provides the relevant reasoning, I remain unconvinced about what general insights this work establishes regarding dependency-ready decoding beyond the limitations of this specific implementation.
  2. I appreciate the additional experiments connecting Sections 5 and 6. However, based on my understanding, the results largely show that maximum-margin decoding selects more stable predictions than DOS. The authors also show that, at equal margins, both strategies have similar prediction stability. This raises further questions about what the proposed KL-based dependency measure contributes beyond conventional confidence-based decoding, particularly since its stability certificate cannot be evaluated during inference.
  3. I am also not fully convinced by the response regarding newer LLaDA models. The authors exclude LLaDA 2.0 because of block diffusion and LLaDA 2.1/2.2 because they allow previous decoding decisions to be revised. However, these are important directions in modern diffusion language model decoding. Rather than simply justifying their exclusion, I believe the authors should better establish the significance and practical relevance of studying dependency readiness under the current restricted setting. This will actually provide a stronger and more meaningful contribution to existing research.

Also a general note, the authors mention that the manuscript has been revised. However, without highlighting or clearly indicating the changes in the revised PDF, it is difficult to see the changes.

AuthorsOpenReview ↗

Response to Reviewer UAfe

We thank the reviewer for thoughtfully considering our revisions. We address the three points below. We have also uploaded a revised PDF in which all changes are highlighted in red.

1. General insights beyond DOS

We would like to clarify that several of our findings extend beyond the specific DOS implementation:

  • Attention mass on the revealed context is a biased readiness signal. The argument in Section 5.3 applies to any readiness score based on ∑u∈RtAttn(v,u)\sum_{u \in R_t} \mathrm{Attn}(v,u), not only to DOS. In particular, RoPE's distance-dependent decay favors positions close to the revealed frontier. This bias is also reflected empirically: even in late layers, attention-based orderings on Sudoku have a Kendall's τ\tau distance of at most 0.02 from left-to-right generation for both models, despite Sudoku dependencies not following a left-to-right order.

  • The oracle analysis identifies and quantifies the headroom for better readiness estimation. Without relying on attention, our oracle analysis identifies commitments made by Maximum-Margin that are overturned by a subsequent revelation. On 9×9 Sudoku with Dream, a single ground-truth revelation overturns 24% of Maximum-Margin's commitments, and these commitments account for more than half of its errors (Table 3). This provides a concrete target for future readiness decoders: regardless of the estimator they use, they can be evaluated on their ability to avoid these premature commitments.

  • Dependency-aware decoding faces a task-dependent ordering trade-off. Our findings highlight a fundamental trade-off: the most suitable generation order depends on the task, and none of the evaluated methods consistently handles both left-to-right reasoning and reasoning that benefits from a more flexible generation order.

  • The EOS confound extends to all comparisons with confidence-based baselines. This issue is not specific to DOS; it affects comparisons against confidence-based decoding methods more broadly.

2. What the Pre-Commit score adds beyond margin

The Pre-Commit score is intended as an analytical yardstick, rather than as a practical decoding signal. Its purpose is to provide a sufficient condition for readiness based on the relationship between the prediction margin and the sensitivity to unresolved tokens. This gives us a way to certify a commitment as stable, rather than relying on confidence alone.

The score also allows us to quantify the difference between margin-based and attention-based readiness: the attention-based criterion certifies fewer commits than the margin criterion (47.4% vs. 62.9%).

Importantly, confidence margin alone does not capture readiness. Among Maximum-Margin's own commitments, 25.9% of those in the 0.7–0.9 margin bin and 16.8% of those in the 0.9–1.0 bin are overturned by a single revelation (Fig. 6b). Thus, two commitments with the same margin can differ in their actual readiness. Margin cannot distinguish between them by construction, whereas μt\mu_t accounts for the effect of unresolved tokens. Our proposition establishes a formal relationship between the two quantities.

3. Significance of the restricted setting We restricted the setting to make it a controlled experiment. Blocks, parallel commits, and revocation each change the decoding order for reasons other than readiness.  Focusing on decoders without these confounders gives our experiments easier interpretability.  For a decoder with revocable decoding decisions, the impact may be on efficiency, rather than on final accuracy.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the three reviewers for their careful reading. Three concerns come up across the reviews. We address them by narrowing the claims, strengthening the theory, and using Section 6 to audit the decoders of Section 5.

1. Proxy vs. principle (UAfe W1–W2, rPxy W1).

We agree that our experiments test dependency readiness only through attention (DOS). The revision separates two claims:

  • (A) About attention-based dependency-ready decoding. At the time of writing, DOS implements the purest form of attention-based dependency-ready decoding of which we are aware. Other recent dependency-aware decoders (DAWN (Luo et al., 2026), Cluster-Level Attention-Guided Parallel Decoding (Qi et al., 2026)) have emerged, which use dependency relations estimated by attention scores to avoid conflicts when tokens are simultaneously decoded, but the prioritization of tokens to decode is first driven by confidence. Their mixture of confidence scores and heuristics with thresholded attention scores introduces confounding phenomena which we avoid in our analysis.
  • (B) About any readiness decoder. Unless the max-margin position's margin is small relative to 2μt(v)\sqrt{2\mu_t(v)}, max-margin decoding already chooses a position that is dependency-ready, in the sense that its prediction would not be changed even if another masked position were revealed. Section 6 supports this claim, qualified by the new measurements.

To make it clear that our results circumscribe the potential for dependency-ready decoding rather than render it unnecessary, we retitle the paper "Pitfalls of Dependency-Ready Decoding for Masked Diffusion Language Models". We also revise the paper and the contributions accordingly.

2. Questions about applicability of the Proposition (UAfe W3, V7bD W1–W3).

  • What if multiple tokens are revealed (Section 6.2)? For a set U⊆Ft∖{v}U \subseteq F_t \setminus \{v\} of kk masked tokens, let

Δt(v←U)=DKL(pθ(Xv∣XU=xU⋆,Rt) ∥ pθ(Xv∣Rt)),μtk(v)=max⁡U⊆Ft∖{v}, ∣U∣=kΔt(v←U).\Delta_t(v \leftarrow U) = D_{\mathrm{KL}}\big(p_\theta(X_v \mid X_U = x^\star_U, R_t) \,\|\, p_\theta(X_v \mid R_t)\big), \qquad \mu^k_t(v) = \max_{U \subseteq F_t \setminus \{v\},\, |U| = k} \Delta_t(v \leftarrow U).

By the same argument as the Proposition, mt(v)>2μtk(v)m_t(v) > \sqrt{2\mu^k_t(v)} guarantees that revealing any kk masked tokens does not change the top-1 prediction; k=1k = 1 recovers the Proposition, and larger kk gives a stricter criterion.

  • Whether large margins are enough. Every statement about large margins is now clarified to be relative to the Pre-Commit score. Additionally, we introduce empirical analysis of the distribution of margins, whether the Proposition's criterion is satisfied, and whether the prediction is correct.
  • Correctness versus stability with respect to revelations. In the added experiments on 9×99 \times 9 Sudoku, commits that are stable under single-token revelations are far more often correct than unstable ones (56–77% vs. 8–21%), and unstable commits account for more than half of each decoder's errors. Satisfying the μ\mu criterion does not guarantee correctness, however: 22.7–26.9% of such commits are still wrong.

3. Sections 5 and 6 were not clearly connected (UAfe W4).

A new empirical subsection 6.3 examines four of the decoders of Section 5 with respect to the criterion of the Proposition, analyzing whether the margin condition is satisfied, the size of the margin, the stability of the prediction with respect to single-token revelations (flips), and the correctness of the prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.