acceptodds
Under review as a conference paper at ICLR 2027

Dependency-Aware Masking for Parallel Decoding in Masked Diffusion Language Models

Abstract

Masked Diffusion Language Models (MDLMs) offer a parallel alternative to autoregressive models by enabling parallel text generation. However, simultaneously unmasking multiple tokens relies on a strict conditional independence assumption. Standard training and sampling are both structure-agnostic, so strongly coupled syntactic pairs can be decoded jointly, introducing significant factorization errors that degrade text quality. To address this at training time, we propose Dependency-Aware Masking, a non-intrusive training formulation that utilizes linguistic dependency trees to replace the standard uniform noise schedule with a depth-aware soft prior. This biases the model to safely reconstruct conditionally independent peripheral modifiers before resolving the densely coupled syntactic core. At inference time, we pair it with a multi-token sampler that unmasks independent sets of an attention-derived interaction graph, avoiding the joint decoding of strongly coupled tokens. Extensive evaluations across MDLM and BD3-LM architectures demonstrate that our framework consistently outperforms uniform-masking baselines on unconditional generation, establishing a superior trade-off between generation quality and decoding efficiency.

Primary area:
generative models
Keywords:
masked diffusion language models, parallel decoding, dependency structure, non-uniform masking

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CyRSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how syntactic dependencies between tokens can be used to improve parallel decoding in masked diffusion language models. The authors propose Dependency-Aware Masking, which adjusts the masking probabilities during training based on the depth of tokens in a dependency tree. Tokens closer to the root are masked more often, while deeper tokens are masked less often. The goal is to encourage the model to generate peripheral tokens before the more central tokens.

At inference time, the authors combine this method with a modified DAPD sampler that uses attention scores to estimate dependencies between masked positions and avoids generating strongly connected tokens together. The method is evaluated on MDLM and BD3-LM using LM1B and OpenWebText. The results show improvements in generative perplexity over uniform masking. The paper also includes ablation studies on different masking strategies, samplers, generation order, and parser choices.

Critical strengths
  1. The main strength of the paper is that the proposed masking method is simple and improves generation quality without changing the model architecture or training objective. The authors only modify the masking probabilities based on dependency-tree depth, which makes the method easy to apply to existing masked diffusion language models. The results show consistent improvements in generative perplexity across MDLM and BD3-LM on both LM1B and OpenWebText. This makes the approach a potentially useful training strategy.

  2. I also find the experimental analysis useful. The authors compare their masking strategy with several alternatives, including random, positional, frequency-based, and reversed depth-based masking. They also evaluate the method with different samplers. These experiments help show that the improvement is not limited to their specific sampler. The analysis of attention-based dependencies and generation order also provides some evidence for the motivation behind the method.

Critical weaknesses
  1. My main concern is the novelty of the proposed method. Non-uniform masking has already been studied using different signals, such as token frequency, entropy, and attention. Here, the main difference is using dependency-tree depth to decide the masking probabilities. The sampler is also based on DAPD, with only a few modifications. The paper argues that this helps reduce dependencies between tokens generated in parallel, but the training method only changes the masking probability of each token independently. It does not directly control which tokens are generated together. I think the connection between the proposed masking strategy and the motivation about conditional independence is not fully clear. I would like to see stronger evidence that the improvements actually come from using syntactic dependencies rather than just applying a different non-uniform masking strategy.

  2. I also have concerns about the experimental results. The improvements in generative perplexity are relatively small, and the paper does not report variance across different runs. The results with different dependency parsers are also somewhat concerning. For example, on LM1B, using Stanza gives a perplexity of 56.11 compared to 53.33 with spaCy, while the uniform baseline is 56.63. So most of the improvement disappears when changing the parser under their sampler. I think this needs more explanation. Also, given how the masking weights are defined, it is not clear whether using the full dependency-tree depth is really necessary or simply masking the root tokens more often would give similar results. A simple baseline for this would be useful. Finally, the experiments are only on 110M-parameter models and unconditional generation, so it is difficult to know whether the method would also help on larger models or other tasks.

Questions for authors
  1. My main question is about the benefit of using the full dependency-tree depth. Given how the masking weights are defined, it seems that root tokens receive most of the additional masking weight, while the differences between other depths are relatively small. Have you compared this with a simpler approach that only masks root tokens more often? I think this comparison is important to understand whether the full dependency-tree structure is really needed.

  2. The improvements in generative perplexity are relatively small in some cases. Are these results consistent across different training runs? Could you report standard deviations or confidence intervals for the main results in Table 1? This would help me understand how reliable the improvements are, especially compared to the other non-uniform masking strategies in Table 4.

  3. I would also like to understand the results with different dependency parsers in Table 5. With Stanza, the improvement under your sampler is much smaller than with spaCy. Why do you think this happens? Also, were the sampler hyperparameters tuned using the spaCy model, or were they selected independently for each parser? I think this is important for understanding how sensitive the method is to the parser choice.

4 comments
AuthorsOpenReview ↗

Response to W1 and W2

We thank the reviewer for the clear summary and the pointed questions.

W1: Novelty, and the link between masking and conditional independence

Thank you for these comments.

Non-uniform masking. We agree. Our contribution is the depth signal itself. We re-ran Table 4 with three seeds (the paper reports one) and added two controls: Shuffled-Depth (depths randomly reassigned within each sentence) and the Root-Only baseline you suggested in Q1 (only the root up-weighted). MDLM/LM1B, mean ± std:

Training-time maskingGen. PPL
Uniform56.73 ± 0.28
Frequency-Based56.46 ± 0.60
Linear Position55.11 ± 1.11
Random Perturbation54.90 ± 0.54
Root-Only55.40 ± 0.85
Shuffled-Depth54.81 ± 1.35
Dependency-Aware (ours)53.57 ± 0.58

Dependency depth gives the lowest Gen. PPL and beats both controls in every seed.

DAPD. Our sampler builds on the core idea of DAPD (Kim et al., 2026a): an attention-derived interaction graph from which an independent set of masked positions is unmasked in parallel. Our two main changes align the sampler with the order learned by Dependency-Aware Masking. First, candidates are ranked by confidence alone, because DAPD's degree-weighted priority favors the highly connected core tokens that our masking teaches the model to resolve last. Second, a per-step cap K prevents committing a very large independent set in one step. There are also smaller implementation differences (Appendix A.2). Appendix F.2 compares our sampler with the released DAPD implementation (MDLM, LM1B, NFE ≈ 80, Gen. PPL, single seed):

SamplerUniform maskingDependency-Aware masking
DAPD (released)59.2657.20
Ours56.6353.33

Our sampler is better under both masking schemes, and the gain from Dependency-Aware Masking grows from 2.06 under DAPD to 3.30 under ours, consistent with the sampler being designed around the masking prior.

Masking and conditional independence. We agree that masking changes per-token probabilities and does not by itself decide which tokens are decoded together. That is decided by the inference-time independent-set rule. What masking does change is the generation order: under the same sampler, the generation-order score AwA_w rises from 0.531 (Uniform) to 0.710 (Dependency-Aware) in Table 3, so the tokens available for parallel decoding at each step follow the dependency tree more closely.

W2: Experimental results

Thank you for these concerns. We answer the variance, parser and Root-Only points under Q1–Q3 below. Training-side results are limited to 110M models, since billion-scale pretraining is beyond our budget. The sampler needs no retraining, so we evaluated it on the off-the-shelf LLaDA-8B-Instruct with the official setup:

SamplerGSM8K (5-shot) Exact Match / NFEHumanEval (0-shot) pass@1 / NFEMBPP (3-shot) pass@1 / NFEIFEval (0-shot) Prompt / Inst. Strict / NFE
Official (256 steps)77.3 / 25639.6 / 25636.4 / 25648.1 / 58.8 / 256
Ours75.7 / 14240.2 / 13536.6 / 5650.1 / 60.4 / 155

Our sampler uses 39–78% fewer NFE, with equal or higher accuracy on all tasks except GSM8K. We will add this evaluation to the appendix.

We also evaluate prefix-conditioned generation on LM1B: given the first 32 of 128 tokens, each model generates the remaining 96 with our sampler:

BackboneGen. PPL (Uniform → Ours)Entropy (Uniform → Ours)
MDLM77.26 → 74.924.136 → 4.141
BD3-LM83.66 → 81.884.130 → 4.141

Dependency-Aware Masking lowers Gen. PPL for both backbones while slightly increasing entropy, so the improvement extends to conditional generation.

AuthorsOpenReview ↗

Response to Q1-Q3

W2 and Q1: Is full tree depth necessary, compared with Root-Only?

Thank you for suggesting this baseline. By construction, the extra weight is not concentrated on the root. The raw weight w~i=1−di/max⁡jdj\tilde{w}_i = 1 - d_i / \max_j d_j is linear in depth, so each level lowers it by the same amount. Interpolating with the uniform weight and rescaling each sequence to mean 1 preserve this spacing.

We also trained the suggested Root-Only baseline (only the root up-weighted, same masking budget) during the rebuttal period. MDLM/LM1B, three sampling seeds:

Training-time maskingGen. PPL
Uniform56.73 ± 0.28
Root-Only55.40 ± 0.85
Dependency-Aware53.57 ± 0.58

Root-Only recovers only part of the gain, and Dependency-Aware is lower in all three seeds.

W2 and Q2: Variance across runs

Thank you for this suggestion. Retraining each model several times was beyond our compute budget during the rebuttal period, so we report the variance over three sampling seeds of the same checkpoint (same number of samples as in the paper, mean ± std):

BackboneMaskingLM1B Gen. PPL (n=300)OWT Gen. PPL (n=100)
MDLMUniform56.73 ± 0.2825.39 ± 0.76
MDLMDependency-Aware53.57 ± 0.5823.20 ± 0.24
BD3-LMUniform61.62 ± 1.2243.74 ± 2.61
BD3-LMDependency-Aware58.64 ± 1.1439.28 ± 1.41

Dependency-Aware Masking lowers Gen. PPL in all four settings, by 2.2–4.5, more than the seed std in each case. Table 4 alternatives with three sampling seeds: Random Perturbation 54.90 ± 0.54, Linear Position 55.11 ± 1.11, Frequency 56.46 ± 0.60, all above Dependency-Aware (53.57 ± 0.58).

W2 and Q3: Parser sensitivity

Thank you for the question. We see two likely reasons for the smaller gain with Stanza. First, the sampler hyperparameters were tuned on the spaCy checkpoint and reused for Stanza without re-tuning. Second, the two parsers use different annotation schemes (spaCy's own scheme vs. Universal Dependencies for Stanza), which assign different depths to some words and therefore teach the model somewhat different generation orders. With three sampling seeds (MDLM, LM1B, our sampler, mean ± std):

Training-time maskingGen. PPL
Uniform56.73 ± 0.28
Dependency-Aware, spaCy53.57 ± 0.58
Dependency-Aware, Stanza55.26 ± 0.74

Both parsers lower Gen. PPL relative to Uniform, with a smaller reduction for Stanza. We have revised Section 5.4 to report this.

Thank you for the additional experiments. The Root-Only and Shuffled-Depth controls address an important part of my concerns and strengthen the evidence for using syntactic depth. However, variability across independent training runs remains untested, and the connection to reduced factorization error is still indirect. On balance, the stronger empirical evidence leads me to update my rating to weak acceptance.

AuthorsOpenReview ↗

Thank You for the Follow-up

Thank you very much for your follow-up and for raising your score. We truly appreciate your constructive feedback.

  • Factorization error. We agree that the link is indirect. One supporting observation is in Appendix F.1: with top-kk samplers, which do not avoid co-decoding dependent tokens, halving the step budget raises Gen. PPL by 93–113 for Uniform but only by 26–47 for Dependency-Aware Masking. This is consistent with reduced factorization error, and we will discuss it in the revised paper.
  • Independent training runs. With limited compute, we prioritized the new Root-Only and Shuffled-Depth controls during the rebuttal, and we plan to add independent runs in the final version.

Thank you again for your time and valuable feedback.

Reviewer_KZDZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

As I understand it, the paper examines factorization errors that arise when masked diffusion language models (MDLMs) generate strongly dependent tokens in parallel. Its main proposal, Dependency-Aware Masking, uses dependency-tree depth to adjust each token's masking probability during training. By masking the shallow syntactic core more often than peripheral tokens, the authors aim to encourage the model to generate peripheral context first and use it to reconstruct the core. For decoding, they use attention weights to build an interaction graph and select an independent set of tokens to generate together, aiming to separate strongly dependent tokens across steps. The approach is described as changing the masking and sampling procedures while keeping the model architecture and the form of the denoising objective unchanged. The experiments with MDLM and BD3-LM on LM1B and OpenWebText seem to support the authors' claim that this approach improves the trade-off between generation quality and decoding efficiency compared with uniform masking.

Critical strengths

S1. I find the training-side comparison in Table 1 particularly useful. With the sampler held fixed and NFE budgets matched, Dependency-Aware Masking improves Gen. PPL in all four combinations of MDLM/BD3-LM and LM1B/OpenWebText. For example, on MDLM/OpenWebText, Gen. PPL decreases from 25.74 to 23.92. Table 2 also reports improvements with the Ancestral and Entropy samplers on MDLM/LM1B. These results suggest that the proposed masking policy helps even without the authors' sampler, which is my main reason to support acceptance.

Critical weaknesses

W1. The comparisons in Section 5.3 and Table 4 suggest that dependency depth is useful, but I am not fully convinced that the gains depend on how the depth-derived weights correspond to the syntactic structure. Table 4 reports lower Gen. PPL for Dependency-Aware Masking than for the alternative masking policies, but I could not find a control that preserves the distribution of depth-derived weights while shuffling their assignment to tokens. This would test whether shuffling the weights across tokens reduces the improvement. Without that comparison, I find the claim that "the improvement comes from syntactic depth specifically rather than from non-uniformity alone" stronger than the evidence supports. This leaves me uncertain about how much of the improvement can be attributed to the syntactic information.

W2. The conclusion claims improvements in generation quality, but the main quantitative evaluation uses Gen. PPL from GPT-2-Large, with entropy as a diversity reference. For example, on MDLM/OpenWebText, Gen. PPL decreases from 25.74 to 23.92, while entropy decreases from 5.29 to 5.22. I would not take this small entropy decrease as evidence of quality degradation or collapse. However, these metrics do not tell me whether the generated text is more natural or coherent while retaining diversity. I would like to see whether readers also prefer the samples with lower Gen. PPL, or whether the improvement remains when diversity is matched.

Questions for authors

Q1. Could the authors discuss how Dependency-Aware Masking differs from LoMDM in Hong et al., "Unifying Masked Diffusion Models with Various Generation Orders and Beyond" (2026)? LoMDM jointly learns the generation order and diffusion backbone, whereas this paper uses dependency-tree depth as a fixed training prior. I would like to understand what the syntactic prior offers compared with learning the order.

Q2. Could the authors check the publication year of Wang et al., "Remasking Discrete Diffusion Models with Inference-Time Scaling"? The reference on p. 13 lists 2026, but the official proceedings list NeurIPS 2025.

Q3. In Section 3, is the posterior expression immediately after Eq. (1) intended only for the masked case, z_t = m? For an already unmasked token, I would expect z_s = z_t with probability one under the absorbing process. Could the expression state these two cases explicitly?

Q4. In Section 4.2, could the masking-probability formula and the forward marginal explicitly account for clipping? As written, (1 - alpha_t)w_i can exceed one when w_i > 1, giving a negative coefficient for the clean token in the categorical distribution. Could the authors use the effective, clipped masking probability in both expressions, or state that the displayed formulas apply only when clipping is inactive?

Q5. In Section 4.1, what does "they cannot attend to one another" mean? I would expect masked positions to exchange information through self-attention, even though they cannot see each other's sampled token values within the same decoding step. Is the intended point that the token values are sampled independently from the predicted distributions, as shown in Eq. (2)?

Q6. How is a word's unmasking step defined for the generation-order score in Section 5.2? If its subword tokens are unmasked at different steps, do the authors use the first step, the last step, or another aggregation? This choice could affect the parent-child ordering measured by Eq. (3).

Q7. Could the authors give the exact formula used to compute the Entropy values in Table 1? In particular, is the underlying distribution estimated from token frequencies within each generated sample? Please also specify the logarithm base and how the per-sample values are combined.

Q8. In Section 5.4, could the authors clarify why the parser comparison indicates that the gains stem from "the coarse tree structure, not fine-grained parsing accuracy"? Table 5 shows that Stanza-derived masking also improves over Uniform, but I do not see a measurement of parsing accuracy on these data or a controlled change to parsing accuracy. Would it be more accurate to conclude that the method improves over Uniform with either parser?

4 comments
AuthorsOpenReview ↗

Response to W1 and W2

We thank the reviewer for the detailed and constructive review.

W1: Does the gain depend on how weights correspond to syntax?

Thank you for suggesting this control. We agree that the existing comparisons do not fully isolate the contribution of syntactic alignment. We added the suggested Shuffled-Depth control, which randomly reassigns depth values to tokens within each sentence (keeping each sentence's depth histogram). We compare it on MDLM/LM1B with the same sampler at matched NFE over three sampling seeds:

Training-time maskingGen. PPL (our sampler, NFE≈80)Entropy
Uniform56.73 ± 0.284.24
Shuffled-Depth54.81 ± 1.354.23
Dependency-Aware53.57 ± 0.584.23

Shuffled-Depth improves over Uniform but remains worse than Dependency-Aware Masking in all three seeds, at the same entropy. Since it keeps the same weight distribution, part of the improvement comes from assigning the weights according to syntax.

W2: Do the metrics reflect quality while retaining diversity?

Thank you for this suggestion. We agree that Gen. PPL alone does not fully characterize sample quality. We report Gen. PPL and entropy for MDLM on LM1B using our sampler at matched NFE, averaged over three sampling seeds.

Training-time maskingGen. PPL ↓Entropy
Uniform56.73 ± 0.284.24
Dependency-Aware53.57 ± 0.584.23

The measured Gen. PPL reduction is consistent across the three sampling seeds.

LLM-judge evaluation. Since a human study is not feasible within the rebuttal period, we use an LLM judge as a proxy for reader preference. GPT-5-mini rates each sample on five criteria without knowing which model produced it, and we average the five scores. We score 300 LM1B samples per model:

BackboneUniformOurs
MDLM2.8652.930
BD3-LM2.7972.822

Our samples score higher on both backbones, so the lower Gen. PPL is not accompanied by a drop in judged quality. We will include the scoring prompt and full results in the revised appendix.

Prefix-conditioned generation. We also evaluate prefix-conditioned generation on LM1B: given the first 32 of 128 tokens, each model generates the remaining 96 with our sampler:

BackboneGen. PPL (Uniform → Ours)Entropy (Uniform → Ours)
MDLM77.26 → 74.924.136 → 4.141
BD3-LM83.66 → 81.884.130 → 4.141

Dependency-Aware Masking lowers Gen. PPL for both backbones while slightly increasing entropy.

AuthorsOpenReview ↗

Response to Q1-Q8

Q1: Relation to LoMDM

Thank you for the pointer to LoMDM. Both methods bias the generation order through a per-position masking schedule. They differ in how the schedule is obtained:

AspectLoMDM (Hong et al., ICML 2026)Ours
ScheduleLearned jointly, context-dependentFixed, from a dependency parse (offline)
Extra parametersTwo small scheduler networks (forward and reverse)None
Training cost+1 backbone forward pass per stepUnchanged
SamplingLearned reverse scheduleAny sampler

LoMDM is more flexible. The two are complementary, since a syntactic prior could initialize or regularize a learned schedule. We will add this comparison to the Related Work.

Q2: Citation year

Thank you for catching this. This is a typo. We have corrected the reference to NeurIPS 2025.

Q3: The posterior after Eq. (1)

Thank you for pointing this out. Yes, the expression applies only to the masked case, and we have revised Section 3 to state both cases explicitly:

  • if zt=mz_t=m: q(zs∣zt,x)=Cat(zs;1−αs1−αtm+αs−αt1−αtx)q(z_s\mid z_t,x)=\mathrm{Cat}\left(z_s;\tfrac{1-\alpha_s}{1-\alpha_t}m+\tfrac{\alpha_s-\alpha_t}{1-\alpha_t}x\right),
  • if zt≠mz_t\neq m: zs=ztz_s=z_t.

Q4: Clipping in Sec. 4.2

Thank you for catching this. You are right that (1−αt)wi(1-\alpha_t)w_i can exceed 1 when wi>1w_i > 1. Our implementation already clips it: token ii is masked with probability pti=min⁡(1,(1−αt)wi)p_t^i=\min(1,(1-\alpha_t)w_i), so the clean-token coefficient 1−pti1-p_t^i is never negative. We have revised Sec. 4.2 and Algorithm 1 to use this clipped probability in both expressions: q(zti∣xi)=Cat(zti;(1−pti)xi+ptim)q(z_t^i\mid x^i)=\mathrm{Cat}(z_t^i;(1-p_t^i)x^i+p_t^i m). Clipping rarely matters in practice: on LM1B, wi≤1.08w_i\le 1.08 at λ=0.125\lambda=0.125, so clipping is active only when 1−αt>1/1.08≈0.931-\alpha_t > 1/1.08 \approx 0.93, i.e., near the end of the forward process, when almost every token is already masked. Below that, pti=(1−αt)wip_t^i=(1-\alpha_t)w_i and the formulas are unchanged.

Q5: "They cannot attend to one another"

Thank you for the question. Our original wording was unclear, and your understanding is exactly right. Masked positions do exchange information through self-attention, so each position's predicted distribution depends on the full context. What they cannot do is condition on each other's sampled values: tokens unmasked in the same step are sampled independently from these distributions (Eq. 2). This is the parallel-decoding error our method aims to reduce. We have rewritten the sentence in Sec. 4.1 to state this precisely.

Q6: Word-level unmasking step for AwA_w

Thank you for the question. We use the earliest step at which any subword is unmasked (now stated in Sec. 5.2). Using the mean or last step instead barely changes AwA_w:

AggregationDependency-Aware + oursUniform + ours
Earliest (used)0.70950.5311
Mean0.71140.5332
Last0.71240.5349

Across all four Table 3 settings, AwA_w changes by at most 0.012 and the ranking is unchanged.

Q7: Entropy definition

Thank you for the question. Yes. Each sample is decoded and retokenized with GPT-2 (special tokens such as [CLS] kept as text), and we compute H=−∑vp(v)ln⁡p(v)H=-\sum_v p(v)\ln p(v) in nats, with p(v)p(v) the token frequency within the sample, averaged over samples. Gen. PPL is computed at corpus level under GPT-2-Large (exponentiated mean token NLL, excluding each sample's first token and EOS). Both definitions are now in Appendix C.

Q8: Parser-accuracy claim

Thank you for pointing it out. We agree and have removed this claim. Section 5.4 now states only that the method lowers Gen. PPL relative to Uniform with either parser, and notes that the sampler hyperparameters were tuned on the spaCy checkpoint and reused for Stanza without re-tuning.

Thank you for the detailed rebuttal and additional experiments.

The Shuffled-Depth comparison directly addresses my main concern and provides stronger evidence that syntactic depth contributes to the improvements beyond non-uniform masking alone. I also appreciate the additional quality evaluation and the clarifications provided in response to my questions.

Some uncertainty remains, particularly regarding the relatively small gain over Shuffled-Depth and the lack of independent training runs. Nevertheless, given the consistent improvements reported in the paper and the new evidence provided in the rebuttal, I am updating my rating.

AuthorsOpenReview ↗

Thank You for the Follow-up

Thank you very much for your thoughtful follow-up and for raising your score. We sincerely appreciate the time and care you put into reviewing our paper, and your suggestions have helped us improve it substantially.

We agree that the gap over Shuffled-Depth is moderate, although Dependency-Aware Masking is lower in all three seeds. Regarding independent training runs, with our limited compute, we prioritized training the new Shuffled-Depth control during the rebuttal period, so we were unable to complete multiple runs. We fully agree that they are important and plan to add them for the main MDLM/LM1B comparison in the final version.

Thank you again for your time and thoughtful review.

Reviewer_R749OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the challenges of parallel generation in masked diffusion models, the authors introduce a dependency-aware masking framework. By leveraging linguistic dependency parse trees, the method dynamically computes token-specific masking probabilities. Furthermore, it incorporates an attention-guided independent-set sampler during inference to break dependency loops and avoid decoding mutually dependent tokens in a single step. Extensive experiments using MDLM and BD3-LM on LM1B and OpenWebText demonstrate that this structural prior consistently lowers generative perplexity over uniform-masking strategies.

Critical strengths
  1. The paper addresses an interesting and largely unresolved problem in MDLMs: independently sampling tokens that may have strong dependencies. Addressing this issue has the potential to improve both decoding efficiency and generation quality.
  2. The paper provides thorough ablation studies to investigate the effects of key design choices and hyperparameters, offering useful insights into the proposed method.
  3. The method leverages syntactic dependency trees to model structural relationships between tokens, providing a practical and interpretable approach to addressing token dependencies during parallel decoding.
Critical weaknesses
  1. The evaluation is limited to relatively small-scale models with 110M parameters, which may be considered outdated for a 2027 paper. It would strengthen the paper to evaluate the proposed method on more recent MDLMs and more challenging tasks, rather than relying primarily on perplexity and entropy as evaluation metrics.
  2. The theoretical implications of modifying the masking distribution without preserving the original ELBO interpretation remain unclear. The paper would benefit from a more rigorous discussion of how this modification affects the training objective.
Questions for authors

Sea weaknesses

4 comments
AuthorsOpenReview ↗

Response to W1 and W2

We thank the reviewer for recognizing the importance of the problem and the thoroughness of our ablations.

W1: Model scale, recent MDLMs, and harder tasks

Thank you for this suggestion. We agree that larger models and harder tasks would strengthen the paper.

Larger model. Pretraining a billion-parameter MDLM with Dependency-Aware Masking is beyond our compute budget, so our training-side results remain at 110M. Our sampler needs no retraining, so we evaluated it on the off-the-shelf LLaDA-8B-Instruct model.

Both samplers share the checkpoint and lm-evaluation-harness setup. The baseline is the official low-confidence remasking sampler with 256 steps:

SamplerGSM8K (5-shot) Exact Match / NFEHumanEval (0-shot) pass@1 / NFEMBPP (3-shot) pass@1 / NFEIFEval (0-shot) Prompt / Inst. Strict / NFE
Official (256 steps)77.3 / 25639.6 / 25636.4 / 25648.1 / 58.8 / 256
Ours75.7 / 14240.2 / 13536.6 / 5650.1 / 60.4 / 155

Our sampler uses 39–78% fewer NFE, with equal or higher accuracy on all tasks except GSM8K. We will add this evaluation to the appendix.

Conditional generation. We also evaluate prefix-conditioned generation on LM1B: given the first 32 of 128 tokens, each model generates the remaining 96 with our sampler:

BackboneGen. PPL (Uniform → Ours)Entropy (Uniform → Ours)
MDLM77.26 → 74.924.136 → 4.141
BD3-LM83.66 → 81.884.130 → 4.141

Dependency-Aware Masking lowers Gen. PPL for both backbones while slightly increasing entropy, so the improvement extends to conditional generation.

LLM-judge evaluation. GPT-5-mini scores of LM1B samples:

BackboneUniformOurs
MDLM2.8652.930
BD3-LM2.7972.822

Our samples score higher on both backbones. We will include the scoring prompt and full results in the revised appendix.

W2: Theoretical implications of modifying the masking distribution

Thank you for raising this point. Dependency-Aware Masking changes which tokens are masked but keeps the standard MDLM loss, which remains the correct NELBO weighting. For a per-token schedule αti\alpha_t^i, the continuous-time NELBO (Shi et al., 2024) weights the loss on each masked token by (αti)′1−αti\frac{(\alpha_t^i)'}{1-\alpha_t^i}. Our masking sets 1−αti=(1−αt)wi1-\alpha_t^i=(1-\alpha_t)w_i, where wiw_i depends only on the parse and not on tt. Hence (αti)′=αt′wi(\alpha_t^i)'=\alpha_t' w_i, and

(αti)′1−αti=αt′wi(1−αt)wi=αt′1−αt.\frac{(\alpha_t^i)'}{1-\alpha_t^i}=\frac{\alpha_t' w_i}{(1-\alpha_t) w_i}=\frac{\alpha_t'}{1-\alpha_t}.

The weight wiw_i cancels, so each masked token receives the same coefficient as in standard MDLM. Instead, wiw_i only changes which tokens are masked, i.e., the distribution over which the loss is averaged. The only exception is clipping, which on LM1B (wi≤1.08w_i \le 1.08 at λ=0.125\lambda=0.125) occurs only above a base masking probability of about 0.93.

However, the objective is not an exact ELBO for our sampler, because ww is computed by parsing the clean sequence and is therefore unavailable at generation. We will discuss this in the revised appendix. Empirically, Dependency-Aware Masking consistently lowers Gen. PPL at matched sampler and NFE (Table 1).

AuthorsOpenReview ↗

Summary of Our Responses

Dear Reviewer R749,

Thank you again for your time and valuable feedback. As the discussion period is coming to an end, we would like to kindly summarize how we addressed your two concerns:

  • W1: We evaluated our sampler on LLaDA-8B-Instruct, where it uses 39–78% fewer NFE with equal or higher accuracy on all tasks except GSM8K. We also added prefix-conditioned generation and an LLM-judge evaluation, and both favor Dependency-Aware Masking.
  • W2: We showed that wiw_i cancels in the per-token NELBO weight, so our training objective keeps the same weighting as standard MDLM, except in rare cases where clipping is active at very high masking levels.

During the rebuttal period, we also re-ran our main comparisons with three sampling seeds and added two controls (Shuffled-Depth and Root-Only). Dependency-Aware Masking remains the best in every seed, which further supports that the gains come from syntactic depth.

We would be very grateful to hear whether our responses address your concerns, and we would be happy to clarify anything further.

Thanks for the detailed response and providing results on larger models. I would increase my score to weak accept

AuthorsOpenReview ↗

Thank You for the Follow-up

Thank you very much for your follow-up and for raising your score. Your insightful comments on model scale and the training objective helped us strengthen the paper.

We sincerely appreciate your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.