- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how syntactic dependencies between tokens can be used to improve parallel decoding in masked diffusion language models. The authors propose Dependency-Aware Masking, which adjusts the masking probabilities during training based on the depth of tokens in a dependency tree. Tokens closer to the root are masked more often, while deeper tokens are masked less often. The goal is to encourage the model to generate peripheral tokens before the more central tokens.
At inference time, the authors combine this method with a modified DAPD sampler that uses attention scores to estimate dependencies between masked positions and avoids generating strongly connected tokens together. The method is evaluated on MDLM and BD3-LM using LM1B and OpenWebText. The results show improvements in generative perplexity over uniform masking. The paper also includes ablation studies on different masking strategies, samplers, generation order, and parser choices.
Critical strengths
-
The main strength of the paper is that the proposed masking method is simple and improves generation quality without changing the model architecture or training objective. The authors only modify the masking probabilities based on dependency-tree depth, which makes the method easy to apply to existing masked diffusion language models. The results show consistent improvements in generative perplexity across MDLM and BD3-LM on both LM1B and OpenWebText. This makes the approach a potentially useful training strategy.
-
I also find the experimental analysis useful. The authors compare their masking strategy with several alternatives, including random, positional, frequency-based, and reversed depth-based masking. They also evaluate the method with different samplers. These experiments help show that the improvement is not limited to their specific sampler. The analysis of attention-based dependencies and generation order also provides some evidence for the motivation behind the method.
Critical weaknesses
-
My main concern is the novelty of the proposed method. Non-uniform masking has already been studied using different signals, such as token frequency, entropy, and attention. Here, the main difference is using dependency-tree depth to decide the masking probabilities. The sampler is also based on DAPD, with only a few modifications. The paper argues that this helps reduce dependencies between tokens generated in parallel, but the training method only changes the masking probability of each token independently. It does not directly control which tokens are generated together. I think the connection between the proposed masking strategy and the motivation about conditional independence is not fully clear. I would like to see stronger evidence that the improvements actually come from using syntactic dependencies rather than just applying a different non-uniform masking strategy.
-
I also have concerns about the experimental results. The improvements in generative perplexity are relatively small, and the paper does not report variance across different runs. The results with different dependency parsers are also somewhat concerning. For example, on LM1B, using Stanza gives a perplexity of 56.11 compared to 53.33 with spaCy, while the uniform baseline is 56.63. So most of the improvement disappears when changing the parser under their sampler. I think this needs more explanation. Also, given how the masking weights are defined, it is not clear whether using the full dependency-tree depth is really necessary or simply masking the root tokens more often would give similar results. A simple baseline for this would be useful. Finally, the experiments are only on 110M-parameter models and unconditional generation, so it is difficult to know whether the method would also help on larger models or other tasks.
Questions for authors
-
My main question is about the benefit of using the full dependency-tree depth. Given how the masking weights are defined, it seems that root tokens receive most of the additional masking weight, while the differences between other depths are relatively small. Have you compared this with a simpler approach that only masks root tokens more often? I think this comparison is important to understand whether the full dependency-tree structure is really needed.
-
The improvements in generative perplexity are relatively small in some cases. Are these results consistent across different training runs? Could you report standard deviations or confidence intervals for the main results in Table 1? This would help me understand how reliable the improvements are, especially compared to the other non-uniform masking strategies in Table 4.
-
I would also like to understand the results with different dependency parsers in Table 5. With Stanza, the improvement under your sampler is much smaller than with spaCy. Why do you think this happens? Also, were the sampler hyperparameters tuned using the spaCy model, or were they selected independently for each parser? I think this is important for understanding how sensitive the method is to the parser choice.
4 comments
Response to Q1-Q3
W2 and Q1: Is full tree depth necessary, compared with Root-Only?
Thank you for suggesting this baseline. By construction, the extra weight is not concentrated on the root. The raw weight is linear in depth, so each level lowers it by the same amount. Interpolating with the uniform weight and rescaling each sequence to mean 1 preserve this spacing.
We also trained the suggested Root-Only baseline (only the root up-weighted, same masking budget) during the rebuttal period. MDLM/LM1B, three sampling seeds:
| Training-time masking | Gen. PPL |
|---|---|
| Uniform | 56.73 ± 0.28 |
| Root-Only | 55.40 ± 0.85 |
| Dependency-Aware | 53.57 ± 0.58 |
Root-Only recovers only part of the gain, and Dependency-Aware is lower in all three seeds.
W2 and Q2: Variance across runs
Thank you for this suggestion. Retraining each model several times was beyond our compute budget during the rebuttal period, so we report the variance over three sampling seeds of the same checkpoint (same number of samples as in the paper, mean ± std):
| Backbone | Masking | LM1B Gen. PPL (n=300) | OWT Gen. PPL (n=100) |
|---|---|---|---|
| MDLM | Uniform | 56.73 ± 0.28 | 25.39 ± 0.76 |
| MDLM | Dependency-Aware | 53.57 ± 0.58 | 23.20 ± 0.24 |
| BD3-LM | Uniform | 61.62 ± 1.22 | 43.74 ± 2.61 |
| BD3-LM | Dependency-Aware | 58.64 ± 1.14 | 39.28 ± 1.41 |
Dependency-Aware Masking lowers Gen. PPL in all four settings, by 2.2–4.5, more than the seed std in each case. Table 4 alternatives with three sampling seeds: Random Perturbation 54.90 ± 0.54, Linear Position 55.11 ± 1.11, Frequency 56.46 ± 0.60, all above Dependency-Aware (53.57 ± 0.58).
W2 and Q3: Parser sensitivity
Thank you for the question. We see two likely reasons for the smaller gain with Stanza. First, the sampler hyperparameters were tuned on the spaCy checkpoint and reused for Stanza without re-tuning. Second, the two parsers use different annotation schemes (spaCy's own scheme vs. Universal Dependencies for Stanza), which assign different depths to some words and therefore teach the model somewhat different generation orders. With three sampling seeds (MDLM, LM1B, our sampler, mean ± std):
| Training-time masking | Gen. PPL |
|---|---|
| Uniform | 56.73 ± 0.28 |
| Dependency-Aware, spaCy | 53.57 ± 0.58 |
| Dependency-Aware, Stanza | 55.26 ± 0.74 |
Both parsers lower Gen. PPL relative to Uniform, with a smaller reduction for Stanza. We have revised Section 5.4 to report this.
Thank you for the additional experiments. The Root-Only and Shuffled-Depth controls address an important part of my concerns and strengthen the evidence for using syntactic depth. However, variability across independent training runs remains untested, and the connection to reduced factorization error is still indirect. On balance, the stronger empirical evidence leads me to update my rating to weak acceptance.
Thank You for the Follow-up
Thank you very much for your follow-up and for raising your score. We truly appreciate your constructive feedback.
- Factorization error. We agree that the link is indirect. One supporting observation is in Appendix F.1: with top- samplers, which do not avoid co-decoding dependent tokens, halving the step budget raises Gen. PPL by 93–113 for Uniform but only by 26–47 for Dependency-Aware Masking. This is consistent with reduced factorization error, and we will discuss it in the revised paper.
- Independent training runs. With limited compute, we prioritized the new Root-Only and Shuffled-Depth controls during the rebuttal, and we plan to add independent runs in the final version.
Thank you again for your time and valuable feedback.
Response to W1 and W2
We thank the reviewer for the clear summary and the pointed questions.
W1: Novelty, and the link between masking and conditional independence
Thank you for these comments.
Non-uniform masking. We agree. Our contribution is the depth signal itself. We re-ran Table 4 with three seeds (the paper reports one) and added two controls: Shuffled-Depth (depths randomly reassigned within each sentence) and the Root-Only baseline you suggested in Q1 (only the root up-weighted). MDLM/LM1B, mean ± std:
Dependency depth gives the lowest Gen. PPL and beats both controls in every seed.
DAPD. Our sampler builds on the core idea of DAPD (Kim et al., 2026a): an attention-derived interaction graph from which an independent set of masked positions is unmasked in parallel. Our two main changes align the sampler with the order learned by Dependency-Aware Masking. First, candidates are ranked by confidence alone, because DAPD's degree-weighted priority favors the highly connected core tokens that our masking teaches the model to resolve last. Second, a per-step cap K prevents committing a very large independent set in one step. There are also smaller implementation differences (Appendix A.2). Appendix F.2 compares our sampler with the released DAPD implementation (MDLM, LM1B, NFE ≈ 80, Gen. PPL, single seed):
Our sampler is better under both masking schemes, and the gain from Dependency-Aware Masking grows from 2.06 under DAPD to 3.30 under ours, consistent with the sampler being designed around the masking prior.
Masking and conditional independence. We agree that masking changes per-token probabilities and does not by itself decide which tokens are decoded together. That is decided by the inference-time independent-set rule. What masking does change is the generation order: under the same sampler, the generation-order score Aw rises from 0.531 (Uniform) to 0.710 (Dependency-Aware) in Table 3, so the tokens available for parallel decoding at each step follow the dependency tree more closely.
W2: Experimental results
Thank you for these concerns. We answer the variance, parser and Root-Only points under Q1–Q3 below. Training-side results are limited to 110M models, since billion-scale pretraining is beyond our budget. The sampler needs no retraining, so we evaluated it on the off-the-shelf LLaDA-8B-Instruct with the official setup:
Our sampler uses 39–78% fewer NFE, with equal or higher accuracy on all tasks except GSM8K. We will add this evaluation to the appendix.
We also evaluate prefix-conditioned generation on LM1B: given the first 32 of 128 tokens, each model generates the remaining 96 with our sampler:
Dependency-Aware Masking lowers Gen. PPL for both backbones while slightly increasing entropy, so the improvement extends to conditional generation.