- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Ordered-DLM, a masked diffusion language model whose token reveal order is governed by a learned syntactic hierarchy. Continuous per-token heights are predicted by a lightweight network, converted into differentiable interval-survival scores via a soft max-Cartesian-tree construction, and fed into the OeMDM scheduling parameterization for both forward masking and reverse denoising. A blockwise variant keeps memory linear in sequence length. Empirically, the method improves likelihood bounds on LM1B and OpenWebText, transfers to seven zero-shot domains, and lowers generative perplexity across sampling budgets.
Critical strengths
- The paper offers a concrete, end-to-end differentiable mechanism for injecting structural bias into the generation order of masked diffusion models, without relying on an external parser or treebank supervision. Compared with LoMDM's generic learned ordering, the Cartesian-tree interval aggregation supplies an explicit inductive bias about how token positions relate to one another hierarchically, which is a meaningful design contribution in its own right.
2.The empirical picture is broad and consistent. On OWT the perplexity bound drops from 20.4 (LoMDM) to 19.28; on unpacked LM1B it falls from 25.4 to 22.65 (Table 1). The zero-shot results in Table 2 show gains on all seven held-out datasets, and Table 3 reports improvements at every low-precision sampling budget.
Critical weaknesses
-
The hierarchical transformation itself is never isolated. The entire method sits on top of the OeMDM learned-order framework, yet no ablation controls for the specific contribution of the interval-aggregation step. A natural baseline—matching predictor capacity, training objective, and budget while replacing the Cartesian-tree hierarchy with directly predicted positive per-token schedule scores—is absent. The sign-reversal experiment in Appendix A shows that the model can flip its height scale to compensate, but this speaks to the parameterization's symmetry, not to whether hierarchical interval scoring adds value over a flat learned score. As it stands, the reported gains are consistent with the hypothesis that any well-parameterized learned scheduler would perform similarly, and the paper does not rule this out. A learned Cartesian hierarchy, moreover, is a structural analogy; without targeted linguistic evaluation it should not be conflated with evidence of syntactic parsing.
-
The contextual-value probe supports a narrower claim than stated. Equations 16–18 reveal ground-truth tokens and measure entropy reduction over the remaining masked positions. Two issues weaken the interpretation. First, lower predictive entropy does not necessarily translate into better prediction of the true continuation; the probe measures distributional sharpness, not accuracy. Second, oracle reveals are qualitatively different from the model's own generated tokens during actual denoising, so the probe does not directly validate the generation-time behavior. Finally, it is unclear whether all compared orderings (Ordered-DLM, LoMDM, MDLM confidence) share the same evaluation denoiser and masking distribution at each timestep. These gaps do not invalidate the language-modeling results, but they limit how strongly the probe can support the "contextual planning" narrative.
Questions for authors
-
Focused validation: Under matched predictor capacity and training budget, how does hierarchical interval aggregation compare with direct positive per-token schedule scores? This comparison would materially affect my assessment of the contribution beyond existing learned-order methods.
-
Clarification and validation: Which denoiser and masking distribution are used for the contextual-value probe? Under a common evaluator, does the ordering also improve remaining-token log loss, with uncertainty over the evaluated contexts?
-
Appendix A examines the sign convention for height comparisons but does not isolate the interval-based hierarchy construction. With predictor capacity, training objective, and training budget matched, how does the proposed construction compare with directly predicting positive token-wise scheduling scores? This would clarify whether the hierarchy itself explains the gains over generic learned ordering.
2 comments
Dear authors,
Thank you for the additional experiments and clarifications. The common-evaluator NLL probes and generated-token rollouts address several of my concerns about the contextual-value analysis.
My remaining concern is the lack of an end-to-end comparison with a flat scheduler under matched capacity and training budget. The fixed-checkpoint interventions provide useful evidence that interval aggregation benefits the trained model, but its heights and backbone were jointly optimized with that aggregation. They therefore do not fully establish the advantage over an independently trained flat scheduler. The reported sigmoid-training collapse does not fully resolve this gap.
I will keep my current score.
Response to Reviewer xZGW
We thank Reviewer xZGW for the insightful comments. In the revised manuscript (highlighted in blue), we added fixed-checkpoint readout interventions isolating the causal contribution of Cartesian interval aggregation across single-step and multi-round rollouts (W1/Q1/Q3), packed LM1B validation PPLs for the sign-reversal ablation alongside UD English-EWT syntactic dependency probing (W1/Q3), and common-evaluator contextual probes scoring both predictive entropy and ground-truth token NLL under shared frozen denoisers (W2/Q2).
W1 / Q1 / Q3 — What does interval aggregation contribute? (Sec. 4.3, App. A & B). In the same learned-order diffusion framework, LoMDM serves as our end-to-end baseline whose per-token scheduler omits Cartesian interval aggregation; in Table 1, Ordered-DLM improves the OWT perplexity bound from 20.4 (LoMDM) to 19.28. Predicting positive token-wise schedule scores directly with a sigmoid collapsed during training (App. A.2). While a separately trained flat scheduler with matched capacity and budget remains open, comparing two separately trained checkpoints conflates differences in learned backbone representations with the effect of the interval readout itself.
To isolate the causal contribution of Cartesian interval aggregation at inference without representation confounding, we held the trained Ordered-DLM checkpoint and its predicted per-token heights fixed across 64 OWT validation contexts and compared the hierarchical interval readout directly against a flat ranking of the exact same heights in two settings (Sec. 4.3, App. A.2):
Regarding Appendix A.1, comparing the forward vs. reversed pairwise height differences tests sign equivariance: reversing the comparison orientation is compensated by learning negatively oriented heights (Spearman ρ=−0.9848, Table 7) while retaining interval aggregation in both variants; their archived validation PPLs on packed LM1B are nearly identical (24.565 vs. 24.571; Table 7). To test syntactic alignment, we added visualizations of learned heights, discrete trees, soft scores, and priorities (App. B.1, Fig. 5), and evaluated our LM1B checkpoint on all 2,077 UD English-EWT test sentences (App. B.2, Table 12), achieving 35.15% UUAS vs. 30.37% (shuffled), 30.39% (random init), and 30.36% (random scores).
W2 / Q2 — Contextual value under a common evaluator (Sec. 4.3, App. A.2 & C). All ordering policies start from the same 64 OWT validation texts and identical initial masks at each masking fraction (25%, 50%, 75%, 90%). In the single-reveal probe (App. C), we run the comparison using a single frozen MDLM denoiser for all orderings (and repeat with frozen Ordered-DLM), excluding the revealed position before and after and separately reporting predictive entropy and ground-truth token NLL under (a) oracle reveals (reference token) and (b) model-generated reveals (sampled from the shared denoiser). At 90% masking, oracle reveals selected by Ordered-DLM reduce remaining-token NLL more than lowest-entropy ranking by +0.00239 nats/token under MDLM (+0.00268 under Ordered-DLM; Fig. 8, Tables 14–15), while the 8-round rollout above (Exp. 2, Fig. 4, Tables 10–11) evaluates multi-round generation on fixed held-out targets.