acceptodds
Under review as a conference paper at ICLR 2027

Ordered-DLM: Ordering Token Generation through Syntactic Hierarchy

Abstract

Masked diffusion language models generate text by iteratively revealing masked tokens, enabling parallel generation and bidirectional conditioning. This flexibility raises a fundamental scheduling question: when multiple positions are masked, which token should be revealed first? Each reveal not only resolves one position but also changes the context available for predicting what remains. Motivated by information-gain principles in LLM reasoning, we prioritize revealing tokens that reduce uncertainty in predicting the remaining tokens, an objective we call *contextual planning*. To realize this objective, we use syntactic hierarchy to identify tokens with broad structural roles in an unsupervised manner. We introduce Ordered-DLM, a masked diffusion framework that predicts continuous syntactic heights, uses their relative ordering to induce differentiable hierarchy scores, and applies these scores to both forward masking and reverse denoising. The hierarchy is learned jointly with the diffusion model , requiring neither an external parser nor syntactic annotations. Experiments on LM1B and OpenWebText demonstrate lower perplexity than strong diffusion baselines on in-domain and zero-shot benchmarks, as well as lower generative perplexity across sampling budgets, while preserving parallel decoding. These results show that syntactic hierarchy-induced scheduling improves both contextual planning and language-modeling quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xZGWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Ordered-DLM, a masked diffusion language model whose token reveal order is governed by a learned syntactic hierarchy. Continuous per-token heights are predicted by a lightweight network, converted into differentiable interval-survival scores via a soft max-Cartesian-tree construction, and fed into the OeMDM scheduling parameterization for both forward masking and reverse denoising. A blockwise variant keeps memory linear in sequence length. Empirically, the method improves likelihood bounds on LM1B and OpenWebText, transfers to seven zero-shot domains, and lowers generative perplexity across sampling budgets.

Critical strengths
  1. The paper offers a concrete, end-to-end differentiable mechanism for injecting structural bias into the generation order of masked diffusion models, without relying on an external parser or treebank supervision. Compared with LoMDM's generic learned ordering, the Cartesian-tree interval aggregation supplies an explicit inductive bias about how token positions relate to one another hierarchically, which is a meaningful design contribution in its own right.

2.The empirical picture is broad and consistent. On OWT the perplexity bound drops from 20.4 (LoMDM) to 19.28; on unpacked LM1B it falls from 25.4 to 22.65 (Table 1). The zero-shot results in Table 2 show gains on all seven held-out datasets, and Table 3 reports improvements at every low-precision sampling budget.

Critical weaknesses
  1. The hierarchical transformation itself is never isolated. The entire method sits on top of the OeMDM learned-order framework, yet no ablation controls for the specific contribution of the interval-aggregation step. A natural baseline—matching predictor capacity, training objective, and budget while replacing the Cartesian-tree hierarchy with directly predicted positive per-token schedule scores—is absent. The sign-reversal experiment in Appendix A shows that the model can flip its height scale to compensate, but this speaks to the parameterization's symmetry, not to whether hierarchical interval scoring adds value over a flat learned score. As it stands, the reported gains are consistent with the hypothesis that any well-parameterized learned scheduler would perform similarly, and the paper does not rule this out. A learned Cartesian hierarchy, moreover, is a structural analogy; without targeted linguistic evaluation it should not be conflated with evidence of syntactic parsing.

  2. The contextual-value probe supports a narrower claim than stated. Equations 16–18 reveal ground-truth tokens and measure entropy reduction over the remaining masked positions. Two issues weaken the interpretation. First, lower predictive entropy does not necessarily translate into better prediction of the true continuation; the probe measures distributional sharpness, not accuracy. Second, oracle reveals are qualitatively different from the model's own generated tokens during actual denoising, so the probe does not directly validate the generation-time behavior. Finally, it is unclear whether all compared orderings (Ordered-DLM, LoMDM, MDLM confidence) share the same evaluation denoiser and masking distribution at each timestep. These gaps do not invalidate the language-modeling results, but they limit how strongly the probe can support the "contextual planning" narrative.

Questions for authors
  1. Focused validation: Under matched predictor capacity and training budget, how does hierarchical interval aggregation compare with direct positive per-token schedule scores? This comparison would materially affect my assessment of the contribution beyond existing learned-order methods.

  2. Clarification and validation: Which denoiser and masking distribution are used for the contextual-value probe? Under a common evaluator, does the ordering also improve remaining-token log loss, with uncertainty over the evaluated contexts?

  3. Appendix A examines the sign convention for height comparisons but does not isolate the interval-based hierarchy construction. With predictor capacity, training objective, and training budget matched, how does the proposed construction compare with directly predicting positive token-wise scheduling scores? This would clarify whether the hierarchy itself explains the gains over generic learned ordering.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xZGW

We thank Reviewer xZGW for the insightful comments. In the revised manuscript (highlighted in blue), we added fixed-checkpoint readout interventions isolating the causal contribution of Cartesian interval aggregation across single-step and multi-round rollouts (W1/Q1/Q3), packed LM1B validation PPLs for the sign-reversal ablation alongside UD English-EWT syntactic dependency probing (W1/Q3), and common-evaluator contextual probes scoring both predictive entropy and ground-truth token NLL under shared frozen denoisers (W2/Q2).


W1 / Q1 / Q3 — What does interval aggregation contribute? (Sec. 4.3, App. A & B). In the same learned-order diffusion framework, LoMDM serves as our end-to-end baseline whose per-token scheduler omits Cartesian interval aggregation; in Table 1, Ordered-DLM improves the OWT perplexity bound from 20.4 (LoMDM) to 19.28. Predicting positive token-wise schedule scores directly with a sigmoid collapsed during training (App. A.2). While a separately trained flat scheduler with matched capacity and budget remains open, comparing two separately trained checkpoints conflates differences in learned backbone representations with the effect of the interval readout itself.

To isolate the causal contribution of Cartesian interval aggregation at inference without representation confounding, we held the trained Ordered-DLM checkpoint and its predicted per-token heights fixed across 64 OWT validation contexts and compared the hierarchical interval readout directly against a flat ranking of the exact same heights in two settings (Sec. 4.3, App. A.2):

  1. Single-token reveal intervention (Exp. 1; Fig. 3, Tables 8–9): When ranking the raw heights flatly without interval aggregation, the policy frequently selects isolated local height spikes that the denoiser cannot reliably predict from partial context (38.28% reveal accuracy, token NLL 3.550 under the shared MDLM denoiser; Table 9), degrading predictions on the remaining masked tokens (ΔNLL=−0.00896\Delta\text{NLL} = -0.00896). By contrast, Cartesian interval aggregation requires a token to dominate a contiguous neighborhood of spans before receiving high priority, more than doubling reveal accuracy (79.98%, token NLL 0.705) and reducing remaining-token NLL deterioration by +0.00744 nats/token relative to flat ranking under MDLM (+0.00655 under Ordered-DLM; Table 8).
  2. Autonomous eight-round rollout (Exp. 2; Fig. 4, Tables 10–11): Using the same frozen Ordered-DLM generator, each policy samples 8 tokens/round for 8 rounds (64 generated tokens) and recomputes priorities at each round, while a seeded 25% subset of initially masked positions is held masked throughout as fixed evaluation targets. As shown in Fig. 4 (left), the gap between hierarchical interval aggregation and flat ranking of the same heights grows monotonically at every round: scored by the common frozen MDLM evaluator, hierarchy achieves a final target NLL of 3.9993 vs. 4.1836 for flat ranking—a paired reduction of 0.1843 nats/token (0.1670 under Ordered-DLM; Table 10), alongside higher cumulative reveal accuracy (54.49% vs. 37.98%, Table 11).

Regarding Appendix A.1, comparing the forward vs. reversed pairwise height differences tests sign equivariance: reversing the comparison orientation is compensated by learning negatively oriented heights (Spearman ρ=−0.9848\rho = -0.9848, Table 7) while retaining interval aggregation in both variants; their archived validation PPLs on packed LM1B are nearly identical (24.565 vs. 24.571; Table 7). To test syntactic alignment, we added visualizations of learned heights, discrete trees, soft scores, and priorities (App. B.1, Fig. 5), and evaluated our LM1B checkpoint on all 2,077 UD English-EWT test sentences (App. B.2, Table 12), achieving 35.15% UUAS vs. 30.37% (shuffled), 30.39% (random init), and 30.36% (random scores).

W2 / Q2 — Contextual value under a common evaluator (Sec. 4.3, App. A.2 & C). All ordering policies start from the same 64 OWT validation texts and identical initial masks at each masking fraction (25%, 50%, 75%, 90%). In the single-reveal probe (App. C), we run the comparison using a single frozen MDLM denoiser for all orderings (and repeat with frozen Ordered-DLM), excluding the revealed position before and after and separately reporting predictive entropy and ground-truth token NLL under (a) oracle reveals (reference token) and (b) model-generated reveals (sampled from the shared denoiser). At 90% masking, oracle reveals selected by Ordered-DLM reduce remaining-token NLL more than lowest-entropy ranking by +0.00239 nats/token under MDLM (+0.00268 under Ordered-DLM; Fig. 8, Tables 14–15), while the 8-round rollout above (Exp. 2, Fig. 4, Tables 10–11) evaluates multi-round generation on fixed held-out targets.

Dear authors,

Thank you for the additional experiments and clarifications. The common-evaluator NLL probes and generated-token rollouts address several of my concerns about the contextual-value analysis.

My remaining concern is the lack of an end-to-end comparison with a flat scheduler under matched capacity and training budget. The fixed-checkpoint interventions provide useful evidence that interval aggregation benefits the trained model, but its heights and backbone were jointly optimized with that aggregation. They therefore do not fully establish the advantage over an independently trained flat scheduler. The reported sigmoid-training collapse does not fully resolve this gap.

I will keep my current score.

Reviewer_bDsuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Ordered-DLM, a masked diffusion language model that learns token generation order through a hierarchical structure. Building on prior work on learned-order diffusion, the method assigns a learned height to each token and uses a differentiable approximation of a Cartesian tree to determine its generation priority. The underlying intuition is that structurally important tokens should be generated earlier to provide useful context for subsequent predictions. Experiments on language modeling benchmarks show consistent improvements over existing diffusion-based baselines, including in-domain and zero-shot perplexity. The paper also presents analyses of contextual entropy and generation quality to support the benefits of the proposed ordering strategy.

Critical strengths

The Cartesian-tree formulation provides a principled way to introduce structural dependencies without requiring an external syntactic parser. I find this direction interesting. Also, the proposed method achieves consistent improvements over existing learned-order diffusion baselines on both in-domain and zero-shot language modeling benchmarks. The improvements across different evaluation settings suggest that the approach has practical potential.

Critical weaknesses

My main concern is that the paper does not sufficiently isolate the contribution of the proposed Cartesian-tree formulation from the broader benefits of learning token generation order. Since Ordered-DLM builds on an existing learned-order diffusion framework, it is important to demonstrate what the hierarchical formulation contributes beyond a direct learned per-token priority or exponent. A controlled comparison with a similarly parameterized, non-hierarchical scheduler would significantly strengthen the paper’s central claim. Relatedly, the connection between the proposed formulation and syntactic hierarchy remains largely conceptual. Although the hard Cartesian-tree construction is mathematically intuitive, the soft relaxation does not necessarily preserve the same ordering or tree structure. The paper would benefit from a clearer explanation of what structural properties the relaxation actually preserves. Furthermore, analyses of the learned hierarchy, such as visualizations or comparisons with linguistic structures, would help substantiate the syntactic interpretation emphasized in the title and motivation.


While the reported improvements are encouraging, several aspects of the evaluation are insufficiently specified, making it difficult to assess the reliability and significance of the comparisons. In particular, it is unclear whether all baseline results were obtained using consistent evaluation pipelines and preprocessing settings. This is especially important for zero-shot perplexity, where tokenization, sequence length, and evaluation protocols can substantially affect the results. The generative perplexity evaluation also raises concerns about numerical precision and comparability. Prior work has shown that sampling precision can substantially influence generative perplexity, and the paper itself reports a considerable difference between sampling precisions. A consistent evaluation across methods, ideally accompanied by entropy measurements, would make the generation-quality comparisons more convincing.


Finally, several implementation details and technical terms are introduced without sufficient explanation, including the hierarchical scheduling configuration and gradient propagation through the masking process. Clarifying these points would improve both reproducibility and accessibility.

Questions for authors
  1. How does Ordered-DLM compare with a direct learned per-token priority or exponent, using comparable model capacity and settings?

  2. What evidence supports the claim that the learned ordering captures syntactic hierarchy? Could the authors provide visualizations or quantitative analyses of the learned structures and clarify how well the soft relaxation preserves the intended hierarchical properties?

  3. Were the baselines evaluated under consistent settings and preprocessing pipelines, particularly for zero-shot and generative perplexity? Could the authors clarify the evaluation protocols and address the potential effects of sampling precision on the reported comparisons?

  4. Could the authors provide further details on the contextual-entropy analysis and hierarchical scheduling implementation, particularly how ordering is determined, which denoiser is used, and how gradients propagate through the masking process?

2 comments
AuthorsOpenReview ↗

Response to Reviewer bDsu

We thank Reviewer bDsu for the constructive feedback. In the revised manuscript (highlighted in blue), we added fixed-checkpoint readout interventions isolating the role of the hierarchy at inference (Q1), syntactic dependency probing and analyses of the soft relaxation (Q2), zero-shot and GenPPL protocols (W2/Q3), and common-evaluator contextual probes alongside a gradient audit (W3/Q4).


Q1 — Direct per-token scores and the role of the hierarchy (Sec. 4.3, App. A.2). Predicting positive token-wise scores directly with a sigmoid collapsed during training (App. A.2), and comparing two separately trained checkpoints cannot separate representation differences from the readout rule itself. To isolate the causal contribution of Cartesian interval aggregation at inference without representation confounding, we held the trained Ordered-DLM checkpoint and its predicted per-token heights fixed across 64 OWT contexts and compared the hierarchical interval readout directly against a flat ranking of the exact same heights:

  1. Single-token reveal intervention (Exp. 1; Fig. 3, Tables 8–9): Flat ranking frequently selects isolated local height spikes that the denoiser cannot reliably predict from partial context (38.28% reveal accuracy, token NLL 3.550 under MDLM; Table 9), degrading predictions on the remaining masked positions (ΔNLL=−0.00896\Delta\text{NLL} = -0.00896). By contrast, Cartesian interval aggregation requires a token to dominate a contiguous neighborhood of spans before receiving high priority, doubling reveal accuracy (79.98%, token NLL 0.705) and reducing remaining-token NLL deterioration by +0.00744 nats/token over flat ranking under MDLM (+0.00655 under Ordered-DLM; Table 8).
  2. Autonomous eight-round rollout (Exp. 2; Fig. 4, Tables 10–11): Across 8 rounds (64 generated tokens) with the shared Ordered-DLM generator, the gain over flat ranking compounds at every round (Fig. 4, left): final NLL on fixed held-out targets is 3.9993 vs. 4.1836 for hierarchy vs. flat—a paired reduction of 0.1843 nats/token under MDLM (0.1670 under Ordered-DLM; Table 10), with higher cumulative reveal accuracy (54.49% vs. 37.98%, Table 11). Training a capacity-matched flat scheduler remains open.

Q2 — Syntactic motivation and properties of the soft relaxation (Sec. 3.3, 4.3, App. B). We added visualizations showing the learned heights, discrete Cartesian tree, soft hierarchy scores, and final reveal priorities on OWT text (App. B.1, Fig. 5). On all 2,077 UD English-EWT test sentences (App. B.2, Table 12), word-pooled trees from our LM1B checkpoint achieved 35.15% UUAS, outperforming shuffled (30.37%), untrained (30.39%), and random (30.36%) controls. Algebraically, for a fixed anchor token, each span weight multiplies pairwise height-comparison factors in [0,1][0,1] (Sec. 3.3, App. B.3): extending a span within a block multiplies an additional factor in [0,1][0,1] (non-increasing in length), and a zero factor blocks all spans crossing that position, so raising an intervening token's height lowers the anchor's hierarchy score even with the anchor's height held fixed (Fig. 6). Across 64 OWT contexts, the soft-vs.-hard Spearman correlation is 0.881 at 25% masking (0.862 at 50%, 0.751 at 75%) and 0.458 at 90% masking (Fig. 7, Table 13).

W2 / Q3 — Zero-shot and generative perplexity protocols (Sec. 4.1–4.2, App. E & G). For zero-shot PPL (Table 2), our runs use the GPT-2 tokenizer, dataset detokenization where applicable, and 1,024-token packing with BOS/EOS at boundaries and no extra EOS between examples (Sec. 4.1, App. G). For GenPPL on OWT (Sec. 4.2, App. E), we swept temperatures at 33 and 129 NFE (64 sequences/temperature, GPT-2 Large evaluator) under matched precision (float32 and float64). Ordered-DLM achieves lower mean GenPPL at matched entropy than MDLM across all four settings (217.27 vs. 232.31 and 121.56 vs. 129.02 in FP32, −6.5% and −5.8%; 101.48 vs. 108.93 and 76.99 vs. 81.83 in FP64, −6.8% and −5.9%; Figs. 10–11, Tables 17–19).

W3 / Q4 — Hierarchical scheduling, contextual probe, and gradients (Sec. 3.4, 4.3, App. C & F). (1) Ordering: At each reverse step, the scheduler predicts one height per position from denoiser hidden states, aggregates within-block span weights into a blockwise hierarchy score (schedule exponent), and ranks masked positions by this exponent. (2) Denoiser & metrics: In App. C, we evaluate all orderings on the same 64 OWT contexts and masks under a shared frozen MDLM denoiser (and frozen Ordered-DLM), excluding the revealed position before/after and reporting entropy and token NLL for oracle and generated reveals (Figs. 3, 8, Tables 8, 14–15). (3) Gradients: Gradients flow through the explicit forward and reverse schedule-velocity terms in Eq. 13; discrete mask draws have no pathwise gradient (Fig. 13).

I appreciate the detailed response and the additional analyses in the revision. The matched-entropy GenPPL comparison with MDLM, along with the protocol and gradient details, addresses my evaluation questions. However, I maintain my rating of 2, as the main concern I share with Reviewer xZGW remains unresolved.

The comparison with a matched-capacity flat scheduler is deferred, and the readout intervention does not replace it, since the heights being ranked were trained specifically for the interval readout. Given that the rest of the method largely follows OeMDM, as Reviewer kYvq also points out, this comparison is necessary to establish whether the gains actually come from the hierarchy.

The syntactic motivation also remains weakly supported. The learned trees achieve 35% UUAS, only about 5 points above random trees. By my count, a simple chain connecting adjacent words performs better on the same test set. Moreover, the rollouts show that revealing the lowest-entropy tokens first using the same checkpoint outperforms the hierarchy, which is consistent with Reviewer kYvq’s concern about prioritizing structural tokens.

Reviewer_kYvqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work (Ordered-DLM) builds upon recent works on Masked Diffusion Ordering Generation. The noise schedule (αt)0≤t≤1(\alpha_t)_{0 \leq t \leq 1} is learned by a neural network αϕ(x,t)\alpha_\phi(x, t), such that contextually important tokens, in a specific syntactic sense, are masked last. To target the corresponding reverse process, the framework parameterizes a generative process through a reverse schedule $ \alpha_\psi(z_t, t) $, taking as input the noisy sample ztz_t, and a standard masked diffusion denoiser pθp_\theta, and learns these quantities through a negative ELBO loss. Because of the quadratic cost in sequence length, a blockwise scoring rule is derived to reduce the computational burden at training and sampling time. Finally, language modeling experiments are run to assess the performance of the framework, together with a priority context value evaluation aimed at validating the effect of the proposed mechanism.

Critical strengths
  • The considered experimental bench is sound: unconditional/conditional generation, and a context value metric quantifying the information revealed of by a selected token.
Critical weaknesses

I have serious doubts about the paper's takeaways.

  • The syntactic interpretation of the proposed construction is actually not established properly. The specific empirical contribution of the cartesian tree is unclear. There are no syntactic analysis of the learned structure.
  • Beyond this mismatch, it is not clear why targeting the most syntactically important token, in the authors sense, would actually help generation. These tokens might also be the most difficult to predict, and thus might not be the best tokens to reveal at time tt. Other baselines, e.g., targeting lowest entropy tokens, are missing.
  • The unconditional generation results do not demonstrate generative performance improvements on OWT. For the GenPPL/entropy results, Ordered-DLM does not demonstrate better generation quality than MDLM: although Ordered-DLM has lower GenPPL, it also has lower entropy. As discussed by Pynadath, Shi, and Zhang (2026) in Generative Frontiers: Why Evaluation Matters for Diffusion Language Models, generative perplexity alone is misleading because of its sensitivity to entropy; they propose generative frontiers for more reliable comparisons. I therefore recommend sweeping the sampling temperature and plotting GenPPL against entropy to compare the models' generative frontiers. In my own experiments, MDLM achieves very similar GenPPL/entropy values at a lower sampling temperature. I therefore have serious doubts about the method's performance. My interpretation is that Ordered-DLM performs very similarly to MDLM, suggesting that token ordering does not play a crucial role in this setting.
  • The reported test PPLs are misleading because they also reflect differences in the ELBO bound/log-likelihood gap: KL[qϕ(z0:1∣x) ∥ pθ,ψ(z0:1∣x)]\text{KL}[q_{\phi}(z_{0:1} | x) \ \parallel \ p_{\theta, \psi}(z_{0:1} | x)] which is optimized by the forward/backward coupled optimization procedure. Furthermore, the optimization through the sampling operation in the ELBO is insufficiently discussed.
  • The effect of the block length is not studied / ablated.
  • The authors largely build on the framework of Hong, Lee, and Ye (2026), Unifying Masked Diffusion Models with Various Generation Orders and Beyond. The only novelty seems to be a different construction of the noise schedule and a blockwise computation that reduces memory requirements. These contributions are interesting, but their novelty is insufficient to compensate for the weaknesses in the experimental evaluation.
  • The effect of numerical precision on generative performance at sampling time has already been studied by Zheng at al. (2024) Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling and is explained by entropic collapse.

As a result I recommend rejection of the paper.

Questions for authors
  • Your likelihood can be decomposed as a weighted reconstruction loss (dependent on theta) + a rate matching loss. Would it be possible to identify the PPL improvements more precisely from this decomposition ? Are there insights to be drawn from the variational diffusion models perspective for the reconstruction part (Kingma et al. (2021), Variational Diffusion Models)?

  • What are the effects of your Blockwise hierarchy scoring ? To what extent does it lower the performance compared to a full hierarchy scoring ? Have you performed ablations on the block length L ?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kYvq

We thank Reviewer kYvq for the constructive feedback. In the revised manuscript (highlighted in blue), we added UD English-EWT dependency probing and tree visualizations (W1), common-evaluator comparisons with lowest-entropy ordering (W2), conditional MAUVE and entropy-matched unconditional frontiers (W3), a variational objective decomposition and gradient audit (W4/Q1), and a block-length LL ablation (W5/Q2), and clarified our sampler-precision control (W7).


W1 — Syntactic motivation and the Cartesian tree (Sec. 1, 3.3, 4.3, App. B). The max-Cartesian tree serves as a syntax-inspired inductive bias: its nested contiguous spans make a token's priority depend on how its height compares with intervening tokens in a span, rather than acting as an isolated per-token score. We clarified this motivation and added visualizations of learned heights, discrete trees, soft scores, and priorities on OWT text (App. B.1, Fig. 5). On all 2,077 UD English-EWT test sentences (App. B.2, Table 12), word-pooled trees from our LM1B checkpoint achieved 35.15% UUAS, outperforming shuffled (30.37%), untrained (30.39%), and random (30.36%) controls.

W2 — Why learn to generate informative tokens early? (Sec. 4.3, App. A.2 & C). Lowest-entropy ordering favors tokens the current denoiser is already confident about, whereas our goal is to train the denoiser to reconstruct informative anchors from sparse context so subsequent predictions become easier. Using a shared frozen denoiser (with full vocabulary logits for candidate positions), revealing ground-truth tokens selected by Ordered-DLM at 90% masking reduced remaining-token NLL more than lowest-entropy ranking by +0.00239 nats/token under MDLM (+0.00268 under Ordered-DLM; App. C, Fig. 8, Table 14). In 8-round generated rollouts, the hierarchy lowered final held-out NLL by 0.1843 nats/token over a flat readout of the same heights, while lowest entropy achieved the lowest NLL among fixed-checkpoint policies by selecting tokens the frozen denoiser already predicts with ~98% accuracy (App. A.2, Fig. 4, Tables 9–10), separating the value of a correct reveal from the difficulty of generating it without retraining.

W3 — Conditional generation (MAUVE) and additional unconditional generation results (Sec. 4.2, 4.4, Table 6, App. E).

  1. Conditional generation (Sec. 4.4, Table 6): Conditional continuation directly evaluates distributional quality against human continuations via MAUVE without GenPPL's temperature–entropy trade-off. We added MDLM to Table 6 under the identical protocol (1,000 OWT prefixes, five 50-token continuations per 50-token prefix, 1,000 steps in float64, GPT-2 Large featurization). Ordered-DLM achieves 0.9771 MAUVE, outperforming MDLM (0.9649, +0.0122) as well as LoMDM (0.9751) and SEDD (0.957), showing a clear generation improvement when conditioning on context.
  2. Additional unconditional GenPPL–entropy frontiers (Sec. 4.2, App. E, Table 17, Figs. 10–11): Following Pynadath et al. (2026), we also swept temperature on OWT at 33 and 129 NFE (64 sequences of length 1,024/temperature, GPT-2 Large evaluator) in both float32 (FP32) and float64 (FP64). Across both budgets and precisions, Ordered-DLM achieves lower mean GenPPL at matched entropy than MDLM (Table 17)—e.g., 217.27 vs. 232.31 (−6.5%) and 121.56 vs. 129.02 (−5.8%) in FP32, and 101.48 vs. 108.93 (−6.8%) and 76.99 vs. 81.83 (−5.9%) in FP64 at 33 and 129 NFE (Figs. 10–11).

W4 / Q1 — Objective decomposition and gradients (Sec. 3.4, App. F). On 32 OWT segments ×\times 32 shared draws, weighted reconstruction was 2.881 vs. 3.169 nats/token (−0.288), partially offset by +0.152 from rate matching (+0.120 on masked positions in Eq. 13), leaving a net reduction of −0.136 nats/token (−0.168 under Eq. 13; Fig. 12, Table 20). Gradients flow through the explicit velocity terms in Eq. 13; discrete mask draws have no pathwise gradient (Fig. 13).

W5 / Q2 & W6 — Block length LL and contribution (Sec. 1, 3.1, 3.5, App. D). Varying L∈{64,128,256,512,1024}L\in\{64,128,256,512,1024\} shows default L=128L=128 preserves the L=1024L=1024 ranking (ρ=0.9921\rho=0.9921), cuts GPU memory by 20× (0.881 vs. 17.754 GiB) and scorer latency by 36× (0.741 vs. 26.880 ms), and matches pilot GenPPL at 129 NFE (105.91 vs. 106.64; Fig. 9, Table 16). Building on Hong et al. (2026b), our span hierarchy and blockwise scoring make hierarchical ordering practical for long sequences (Figs. 3–4, 9).

W7 — Role of sampler precision (Sec. 4.1, App. E). We do not view this as a weakness or claim precision effects as a contribution: because float32 sampling can lower GenPPL via entropic collapse (Zheng et al., 2024), our submission already reported both float32 (Table 3) and float64 (Table 4) as an evaluation control, and we now cite Zheng et al. (2024) explicitly (Tables 18–19).

I thank the authors for their detailed response and substantial additional experiments. In particular, the entropy-matched generative frontiers address my concerns regarding GenPPL comparisons and provide evidence of moderate improvements.

However, my main reservations remain. The specific contribution of the hierarchical construction is still not convincingly isolated from that of a generic learned scheduler, as the fixed-checkpoint experiments cannot replace an end-to-end comparison. Furthermore, the syntactic interpretation remains weakly supported, and the generated-token experiments do not convincingly establish the benefits of prioritizing structurally important tokens over confidence-based ordering.

Nevertheless, I acknowledge that the additional experiments strengthen the evaluation, and I therefore raise my score from 1 to 2. I thank the authors again for their efforts and constructive engagement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.