- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Hierarchical Masked Diffusion Language Models (HMDLM), which incorporate position-level hierarchy into masked diffusion through level-dependent noise schedules and a two-stage architecture that first predicts hierarchy levels and then generates tokens conditioned on them. The authors derive a corresponding training objective and demonstrate improvements in perplexity and generation quality on LM1B and WikiHow, as well as solution accuracy on Sudoku.
Critical strengths
S1. Intuitive and well-motivated approach. Incorporating position-level priorities into masked diffusion is a reasonable approach to improving generation order, particularly for structured tasks such as Sudoku.
S2. Lightweight architectural modifications. The method introduces relatively few additional parameters and can be integrated into existing masked diffusion backbones without substantial architectural changes.
Critical weaknesses
W1. Insufficient component-wise ablations. HMDLM simultaneously introduces hierarchy supervision, embeddings, level-dependent schedules, two-pass computation, and additional losses. Without isolating these components, it remains unclear whether the gains stem from hierarchy-aware modeling or increased supervision and computation.
W2. Limited comparison with closely related work. GenMD4 already formulates state-dependent masking schedules and their ELBO, while OeMDM/LoMDM model and optimize generation orders. The paper should clarify its theoretical novelty and compare against these baselines.
Questions for authors
Q1. Hierarchy-aware training vs. decoding strategy. Could the authors compare HMDLM with MDLM using confidence, margin, entropybased, or blockwise decoding on Gen. PPL and MAUVE? This would clarify whether the gains arise from hierarchy-aware training or improved inference-time ordering.
Q2. Validity of the perplexity estimator. Appendix E replaces gold hierarchy-dependent weights with expectations under predicted hierarchies, which does not necessarily preserve the ELBO. Lower predicted reveal rates might reduce the weighted loss without improving token predictions. Is the estimator still a valid likelihood bound? A comparison with the full joint NELBO would help.
Q3. Performance under matched inference compute. HMDLM requires two backbone passes per denoising step despite its small parameter overhead. Could the authors report FLOPs, forward passes, and latency, and compare HMDLM with T steps against MDLM with approximately 2T steps under matched compute?
Response to Reviewer: 4o1f
We thank the reviewer for raising this. We clarify below how HMDLM differs from these methods, and we add a direct comparison on Sudoku.
Theoretical novelty. GenMD4 and LoMDM both use state-dependent masking schedules, so different tokens can be unmasked at different rates.
HMDLM differs in that the token predictions themselves use the schedule. We introduce a latent hierarchy state with a one-to-one mapping between a position’s hierarchy level and its masking schedule. The model runs in two passes. The first pass predicts hierarchy probabilities and samples a level for each position. The second pass predicts each token conditioned on the hierarchy levels of the entire sequence. Each level corresponds to exactly one schedule, so every token prediction uses the schedule information of the whole context, both at the position being predicted and at the positions around it. Neither prior method does this.
Sudoku-hard
HMDLM v1 is the model from the submitted paper. HMDLM v2 has the same architecture and objective. The only change is in the training and inference algorithm: the second pass is conditioned on hierarchy levels sampled from the first pass (for all the masked tokens), as described in the common response. Even v1 beats the baselines on cell accuracy at 96.24% against 95.23% for GenMD4, and its solve rate is close to the baselines. With the revised training, v2 outperforms all baselines on both metrics, reaching a 90.0% solve rate against 85.96% for the strongest baseline, LoMDM.