acceptodds
Under review as a conference paper at ICLR 2027

Hierarchical Masked Diffusion Language Model

Abstract

Masked diffusion language models have emerged as an alternative to autoregressive models, enabling parallel token generation. However, standard masked diffusion language models use a shared noise schedule across positions without explicitly encoding which positions should be resolved first. Many tasks exhibit a natural priority among positions: document titles and section headings can guide the generation of body text, while Sudoku cells determined directly from the initial clues can help resolve more difficult cells. To incorporate this notion of priority (or hierarchy) among token positions, we introduce the Hierarchical Masked Diffusion Language Model (HMDLM), which augments each token position with a hierarchy level available as part of the training data, and requires zero or minimal additional annotations. It factors generation into two passes: the first predicts a hierarchy distribution over masked positions, the second predicts token distributions conditioned on it. Giving each hierarchy level its own noise schedule induces a soft, controllable preference for revealing higher-priority positions earlier while keeping decoding fully parallel. Our formulation is mathematically grounded, and we re-derive the forward and reverse processes and the NELBO for this factorization. Our evaluation shows that for the task of language generation on LM1B, HMDLM reduces perplexity by more than 1.5 points over Anchored Diffusion Language Model (ADLM), a recently proposed strong diffusion baseline; on WikiHow, we see a drop in perplexity of upto 7 pts on ADLM and upto 4.4 pts against MDLM (Masked Diffusion Language Model). A LLM judge prefers HMDLM generations more than as often as the baseline. On the task of solving Sudoku puzzles requiring multi-step reasoning, it improves solution accuracy by up to absolute points over existing diffusion baselines.

Primary area:
generative models
Keywords:
diffusion model, generative AI, Gen AI, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4o1fOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Hierarchical Masked Diffusion Language Models (HMDLM), which incorporate position-level hierarchy into masked diffusion through level-dependent noise schedules and a two-stage architecture that first predicts hierarchy levels and then generates tokens conditioned on them. The authors derive a corresponding training objective and demonstrate improvements in perplexity and generation quality on LM1B and WikiHow, as well as solution accuracy on Sudoku.

Critical strengths

S1. Intuitive and well-motivated approach. Incorporating position-level priorities into masked diffusion is a reasonable approach to improving generation order, particularly for structured tasks such as Sudoku.

S2. Lightweight architectural modifications. The method introduces relatively few additional parameters and can be integrated into existing masked diffusion backbones without substantial architectural changes.

Critical weaknesses

W1. Insufficient component-wise ablations. HMDLM simultaneously introduces hierarchy supervision, embeddings, level-dependent schedules, two-pass computation, and additional losses. Without isolating these components, it remains unclear whether the gains stem from hierarchy-aware modeling or increased supervision and computation.

W2. Limited comparison with closely related work. GenMD4 already formulates state-dependent masking schedules and their ELBO, while OeMDM/LoMDM model and optimize generation orders. The paper should clarify its theoretical novelty and compare against these baselines.

Questions for authors

Q1. Hierarchy-aware training vs. decoding strategy. Could the authors compare HMDLM with MDLM using confidence, margin, entropybased, or blockwise decoding on Gen. PPL and MAUVE? This would clarify whether the gains arise from hierarchy-aware training or improved inference-time ordering.

Q2. Validity of the perplexity estimator. Appendix E replaces gold hierarchy-dependent weights with expectations under predicted hierarchies, which does not necessarily preserve the ELBO. Lower predicted reveal rates might reduce the weighted loss without improving token predictions. Is the estimator still a valid likelihood bound? A comparison with the full joint NELBO would help.

Q3. Performance under matched inference compute. HMDLM requires two backbone passes per denoising step despite its small parameter overhead. Could the authors report FLOPs, forward passes, and latency, and compare HMDLM with T steps against MDLM with approximately 2T steps under matched compute?

1 comment
AuthorsOpenReview ↗

Response to Reviewer: 4o1f

Q: Compare against GenMD4 and LoMDM baselines, and clarify its theoretical novelty.

We thank the reviewer for raising this. We clarify below how HMDLM differs from these methods, and we add a direct comparison on Sudoku.

Theoretical novelty. GenMD4 and LoMDM both use state-dependent masking schedules, so different tokens can be unmasked at different rates.

  • GenMD4 ties the schedule to the token identity. Each vocabulary item has one fixed rate that is the same in every context. So it cannot express priorities that depend on position or context, such as highly constrained cells in Sudoku.
  • LoMDM is closer to our setting. It learns a context-dependent rate for each position. However, these rates only decide which positions are unmasked at each step. The token predictor never sees them, so the predicted token distributions do not depend on the schedule.

HMDLM differs in that the token predictions themselves use the schedule. We introduce a latent hierarchy state with a one-to-one mapping between a position’s hierarchy level and its masking schedule. The model runs in two passes. The first pass predicts hierarchy probabilities and samples a level for each position. The second pass predicts each token conditioned on the hierarchy levels of the entire sequence. Each level corresponds to exactly one schedule, so every token prediction uses the schedule information of the whole context, both at the position being predicted and at the positions around it. Neither prior method does this.

Sudoku-hard

MethodSolve rate (%)Cell accuracy (%)
HMDLM v290.097.4
HMDLM v186.1 ± 0.0896.24 ± 0.07
LoMDM85.96 ± 0.2195.17 ± 0.05
GenMD485.59 ± 0.2295.23 ± 0.07
LoMDM (sched_lr = 1e-4)83.36 ± 0.1894.43 ± 0.08
MDLM81.393.7

HMDLM v1 is the model from the submitted paper. HMDLM v2 has the same architecture and objective. The only change is in the training and inference algorithm: the second pass is conditioned on hierarchy levels sampled from the first pass (for all the masked tokens), as described in the common response. Even v1 beats the baselines on cell accuracy at 96.24% against 95.23% for GenMD4, and its solve rate is close to the baselines. With the revised training, v2 outperforms all baselines on both metrics, reaching a 90.0% solve rate against 85.96% for the strongest baseline, LoMDM.

Reviewer_3fkQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchical Masked Diffusion Language Models (HMDLMs), which introduce a generation order by assigning priority levels to token positions while retaining parallel decoding. Each hierarchy level has its own masking schedule, and separate heads predict token priorities and token identities. The authors derive the corresponding NELBO training objective and evaluate the method on different tasks of language modeling, summarization, and Sudoku. The results show an improvement in perplexity over ADLM on LM1B, better summarization quality than MDLM and ADLM, and higher Sudoku accuracy than MDLM and MGDM.

Critical strengths
  • Interesting and simple idea: the task-dependent priority is directly built into both the forward and reverse diffusion processes, rather than being a decoding trick. This results in a controllable generation order without sacrificing parallel decoding.
  • Clear formulation: the authors derive a NELBO that decomposes into a hierarchy term and a token term. There is a clear connection between this NELBO and the proposed two-stage architecture.
  • Lightweight and flexible model: the method only adds about 0.15% parameters more, and the hierarchy can be defined in several ways (frequency, position, constraint rounds).
  • Promising gains: the method shows improvements on both natural text generation and Sudoku. The frequency variant improves over baselines on LM1B and WikiHow, and HMDLM generally outperforms both MDLM and MGDM on Sudoku. Also on Sudoku, the qualitative analysis supports the hierarchical mechanism: ambiguous or hard cells are not decoded until fewer candidates remain.
Critical weaknesses
  • Limited baselines: the comparisons are somewhat incomplete, since several relevant ordering methods missing. In particular, comparing against MDLM with adaptive (confidence- or entropy-based) decoding would help establish how much HMDLM adds beyond adaptive decoding. Stronger recent baselines such as ReMDM, DUO, or block diffusion would also better position the gains. In addition, the LM1B baseline is taken from prior work and may not have been trained under identical conditions. The ADLM results also use a different hierarchy threshold than HMDLM, so part of the reported gain over ADLM may be due to the threshold choice rather than the method itself.
  • Unclear source of improvement: besides the threshold issue noted above, the experiments do not disentangle the contributions of individual components. They do not separate errors from hierarchy prediction and token prediction, or isolate the effects of the hierarchy, masking schedule, and predicted order. Further ablations could help clarify which components drive the gains.
  • Insufficient diversity analysis: some WikiHow results suggest that HMDLM may produce less diverse outputs than the reference data (lower GenPPL but also lower entropy). It would be important to verify that the quality gains do not come at the cost of reduced diversity. Providing qualitative samples would also help.
  • Limited discussion of limitations: the paper does not quantify the extra computational cost of its two-stage design. Moreover, the experiments are limited to relatively simple datasets, small models, short contexts, and simple hierarchies, and it remains unclear how easily the hierarchy can be designed for new tasks.
Questions for authors
  1. How much do hierarchy prediction errors affect performance? Can the authors compare predicted and ground-truth hierarchies and report the two NELBO terms separately?

  2. Ablations of the main components: how much do the hierarchy definition, the per-level masking schedules, and the predicted ordering each contribute to the gains?

  3. Comparison with alternative decoding methods and more complex datasets: how does HMDLM compare with MDLM using adaptive decoding, or with learned-order and planning methods? Regarding datasets, as suggestion for future work, adding experiments on other benchmarks such as Zebra puzzles or Countdown, which are also small-scale but pose different challenges, would strengthen the paper.

  4. What is the computational overhead of the two-stage design at training and inference time? How well would HMDLM transfer to larger models and to tasks where the hierarchy is less obvious, such as code or mathematical reasoning?

Minor: Figure 1 appears blurry, the authors are encouraged to use a vector format. Also, Section 6 is titled “Conclusion and limitations,” but no limitations are stated (beyond briefly mentioning the additional computation).

1 comment
AuthorsOpenReview ↗

Response to the reviewer: 3fkQ

Q: Insufficient diversity analysis (on reduced entropy).

We thank the reviewer for their comment. The entropy we report is the unigram entropy within each sample, so the right target is the reference value, not the highest possible value. Text that is incoherent or drifts off topic has unusually high entropy. This is the case for MDLM and ADLM, which exceed the reference at every length (e.g. 7.60 and 7.77 vs. 6.70 at L = 1024), as their samples show.

EntropyL = 256L = 512L = 1024
Reference5.866.426.70
MDLM6.477.107.60
ADLM6.497.177.77
HMDLM-P5.916.266.45
HMDLM-F5.986.606.83

HMDLM-F stays slightly above the reference entropy at every length. It also has Gen PPL close to the reference and the highest MAUVE at every length (0.52 / 0.41 / 0.20 vs. 0.33 / 0.11 / 0.12 for MDLM). This indicates a closer match to the reference distribution, not reduced diversity.

HMDLM-P. We agree that this variant shows a mild reduction in diversity. Its entropy is slightly below the reference at L = 512 and 1024, and its Gen PPL is below the reference. Its generations were still preferred most often in the blinded LLM-as-judge evaluation of fluency, coherence, and adherence: 40.75%, vs. 31.88% for HMDLM-F, 15.25% for ADLM, and 12.12% for MDLM. We therefore present HMDLM-F as the variant that improves quality without sacrificing diversity.

Qualitative samples from MDLM, ADLM, and HMDLM-F on two WikiHow prompts are provided in Appendix E. They show that the baselines' higher entropy reflects topic drift and invented words rather than useful diversity. For example, MDLM moves from rear-shock replacement to antenna kits and zoom lenses, and ADLM produces non-words such as “valvages” and “cussist.”

Q. How much do hierarchy prediction errors affect performance? Can the authors compare predicted and ground-truth hierarchies and report the two NELBO terms separately?

We thank the reviewer for this question. On Sudoku, we compared HMDLM with the predicted hierarchy (Pass 1) against an oracle that receives the gold hierarchy and skips Pass 1.

Hierarchy accuracy. The Stage-1 predictor recovers the gold hierarchy level for 91.25% of cells with argmax decoding and 90.71% with sampling.

Hierarchy - Only inferenceSolve rateCell accuracyNELBO
Gold hierarchy (oracle, no Pass 1) Inference only0.8596 ± 0.00050.96450.0070 ± 0.0007
Predicted hierarchy (Pass 1)0.8610 ± 0.00080.96440.0759 ± 0.0020

Findings. On Sudoku, solve rate and cell accuracy are statistically indistinguishable between the two settings, so hierarchy prediction errors cause no measurable loss in performance. This is notable because Stage 2 is trained only on gold hierarchy levels. Predicted levels therefore create a train–test mismatch that the oracle does not have, and HMDLM is robust to both this mismatch and the roughly 9% of Stage-1 errors. Since the oracle hierarchy brings no gain, the remaining errors come from token prediction, not from the hierarchy stage.

Reviewer_3anaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed HMDLM, an extension of masked diffusion language models (MDLM) in which each token position carries a discrete hierarchy level, represented as one-hot vector over K levels. The level is derived from the data. Token and level are masked and revealed jointly through a shared Bernoulli random variable. The reverse process has two stages: 1. Predict a distribution over levels at masked positions; 2. predict the token, conditioned on the newly revealed levels. The authors derived the loss function that contains the hierarchy KL and token KL. They showed experimental results on three datasets: LM1B, WikiHow, and Sudoku.

Critical strengths
  1. The idea is well motivated and generalizes the anchor-based approaches.
  2. The Sudoku experiment shows the advantage of the proposed method.
  3. The approach adds a relatively small amount of model parameters on top of a standard MDLM backbone.
Critical weaknesses
  1. OpenWebText (OWT) is the standard larger benchmark in assessing discrete diffusion language models. Both MDLM and ADLM have reported OWT results. I think LM1B is a weak test of the hierarchy idea. Longer OWT documents would be a more meaningful test for the proposed approach. There is no evidence about whether gains persist, grow, or vanish at larger data or model scale.
  2. All Table 1 baselines are copied from Rout et al. (2025). The HMDLM gain over ADLM uses τ=1\tau= 1, while ADLM’s published numbers use τ=5\tau= 5, and HMDLM at τ=5\tau= 5 is reported to match MDLM. The improvement over ADLM may therefore come from how the hierarchy is defined rather than the method. It might be a more fair comparison to add ADLM with τ=1\tau= 1 or ADLM with tuned τ\tau.
  3. Table 2 does not have results from ADLM.
  4. Compute or wall-clock-matched comparisons are missing. It seems that HMDLM is trained for more steps to match supervised tokens, and each step requires two forward passes. The extra training cost is not clearly discussed in the paper.
Questions for authors
  1. Could the authors add results on OpenWebText to show benefits of the proposed method on more complex text data?
  2. Could the authors evaluate ADLM with the same frequency threshold, and include it in Table 2?
  3. Could the authors provide wall-clock time or flops for training and inference against the baselines?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 3ana

1. Results on OWT.

We thank the reviewer for this suggestion. We agree that OpenWebText (OWT) is the standard larger-scale benchmark for discrete diffusion language models, and its longer documents provide a more direct test of our hierarchical design than LM1B. We are currently running these experiments and will include the results in the revised version.

2. Can ADLM be evaluated on the Tau=1 (same HMDLM) ?

We agree this is the right comparison, and we do run it — on WikiHow, in the submitted paper.

We fine-tune the released ADLM-OWT checkpoint ourselves, from the same initialization and for the same number of steps as HMDLM, and we define ADLM's anchor positions with exactly the same τ=1 within-sentence frequency rule that defines HMDLM's hierarchy (Section 5.2, "Hierarchy Assignment"). The hierarchy definition and the training conditions are therefore held fixed, and the only difference is the method. From Table 2 of the updated submission:

ModelGen PPL (256)Gen PPL (512)Gen PPL (1024)MAUVE (256)MAUVE (512)MAUVE (1024)Entropy (256)Entropy (512)Entropy (1024)
Gold25.221.020.81.001.001.005.866.426.70
MDLM76.067.462.70.330.110.126.477.107.60
ADLM84.082.193.80.400.210.086.497.177.77
HMDLM-P23.117.614.60.380.130.075.916.266.45
HMDLM-F29.225.419.90.520.410.205.986.606.83

HMDLM-F improves Gen PPL at every length, improves MAUVE at every length, and keeps entropy close to the reference where ADLM exceeds it by 0.6–1.1 bits. In the blinded LLM-as-judge study (400 samples, 2 runs, 800 comparisons), HMDLM-F is preferred 31.88% of the time against ADLM's 15.25%.

3. Table 2 does not have results from ADLM

We note that the ADLM paper does not report results for the Table 2 evaluation (Table 1 in the updated submission). We tried to train the ADLM from scratch using the setting described in the paper; however, our training did not converge. The LM1B checkpoint used in the ADLM paper is not publicly released, and we contacted the authors but received no reply. Hence, we could not evaluate Table 2 metrics (Table 1 in the updated submission) on ADLM. We will be happy to report the comparisons once we get access to the checkpoint.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to common questions

Common

For All Reviewer's:

We thank all the reviewers' for their insightful comments, providing feedback on the paper, and appreciating the novelty of the paper. While we are responding to questions/concerns raised by each reviewer individually, there are some points common to multiple reviewers, which we are responding to in the text below. We would be happy to answer any further questions with regards to our response.


Correction on model perplexity

Reviewer 4o1f pointed to a potential issue with the way we are computing the model PPL for HMDLM. The following note relates to the said concern.

Since our model involves likelihood computation over both the token and hierarchy, we need to marginalize over hierarchy, resulting in an intractable sum. We had proposed a specific computation for the same in the paper, and used a sampling based estimate, but as pointed out by Reviewer 4o1f, there may be some mistake in the way we derived the specific expression for computing the lower bound of the likelihood. On further examination, we found that another way to compute the correct bound, by using the fact that NELBO for data sample (and p(x0)≥p(x0,h0)p(x_0) \geq p(x_0, h_0), x0,h0x_0, h_0 being the gold sample), results in a very loose bound, given that it is trying to bound the marginalized probability using NELBO for data sample.

Accordingly, we are working on an updated tighter bound, which 1.applies NELBO for each possible hierarchy value separately, and 2. approximates the sum via sampling over hierarchy values, while accounting for all the relevant terms in the NELBO expression.We are currently carefully checking its correctness. Once validated, we will update the paper with the correct bound, and also report the corresponding experiments. In general, we would also like to add that deriving a bound, which can be estimated efficiently via Monte Carlo sampling, may be a hard problem, due to an intractable sum, and possibly having an unbiased, but very high variance esimator. We looked at existing literature, and could not find many pointers on computing the perplexity over a marginalized set of variables in discrete diffusion models. Nevertheless, we would be happy to report our findings once we validate our new formulation, and are able to do experiments, and report them in a newer version of the paper.

In the meanwhile, we would like to submit that any results reported on the model PPL for our model in the paper are withdrawn. We regret any confusion caused in this regard due to our earlier reported numbers, any in incorrectness that might have creeped in due to an oversight in the earlier submitted version of the paper.. Since the originally reported model perplexities do not represent valid likelihood bounds, we do not discuss questions based on them here. Further, we are updating the paper removing any results containing the model PPL computation. We will add results on model PPL once we have carefully examined our new formulation, and done experimentation with the same.

1. On ablations of the main components: hierarchy definition, the per-level masking schedules

We thank the reviewer for raising this. Further, based on reviewer’s questions and comments, we also experimented with another variant of approach and we are reporting the numbers for the same in the comparisons below for a comprehensive analysis.

To identify which components drive the gains, we ran ablations on Sudoku-hard with random-order decoding.

HMDLM v1 vs. v2. Both variants use the same two-pass model. The first pass predicts the hierarchy level of each masked position, and the second pass predicts the token. They differ only in what the second pass sees at masked positions:

  • v1 (submitted paper): At inference, the positions scheduled to be unmasked at the current step (the reveal set R) receive their hierarchy level as predicted by the first pass; during training they receive the gold level. All other masked positions receive a mask token. The second pass therefore knows the levels of only the tokens it is about to reveal.
  • v2: (new variant): The second pass receives a hierarchy level at every masked position. During training these are sampled hierarchy levels. At inference, we sample a level for each masked position from the first pass’s prediction. The second pass therefore always conditions on the levels of all masked tokens, including those revealed in later steps.

This changes only how the hierarchy is passed to the second pass. The architecture and parameter count are the same. (Response continued in the next comment.)

AuthorsOpenReview ↗

Response to common questions [continued]

(Continued from the previous comment.)

1. On ablations of the main components (continued)

Sudoku-hard

MetricSolve rateCell accuracy
MDLM0.8130.937
HMDLM v1 (random sampling)0.8610.964
HMDLM v2 (random sampling)0.9000.974
Ablation
HMDLM v1 - same noise schedule (ablation on hierarchy component)0.8430.965
HMDLM - no hierarchy conditioning, (ablation on effect of different noise schedules)0.8500.956

Effect of the hierarchy. With MDLM’s single noise schedule, adding hierarchy conditioning alone improves over MDLM by +3.0 pp (solve) and +2.8 pp (cell). It matches HMDLM v1 in cell accuracy (96.5 vs. 96.4) but trails v1 by 1.8 pp and v2 by 5.7 pp in solve rate. The hierarchy therefore improves token prediction, but the level-specific noise schedules are needed for the full gain in solve rate.

Effect of the noise schedules. To isolate the schedules, we train a single-pass model with two heads, one for hierarchy and one for tokens. The predicted hierarchy sets each position’s masking schedule, but the token head is not conditioned on it. This variant improves over MDLM by +3.7 pp in solve rate and +1.9 pp in cell accuracy, so level-specific schedules help on their own. It still trails HMDLM v1 and v2 by 1.1 and 5.0 pp in solve rate, and by 0.8 and 1.8 pp in cell accuracy. The full gain therefore also needs hierarchy conditioning in the token predictor.

We also run experiments on WikiHow

MetricGen PPLMauveEntropy
Gold20.81.006.70
MDLM62.70.127.60
HMDLM-F19.920.206.83
HMDLM-F, same noise schedule (ablation on hierarchy component)17.570.156.56
HMDLM-F - no hierarchy conditioning, (ablation on effect of different noise schedules)12.950.046.16

Effect of the hierarchy. On WikiHow, hierarchy conditioning provides most of the gain. Under MDLM’s noise schedule, it cuts generative perplexity from 62.7 to 17.57 and brings entropy from 7.60 to 6.56, close to gold (6.70); MDLM’s higher entropy reflects incoherent text, not diversity. MAUVE rises from 0.12 to 0.15.

Effect of the noise schedules. Combined with the hierarchy, level-specific schedules raise MAUVE to 0.20 and bring perplexity (19.92) and entropy (6.83) closest to gold. Without hierarchy conditioning, MAUVE falls to 0.04, and perplexity (12.95) and entropy (6.16) drop well below gold, indicating repetitive text. The schedules thus help only together with the hierarchy.

Conclusion. On Sudoku, each component improves over MDLM on its own, with the hierarchy mainly helping cell accuracy and the schedules mainly helping solve rate. On WikiHow, the hierarchy provides most of the gain, and the schedules add to it only in combination. The full models (HMDLM v2 on Sudoku, HMDLM-F on WikiHow) perform best overall, showing that the components are complementary.

2. Comparison with adaptive decoding methods and learned order baselines.

We thank the reviewer for these suggestions. We have added the following experiments.

Adaptive decoding. To isolate what HMDLM adds beyond adaptive decoding, we compare MDLM and HMDLM under the same decoding strategies on Sudoku-extreme, a harder variant of Sudoku-hard with fewer given clues. On Sudoku-hard, confidence-based sampling already reaches 100% accuracy, so differences between methods can’t show up there.

DecodingMDLMHMDLM v1HMDLM − MDLM
Random order, sampled digits0.1165 ± 0.00260.1377 ± 0.0017+2.1pp
Random order, greedy digits0.1236 ± 0.00170.1524 ± 0.0017+2.9pp
Max-prob confidence0.18510.2415 ± 0.0004+5.6pp
Margin confidence0.27590.2905 ± 0.0006+1.5pp
Entropy confidence0.27750.2935 ± 0.0002+1.6pp

HMDLM improves over MDLM under every decoding strategy. So the gain from the hierarchy adds to the gain from adaptive decoding; adaptive decoding alone doesn’t account for it.

Learned-order baselines. On Sudoku-hard, we compare against learned-order methods that learn per-position (LoMDM) or per-value (GenMD4) unmasking schedules:

MethodSolve rate (%)Cell accuracy (%)
HMDLM v290.097.4
HMDLM v186.196.24 ± 0.07
LoMDM85.96 ± 0.2195.17 ± 0.05
GenMD485.59 ± 0.2295.23 ± 0.07
MDLM81.393.7

HMDLM v2 outperforms all learned-order baselines.

WikiHow. We also evaluated confidence-based decoding on WikiHow (L = 1024, 256 steps, 4,000 samples) for MDLM and HMDLM-P. Neither model produces usable text: max-prob decoding collapses into repetition (Gen. PPL ≈ 1.4 for both), while entropy- and margin-based decoding yield incoherent text (Gen. PPL 360–413). In these regimes, Gen. PPL and MAUVE no longer reflect quality; since both models fail alike, the failure stems from the decoders, not the hierarchy.

AuthorsOpenReview ↗

Response to common questions [continued]

(Continued from the previous comment.)

3. Could the authors provide wall-clock time or flops for training and inference against the baselines?

We thank the reviewer for this question. HMDLM does run the backbone twice per denoising step. So with T steps it uses 2T+2 function evaluations (NFEs, or the Network Function Evaluations), while MDLM uses T+1. The extra parameters in HMDLM are negligible, so FLOPs and latency are set almost entirely by the number of backbone passes.

Below we report compute costs and then compare the two models under matched compute.

Sudoku

ModelStepsNFEsFLOPsGPU train hoursLatency / sample
MDLM (25M)8182≈ 282 GFLOPs10.06.41 ms
HMDLM (25M)81164≈ 564 GFLOPs22.312.51 ms
GenMD4 (25M)8182≈ 282 GFLOPs17.86.31 ms
LoMDM (29.3M)8182≈ 305 GFLOPs21.25.30 ms
MGDM (6M)8182≈ 58.8 GFLOPs7.73.9 ms
HMDLM (6M)81164≈ 117.5 GFLOPs13.27.5 ms

As expected, at the same number of steps HMDLM uses about 2× the FLOPs and about 2× the wall-clock time of MDLM.

WikiHow

ModelStepsNFEsFLOPsGPU train hoursLatency / sample
MDLM256257≈ 65.0 TFLOPs52.91.09 s
HMDLM-(f,p)256514≈ 109.7 TFLOPs85.72.06 s
GenMD4256257≈ 65.0 TFLOPs90.81.56s
LoMDM256257≈ 68.7 TFLOPs70.41.23s
ADLM256257≈ 128.0 TFLOPs59.14.48 s

Matched Compute: To match compute, we halve the number of HMDLM steps and compute the accuracy on Sudoku and GEN PPL, MAUVE, and Entropy for WikiHow.

WikiHow

Model / SettingGenPPLMAUVEEntropy
HMDLM-P, T=25614.560.076.46
HMDLM-P, T=12815.230.0676.42
HMDLM-F, T=25619.920.206.83
HMDLM-F, T=12821.030.246.88
ADLM, T=25693.810.087.77
MDLM, T=25662.50.127.61
Gold20.81.06.73

At the same compute, HMDLM-F improves on MDLM in every metric:

  • Generative perplexity: about 3× lower (21.03 vs. 62.5).
  • MAUVE: doubled (0.24 vs. 0.12).
  • Entropy: 6.88, close to the gold value of 6.73. MDLM’s entropy is 7.61, which points to less coherent, noisier text rather than more diverse text.

Sudoku

Model / SettingBoard AccuracyCell Accuracy
HMDLM-v2, T=810.890.97
HMDLM-v2, T=400.880.97
HMDLM-v1, T=810.860.96
HMDLM-v1, T=400.850.96
LoMDM — learned per-POSITION, T=810.860.95
GenMD4 — learned per-VALUE, T=810.850.95
MDLM, T=810.810.93

At the same compute, both HMDLM variants still beat MDLM by a wide margin. Board accuracy is +4 points for v1 and +7 points for v2, and cell accuracy is +3 to +4 points. Halving the steps costs HMDLM only about 1 point of board accuracy and no cell accuracy.

HMDLM v2 has the same architecture and objective. The only change is in the training and inference algorithm: the second pass is conditioned on hierarchy levels sampled from the first pass (for all the masked tokens), as described in the common response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.