- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether masked diffusion models (MDMs), whose any-order training objective gives them an any-order prediction interface, actually perform any-order inference at decoding time. The first answer is negative: for several 7B MDMs (Dream and DiffuCoder variants), programs produced by confidence-based decoding are structurally close to programs from strict left-to-right decoding of the same model, measured by AST tree-edit similarity with best-match aggregation against 32 L2R samples per prompt. The authors attribute this "causal collapse" to positional uncertainty — the model knows what token to emit but not where — and visualize it with aggregate-mass vs. localization (LOC) plots, showing the pattern persists even for an RL-learned unmasking policy. They then propose two remedies: (1) FlexMDM (insertion-based MDM, building on Kim et al. 2025a), fine-tuned from Dream-Coder 7B on ~2.6M Python sequences (~1150 GPU hours), which decouples unmasking from fixed positions, exhibits non-contiguous generation (CBC/RUB/OBW metrics), and beats Dream-Coder at Pass@16 on HumanEval/HumanEval+; (2) LatentMDM, a new segment-level latent masked diffusion model pretrained from scratch at 125M on TinyGSM (500k steps, batch 256), which chooses which latent segment to realize next and beats MDM, AR-with-KV-cache, DUO, and S-FLM baselines as well as its own L2R/random selection variants under matched wall-clock.
Critical strengths
S1. The interface-vs-inference framing is a genuinely useful lens for the diffusion-LM community. The positional-uncertainty diagnosis (aggregate mass vs. LOC scatter plots) is simple, reusable, and explains why confidence-based decoding collapses to left-to-right, and the check that an RL-learned planner (Chen et al., 2025) does not escape the collapse strengthens the claim that this is a property of fixed-canvas token-level decoding, not of one particular score function.
S2. LatentMDM is a real modeling contribution with the right controls: the L2R and random segment-selection variants differ from the full model only in the selection policy (Table 7, k=1: 45.5 vs. 41.3 vs. 40.3), and the matched-wall-clock comparison against AR with KV caching, DUO, and S-FLM with Pareto-tuned step counts is exactly how a diffusion LM should be evaluated. Reporting full pass@k tables, the insertion-temperature sweep, and concrete compute (~1150 GPU hours for the 7B fine-tune; ~3 days on 4×A100-80GB for LatentMDM) is appreciated.
Critical weaknesses
W1. The downstream evidence does not robustly support the abstract's claim that the two approaches "improve downstream performance." In Table 5, FlexMDM loses 8.00 pass@1 on HumanEval (50.65 vs. 58.65) and 7.28 on HumanEval+ (46.61 vs. 53.89), winning only at pass@16 (+1.22 and +2.43); on MBPP/MBPP+ the pattern flips (pass@1 +0.17 and +1.00, but pass@16 −1.59 and −1.32). The main text picks the favorable metric per benchmark family (Pass@16 for HumanEval, Pass@1 for MBPP) and describes the rest as "on par," which overstates the case. As presented, the results demonstrate a diversity-vs-local-precision trade-off whose direction depends on the benchmark, not a consistent improvement.
W2. The main modeling contribution is validated only at a very small scale, and its headline margins are not tabulated. LatentMDM is tested only at 125M on TinyGSM; the "more than 6%" gap over the L2R/random variants and the "5.5%" gap over baselines at matched wall-clock appear only in Figure 5, while the only tabular policy ablation (Table 7, greedy decoding) shows smaller gaps (+4.2 over L2R, +5.2 over random at k=1). Relatedly, CBC/RUB/OBW show that FlexMDM's generation order differs from L2R, but nothing ties that behavioral difference causally to quality, and the FlexMDM half of the remedy side is largely an extension and re-scaling of prior work (Kim et al., 2025a) rather than a new model.
Questions for authors
Q1. Could you report the Figure 5 (left) numbers in a table (pass@K and wall-clock per model), and clarify whether the "6%" and "5.5%" margins are absolute points or relative? Given Table 7 shows +4.2/+5.2 at k=1 with greedy decoding, it matters how much the headline margins depend on the sampling configuration.
Q2. How do you interpret FlexMDM's pass@1 regression on HumanEval/HumanEval+ (−8.00/−7.28)? Does insertion hurt locally-determined code, and does any insertion temperature in the Table 6 sweep recover it on HumanEval, or is the knob only useful on MBPP-style prompts?
Q3. Does the positional-uncertainty diagnosis hold outside Python code and at other scales? A mass-vs-LOC plot on a non-code task (or a much smaller/larger MDM) would support the claim that this is a property of fixed-canvas token-level MDMs in general, rather than of code targets with rigid syntactic scaffolds.
5 comments
Q3. Does the positional-uncertainty diagnosis hold outside Python code and at other scales? A mass-vs-LOC plot on a non-code task (or a much smaller/larger MDM) would support the claim that this is a property of fixed-canvas token-level MDMs in general, rather than of code targets with rigid syntactic scaffolds.
To test this, we repeat the same positional uncertainty analysis on LLaDA 2.0 16B, a bigger model, and Humaneval (Python code) and GSM8k (math). As shown in https://ibb.co/hxk7LNNf, we observe a similar trend. These results suggest that the positional uncertainty phenomenon is a rather general issue of fixed-canvas masked diffusion rather than an observation confined to our evaluated setup in the manuscript.
Q2. How do you interpret FlexMDM's pass@1 regression on HumanEval/HumanEval+ (−8.00/−7.28)? W1. The downstream evidence does not robustly support the abstract's claim that the two approaches "improve downstream performance.
Prompt format on HumanEval. Re-examining this gap, we found a difference in our HumanEval setup that was previously overlooked. FlexMDM receives the standard HumanEval prompt (function signature and docstring), the same prompt Dream-Coder's own evaluation uses for its base model. For Dream-Coder, our paper instead used the EvalPlus instruction wrapper, because our any-order metrics are computed on the whole generated function, and with the standard prompt Dream-Coder usually writes only the function body. Table 5 reused this setup. (On MBPP, both models receive Dream-Coder's official MBPP prompt.) To isolate the effect of the prompt, we ran the two missing combinations (164 problems × 16 samples; † marks the pairing in Table 5):
HumanEval:
| Model, prompt | Pass@1 | Pass@2 | Pass@4 | Pass@8 | Pass@16 |
|---|---|---|---|---|---|
| FlexMDM, standard † | 50.65 | 66.60 | 78.69 | 86.86 | 92.07 |
| FlexMDM, wrapper | 49.16 | 63.53 | 75.49 | 84.74 | 90.85 |
| Dream-Coder-7B, standard | 39.02 | 55.72 | 70.57 | 81.60 | 89.63 |
| Dream-Coder-7B, wrapper † | 58.65 | 72.22 | 81.34 | 86.94 | 90.85 |
HumanEval+:
| Model, prompt | Pass@1 | Pass@2 | Pass@4 | Pass@8 | Pass@16 |
|---|---|---|---|---|---|
| FlexMDM, standard † | 46.61 | 61.89 | 73.83 | 82.07 | 87.80 |
| FlexMDM, wrapper | 45.54 | 59.18 | 70.89 | 80.43 | 86.59 |
| Dream-Coder-7B, standard | 34.68 | 50.40 | 64.87 | 75.12 | 81.10 |
| Dream-Coder-7B, wrapper † | 53.89 | 67.05 | 75.96 | 81.53 | 85.37 |
† rows: Table 5 as published. The other two rows are new runs with Table 5's settings and seeds, with outputs cut at the first end-of-sequence token, as in Dream-Coder's official evaluation.
With the standard prompt for both models, FlexMDM outperforms Dream-Coder at every k, by 11.62 points at Pass@1 on HumanEval and 11.93 on HumanEval+ (95% CIs [7.96, 15.28] and [8.16, 15.62]). The wrapper raises Dream-Coder's Pass@1 by 19.63 points but changes FlexMDM's by only −1.49 [−3.93, +0.91]. Table 5 thus paired each model with the prompt that suits it better, so its 8.00-point gap is a best-versus-best comparison, not an artifact; but which model leads on HumanEval depends on the baseline's prompt, to which Dream-Coder is highly sensitive and FlexMDM is not. We will state the prompt formats in Appendix D.4 and report this experiment in the appendix.
Does insertion hurt locally-determined code, and does any insertion temperature in the Table 6 sweep recover it on HumanEval, or is the knob only useful on MBPP-style prompts?
Reducing the insertion temperature provides a small Pass@1 improvement on HumanEval, but does not recover the gap to Dream-Coder.
| Benchmark | Pass@1 | Pass@2 | Pass@4 | Pass@8 | Pass@16 | |
|---|---|---|---|---|---|---|
| HumanEval | Dream-Coder-7B-Base | 58.65 | 72.22 | 81.34 | 86.94 | 90.85 |
| 1.0 (default) | 50.65 | 66.60 | 78.69 | 86.86 | 92.07 | |
| 0.70 | 51.64 | 67.02 | 77.90 | 85.49 | 91.46 | |
| HumanEval+ | Dream-Coder-7B-Base | 53.89 | 67.05 | 75.96 | 81.53 | 85.37 |
| 1.0 (default) | 46.61 | 61.89 | 73.83 | 82.07 | 87.80 | |
| 0.70 | 47.45 | 62.20 | 73.31 | 81.51 | 87.80 |
Sharpening to raises Pass@1 by 0.99 (HumanEval) and 0.84 (HumanEval+) points, about one-eighth of the gap, and lowers Pass@4 and Pass@8. On MBPP/MBPP+, by contrast, raises Pass@1 by 2.48/2.12 points and improves every k (Table 6).
So in our sweep, the knob is useful on MBPP-style prompts but not on HumanEval. This fits what the knob does: it sharpens where new slots are inserted without changing how many are inserted. MBPP prompts give only a description and tests, so the model must build the program's structure itself, and noisy early placements can derail it; sharpening them helps at every k.
HumanEval prompts already fix the structure (signature and docstring), so placement noise mostly varies otherwise similar programs, which adds diversity at large k and costs about a point at k = 1. Together with the prompt experiment above: across the three FlexMDM settings we evaluated on HumanEval (either prompt at , or with the standard prompt), FlexMDM's Pass@1 stays between 49.16 and 51.64, while Dream-Coder's moves by 19.63 points with its prompt alone.
In short. On HumanEval, Table 5 gives Dream-Coder the more favorable of two prompt formats. FlexMDM receives the standard prompt (signature and docstring), which is used in Dream-Coder's own evaluation. But we also tried giving Dream-Coder that prompt inside an instruction wrapper that explicitly asks for a complete solution. This wrapper raises Dream-Coder's Pass@1 by about 20 points, while it does not help FlexMDM (−1.5 points). With the standard prompt for both models, FlexMDM leads Dream-Coder by 11.6 Pass@1 points on HumanEval. In prompt format, Table 5 thus favors Dream-Coder. Sharpening insertion helps on MBPP but adds only about one Pass@1 point on HumanEval and lowers Pass@4 and Pass@8, likely because HumanEval's signature and docstring already fix the program's structure. Across all FlexMDM settings we tried, its HumanEval Pass@1 stays between 49.2 and 51.6, while Dream-Coder's moves by about 20 points with the prompt alone. FlexMDM's HumanEval performance is therefore robust to the decoding settings, and the direction of the gap depends strongly on how Dream-Coder is prompted.
We thank the reviewer for acknowledging our contribution and suggesting meaningful clarification. Below, we address these points.
We apologize for the lack of clarity in the evaluation setup. Figure 5 reports pass@K as a function of wall-clock inference time using stochastic sampling with temperature (T=1.0). The reported margins of 6% over the L2R/random variants and 5.5%over the modeling baselines (ARM) were estimated by interpolating between measured points to compare performance at approximately matched inference budgets.
In contrast, Table 7 uses greedy decoding (T=0). Under k=1, sampling temperature is the only difference between these two experiments. These results therefore reflect different sampling regimes and need not yield the same accuracy gaps. We will clarify this distinction and the interpolation procedure in the manuscript.
The table above provides the tabulated pass@K values and corresponding inference times underlying Figure 5, measured on a single 80GB NVIDIA A100 GPU. Here, (K) denotes the number of sampled solutions, whereas (k) in Table 7 denotes the number of segments committed per iteration.
Further, we extended the TinyGSM experiments to 537M-parameter models. Specifically, we trained LatentMDM with capacity for 64 segments of up to 32 tokens each, alongside same-size baselines, for 500k optimizer steps. Under greedy decoding, LatentMDM achieves 56.48% accuracy, compared with 57.54% for ARM and 33.97% for MDM, while providing 5× higher throughput and lower peak allocated memory(0.6-0.8x) than ARM. The additional table reports pass@K and corresponding inference times for the new 537M models, measured on a single 180GB NVIDIA GB200 GPU. While TinyGSM remains a narrow evaluation setting, these results provide encouraging evidence that LatentMDM’s accuracy–efficiency benefits extend to a larger model scale.