acceptodds
Under review as a conference paper at ICLR 2027

From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

Abstract

Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a process we call any-order inference. For autoregressive language models, which lack a native any-order interface, non-causal abilities such as infilling and next-edit prediction require hand-designed mechanisms. Can we instead design models that natively support any-order inference? Masked diffusion models have recently emerged as compelling candidates, as their any-order training objective naturally offers an any-order prediction interface. This interface, however, does not automatically yield any-order inference. We demonstrate that this interface-inference gap stems from positional uncertainty: fixed-canvas, token-level models may know what semantic component should appear without knowing where to place it. In light of this, we propose two complementary approaches: (1) Insertion-based masked diffusion, building on FlexMDM, relaxes fixed-position commitments via insertions, enabling generation across non-contiguous regions. (2) Latent-space masked diffusion shifts prediction to coarser semantic segments, enabling search over latent generation orders. Empirically, we train a 7B FlexMDM for Python coding and a 125M LatentMDM for GSM8K and show that both approaches induce distinct any-order inference behaviors and improve downstream performance.

Primary area:
generative models
Keywords:
masked diffusion models, any-order inference, insertion-based generation, latent-space diffusion, code generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FAzXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether masked diffusion models (MDMs), whose any-order training objective gives them an any-order prediction interface, actually perform any-order inference at decoding time. The first answer is negative: for several 7B MDMs (Dream and DiffuCoder variants), programs produced by confidence-based decoding are structurally close to programs from strict left-to-right decoding of the same model, measured by AST tree-edit similarity with best-match aggregation against 32 L2R samples per prompt. The authors attribute this "causal collapse" to positional uncertainty — the model knows what token to emit but not where — and visualize it with aggregate-mass vs. localization (LOC) plots, showing the pattern persists even for an RL-learned unmasking policy. They then propose two remedies: (1) FlexMDM (insertion-based MDM, building on Kim et al. 2025a), fine-tuned from Dream-Coder 7B on ~2.6M Python sequences (~1150 GPU hours), which decouples unmasking from fixed positions, exhibits non-contiguous generation (CBC/RUB/OBW metrics), and beats Dream-Coder at Pass@16 on HumanEval/HumanEval+; (2) LatentMDM, a new segment-level latent masked diffusion model pretrained from scratch at 125M on TinyGSM (500k steps, batch 256), which chooses which latent segment to realize next and beats MDM, AR-with-KV-cache, DUO, and S-FLM baselines as well as its own L2R/random selection variants under matched wall-clock.

Critical strengths

S1. The interface-vs-inference framing is a genuinely useful lens for the diffusion-LM community. The positional-uncertainty diagnosis (aggregate mass vs. LOC scatter plots) is simple, reusable, and explains why confidence-based decoding collapses to left-to-right, and the check that an RL-learned planner (Chen et al., 2025) does not escape the collapse strengthens the claim that this is a property of fixed-canvas token-level decoding, not of one particular score function.

S2. LatentMDM is a real modeling contribution with the right controls: the L2R and random segment-selection variants differ from the full model only in the selection policy (Table 7, k=1: 45.5 vs. 41.3 vs. 40.3), and the matched-wall-clock comparison against AR with KV caching, DUO, and S-FLM with Pareto-tuned step counts is exactly how a diffusion LM should be evaluated. Reporting full pass@k tables, the insertion-temperature sweep, and concrete compute (~1150 GPU hours for the 7B fine-tune; ~3 days on 4×A100-80GB for LatentMDM) is appreciated.

Critical weaknesses

W1. The downstream evidence does not robustly support the abstract's claim that the two approaches "improve downstream performance." In Table 5, FlexMDM loses 8.00 pass@1 on HumanEval (50.65 vs. 58.65) and 7.28 on HumanEval+ (46.61 vs. 53.89), winning only at pass@16 (+1.22 and +2.43); on MBPP/MBPP+ the pattern flips (pass@1 +0.17 and +1.00, but pass@16 −1.59 and −1.32). The main text picks the favorable metric per benchmark family (Pass@16 for HumanEval, Pass@1 for MBPP) and describes the rest as "on par," which overstates the case. As presented, the results demonstrate a diversity-vs-local-precision trade-off whose direction depends on the benchmark, not a consistent improvement.

W2. The main modeling contribution is validated only at a very small scale, and its headline margins are not tabulated. LatentMDM is tested only at 125M on TinyGSM; the "more than 6%" gap over the L2R/random variants and the "5.5%" gap over baselines at matched wall-clock appear only in Figure 5, while the only tabular policy ablation (Table 7, greedy decoding) shows smaller gaps (+4.2 over L2R, +5.2 over random at k=1). Relatedly, CBC/RUB/OBW show that FlexMDM's generation order differs from L2R, but nothing ties that behavioral difference causally to quality, and the FlexMDM half of the remedy side is largely an extension and re-scaling of prior work (Kim et al., 2025a) rather than a new model.

Questions for authors

Q1. Could you report the Figure 5 (left) numbers in a table (pass@K and wall-clock per model), and clarify whether the "6%" and "5.5%" margins are absolute points or relative? Given Table 7 shows +4.2/+5.2 at k=1 with greedy decoding, it matters how much the headline margins depend on the sampling configuration.

Q2. How do you interpret FlexMDM's pass@1 regression on HumanEval/HumanEval+ (−8.00/−7.28)? Does insertion hurt locally-determined code, and does any insertion temperature in the Table 6 sweep recover it on HumanEval, or is the knob only useful on MBPP-style prompts?

Q3. Does the positional-uncertainty diagnosis hold outside Python code and at other scales? A mass-vs-LOC plot on a non-code task (or a much smaller/larger MDM) would support the claim that this is a property of fixed-canvas token-level MDMs in general, rather than of code targets with rigid syntactic scaffolds.

5 comments

We thank the reviewer for acknowledging our contribution and suggesting meaningful clarification. Below, we address these points.

the "more than 6%" gap over the L2R/random variants and the "5.5%" gap over baselines at matched wall-clock appear only in Figure 5, while the only tabular policy ablation (Table 7, greedy decoding) shows smaller gaps (+4.2 over L2R, +5.2 over random at k=1), (Q1)

Accuracy (%)Latency (s/question)
KLatentMDM-confLatentMDM-L2RLatentMDM-randomMDM-confARMLatentMDM-confLatentMDM-L2RLatentMDM-randomMDM-confARM
138.2625.0123.8124.9439.810.460.420.433.161.65
247.7835.4934.2837.4651.770.910.850.876.323.30
352.8841.6840.5845.0258.011.371.271.319.484.94
456.3245.9244.9650.1262.081.831.701.7412.646.59
558.8849.1248.2853.8465.052.292.122.1815.808.24
660.9151.6750.9456.7067.352.742.542.6118.969.89
762.5753.8053.1758.9969.203.202.973.0522.1211.54
863.9955.6455.0960.8970.743.663.393.4825.2813.18
965.2357.2556.7862.4872.044.113.823.9128.4414.83
1066.3458.6858.3063.8473.164.574.244.3531.6016.48

We apologize for the lack of clarity in the evaluation setup. Figure 5 reports pass@K as a function of wall-clock inference time using stochastic sampling with temperature (T=1.0). The reported margins of 6% over the L2R/random variants and 5.5%over the modeling baselines (ARM) were estimated by interpolating between measured points to compare performance at approximately matched inference budgets.

In contrast, Table 7 uses greedy decoding (T=0). Under k=1, sampling temperature is the only difference between these two experiments. These results therefore reflect different sampling regimes and need not yield the same accuracy gaps. We will clarify this distinction and the interpolation procedure in the manuscript.

The table above provides the tabulated pass@K values and corresponding inference times underlying Figure 5, measured on a single 80GB NVIDIA A100 GPU. Here, (K) denotes the number of sampled solutions, whereas (k) in Table 7 denotes the number of segments committed per iteration.

W2. The main modeling contribution is validated only at a very small scale, and its headline margins are not tabulated. LatentMDM is tested only at 125M on TinyGSM.

Further, we extended the TinyGSM experiments to 537M-parameter models. Specifically, we trained LatentMDM with capacity for 64 segments of up to 32 tokens each, alongside same-size baselines, for 500k optimizer steps. Under greedy decoding, LatentMDM achieves 56.48% accuracy, compared with 57.54% for ARM and 33.97% for MDM, while providing 5× higher throughput and lower peak allocated memory(0.6-0.8x) than ARM. The additional table reports pass@K and corresponding inference times for the new 537M models, measured on a single 180GB NVIDIA GB200 GPU. While TinyGSM remains a narrow evaluation setting, these results provide encouraging evidence that LatentMDM’s accuracy–efficiency benefits extend to a larger model scale.

Accuracy (%)Latency (s/question)
SizeKLatentMDM-confLatentMDM-L2RLatentMDM-randomMDM-confARMLatentMDM-confLatentMDM-L2RLatentMDM-randomMDM-confARM
537M149.2637.7036.4936.5752.171.100.920.9416.335.18
537M258.6949.4048.8351.0563.502.201.841.8732.6610.35
537M363.3655.4155.3458.6968.983.302.762.8148.9915.53
537M466.3959.3159.5263.4472.394.403.683.7565.3220.70
537M568.6262.1762.5266.7474.805.504.604.6981.6425.88
537M670.3964.4464.8469.2276.596.605.525.6297.9731.05
537M771.8766.3366.7571.1877.997.716.446.56114.3036.23
537M873.1467.9568.3772.8079.108.817.367.50130.6341.40
537M974.2769.3869.7874.1780.009.918.298.43146.9646.58
537M1075.2870.6671.0475.3680.7411.019.219.37163.2951.75

Q3. Does the positional-uncertainty diagnosis hold outside Python code and at other scales? A mass-vs-LOC plot on a non-code task (or a much smaller/larger MDM) would support the claim that this is a property of fixed-canvas token-level MDMs in general, rather than of code targets with rigid syntactic scaffolds.

To test this, we repeat the same positional uncertainty analysis on LLaDA 2.0 16B, a bigger model, and Humaneval (Python code) and GSM8k (math). As shown in https://ibb.co/hxk7LNNf, we observe a similar trend. These results suggest that the positional uncertainty phenomenon is a rather general issue of fixed-canvas masked diffusion rather than an observation confined to our evaluated setup in the manuscript.

Q2. How do you interpret FlexMDM's pass@1 regression on HumanEval/HumanEval+ (−8.00/−7.28)? W1. The downstream evidence does not robustly support the abstract's claim that the two approaches "improve downstream performance.

Prompt format on HumanEval. Re-examining this gap, we found a difference in our HumanEval setup that was previously overlooked. FlexMDM receives the standard HumanEval prompt (function signature and docstring), the same prompt Dream-Coder's own evaluation uses for its base model. For Dream-Coder, our paper instead used the EvalPlus instruction wrapper, because our any-order metrics are computed on the whole generated function, and with the standard prompt Dream-Coder usually writes only the function body. Table 5 reused this setup. (On MBPP, both models receive Dream-Coder's official MBPP prompt.) To isolate the effect of the prompt, we ran the two missing combinations (164 problems × 16 samples; † marks the pairing in Table 5):

HumanEval:

Model, promptPass@1Pass@2Pass@4Pass@8Pass@16
FlexMDM, standard †50.6566.6078.6986.8692.07
FlexMDM, wrapper49.1663.5375.4984.7490.85
Dream-Coder-7B, standard39.0255.7270.5781.6089.63
Dream-Coder-7B, wrapper †58.6572.2281.3486.9490.85

HumanEval+:

Model, promptPass@1Pass@2Pass@4Pass@8Pass@16
FlexMDM, standard †46.6161.8973.8382.0787.80
FlexMDM, wrapper45.5459.1870.8980.4386.59
Dream-Coder-7B, standard34.6850.4064.8775.1281.10
Dream-Coder-7B, wrapper †53.8967.0575.9681.5385.37

† rows: Table 5 as published. The other two rows are new runs with Table 5's settings and seeds, with outputs cut at the first end-of-sequence token, as in Dream-Coder's official evaluation.

With the standard prompt for both models, FlexMDM outperforms Dream-Coder at every k, by 11.62 points at Pass@1 on HumanEval and 11.93 on HumanEval+ (95% CIs [7.96, 15.28] and [8.16, 15.62]). The wrapper raises Dream-Coder's Pass@1 by 19.63 points but changes FlexMDM's by only −1.49 [−3.93, +0.91]. Table 5 thus paired each model with the prompt that suits it better, so its 8.00-point gap is a best-versus-best comparison, not an artifact; but which model leads on HumanEval depends on the baseline's prompt, to which Dream-Coder is highly sensitive and FlexMDM is not. We will state the prompt formats in Appendix D.4 and report this experiment in the appendix.

Does insertion hurt locally-determined code, and does any insertion temperature in the Table 6 sweep recover it on HumanEval, or is the knob only useful on MBPP-style prompts?

Reducing the insertion temperature provides a small Pass@1 improvement on HumanEval, but does not recover the gap to Dream-Coder.

BenchmarkTinsT_{\mathrm{ins}}Pass@1Pass@2Pass@4Pass@8Pass@16
HumanEvalDream-Coder-7B-Base58.6572.2281.3486.9490.85
1.0 (default)50.6566.6078.6986.8692.07
0.7051.6467.0277.9085.4991.46
HumanEval+Dream-Coder-7B-Base53.8967.0575.9681.5385.37
1.0 (default)46.6161.8973.8382.0787.80
0.7047.4562.2073.3181.5187.80

Sharpening to Tins=0.7T_{\mathrm{ins}}=0.7 raises Pass@1 by 0.99 (HumanEval) and 0.84 (HumanEval+) points, about one-eighth of the gap, and lowers Pass@4 and Pass@8. On MBPP/MBPP+, by contrast, Tins=0.6T_{\mathrm{ins}}=0.6 raises Pass@1 by 2.48/2.12 points and improves every k (Table 6).

So in our sweep, the knob is useful on MBPP-style prompts but not on HumanEval. This fits what the knob does: it sharpens where new slots are inserted without changing how many are inserted. MBPP prompts give only a description and tests, so the model must build the program's structure itself, and noisy early placements can derail it; sharpening them helps at every k.

HumanEval prompts already fix the structure (signature and docstring), so placement noise mostly varies otherwise similar programs, which adds diversity at large k and costs about a point at k = 1. Together with the prompt experiment above: across the three FlexMDM settings we evaluated on HumanEval (either prompt at Tins=1.0T_{\mathrm{ins}}=1.0, or Tins=0.7T_{\mathrm{ins}}=0.7 with the standard prompt), FlexMDM's Pass@1 stays between 49.16 and 51.64, while Dream-Coder's moves by 19.63 points with its prompt alone.

In short. On HumanEval, Table 5 gives Dream-Coder the more favorable of two prompt formats. FlexMDM receives the standard prompt (signature and docstring), which is used in Dream-Coder's own evaluation. But we also tried giving Dream-Coder that prompt inside an instruction wrapper that explicitly asks for a complete solution. This wrapper raises Dream-Coder's Pass@1 by about 20 points, while it does not help FlexMDM (−1.5 points). With the standard prompt for both models, FlexMDM leads Dream-Coder by 11.6 Pass@1 points on HumanEval. In prompt format, Table 5 thus favors Dream-Coder. Sharpening insertion helps on MBPP but adds only about one Pass@1 point on HumanEval and lowers Pass@4 and Pass@8, likely because HumanEval's signature and docstring already fix the program's structure. Across all FlexMDM settings we tried, its HumanEval Pass@1 stays between 49.2 and 51.6, while Dream-Coder's moves by about 20 points with the prompt alone. FlexMDM's HumanEval performance is therefore robust to the decoding settings, and the direction of the gap depends strongly on how Dream-Coder is prompted.

Reviewer_11bTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the gap between an any-order prediction interface and the generation behavior actually induced by masked diffusion models. Through structural comparisons of generated Python programs, the authors show that confidence-based decoding often produces outputs resembling those obtained with left-to-right decoding. They attribute this phenomenon to positional uncertainty and investigate two remedies: an extension of insertion-based FlexMDM and a new LatentMDM that performs masked prediction over variable-length segment representations with local autoregressive decoding. Experiments evaluate a 7B FlexMDM on code-generation benchmarks and a 125M LatentMDM on GSM8K using Python solutions learned from TinyGSM.

Critical strengths

The distinction between interface flexibility and realized inference behavior is meaningful. The paper goes beyond measuring token-order deviations and examines program structure and generation trajectories. The AST comparisons and the CBC/RUB/OBW metrics provide complementary ways to investigate whether decoding actually moves across unfinished program regions.

The LatentMDM policy ablation provides direct evidence that segment selection matters. In Table 7, with the same trained model and greedy token decoding, average-log-likelihood selection achieves 45.5%, compared with 41.3% for left-to-right selection and 40.3% for random selection when one segment is committed per iteration. This supports a concrete, appropriately scoped finding: adaptive segment selection can improve performance on the evaluated program-based reasoning task.

Critical weaknesses

1. The evidence establishes a useful change in granularity more clearly than a general resolution of the proposed inference bottleneck.

LatentMDM reduces token-level placement ambiguity by moving prediction to fixed segment slots, but it retains confidence-based selection and positional commitments at the segment level.

It can therefore still favor segments adjacent to revealed content, or segments whose form is already predictable from the prompt. Bidirectional attention can exploit a revealed suffix, but when the suffix remains unknown, it does not supply the missing semantic information needed to generate that suffix correctly.

I do not expect a successful model to avoid sequential generation when the task requires it.

The concern is that the paper does not sufficiently distinguish useful nonlocal semantic decisions from easier effects such as selecting predictable scaffolding, empty segments, or locally constrained continuations. The policy ablations demonstrate that adaptive ordering helps, but do not by themselves establish that the gains arise from resolving the positional uncertainty diagnosed in Section 3.2.

This distinction matters because both experimental branches ultimately generate Python code, including the TinyGSM/GSM8K setup. There is no direct evaluation of non-code reasoning where useful future segments and their boundaries are less predictable. The authors acknowledge this limitation, but the demonstrated scope remains substantially narrower than the broader framing around genuine any-order reasoning.

The positional-uncertainty diagnostic also needs stronger controls. Aggregate mass measures expected token count under the modeled marginals, so repeated tokens can have high mass without corresponding to an identifiable semantic component whose location is uncertain. Moreover, the definition implies

LOC⁡(v;z)≤min⁡{1,1/m(v;z)},\operatorname{LOC}(v;z)\leq \min\{1,1/m(v;z)\},

because each positional probability is at most one.

Thus, part of the inverse relationship in the mass–localization plots follows algebraically. These plots are suggestive, but do not independently establish positional uncertainty as the principal cause of the observed collapse. Likewise, similarity between final ASTs does not directly identify the reasoning process that produced them.

2. The incremental contribution of the FlexMDM branch is limited and insufficiently isolated, leaving the overall novelty and significance unconvincing.

The central insertion-and-unmasking mechanism, its any-order capability, and adaptation of pretrained models at 8B scale were already established by Kim et al. (2025), Any-Order Flexible Length Masked Diffusion. The present paper adds time reparameterization, count-preserving insertion temperature, log-space insertion prediction, and broader Python fine-tuning. These are useful extensions, but the underlying mechanism for relaxing the fixed canvas is inherited.

The insertion-temperature ablation is informative. However, I could not find a controlled comparison against the original FlexMDM formulation on the same backbone and training data that isolates the effects of the other modifications. A standard MDM continued-fine-tuning baseline on the same corpus would also help distinguish the effect of the proposed mechanism from additional training.

The downstream improvements are mixed. On HumanEval, FlexMDM increases Pass@16 from 90.85 to 92.07, while decreasing Pass@1 from 58.65 to 50.65. On MBPP, Pass@1 increases only from 64.53 to 64.70, while Pass@16 decreases from 92.86 to 91.27. Different sampling temperatures and length constraints further complicate attribution. These results support a quality–exploration trade-off, but provide limited evidence of a substantial practical advance.

Using an existing method as an experimental vehicle is legitimate. Nevertheless, this means that the paper’s main novelty must come from its mechanistic diagnosis and LatentMDM. For the latter, comparisons with matched block- or line-level approaches are needed to separate the value of learned semantic latents from coarser prediction units, local autoregressive decoding, and whole-segment confidence scoring. Taken together, the current evidence does not sufficiently establish the significance of the combined contribution.

Questions for authors

What evidence distinguishes useful nonlocal segment decisions from selecting predictable scaffolding or empty segments? Can the existing generation traces be analyzed by distance from revealed content, excluding empty/EOS-only segments and separating scaffolding from substantive computations?

Can the positional-uncertainty diagnosis be supported by controls for token frequency and repeated occurrences, or by a controlled manipulation of placement ambiguity? How should the mass–localization plots be interpreted beyond the inverse relationship already implied by their definitions?

Which controlled results isolate the new FlexMDM components from the original formulation and additional fine-tuning? For LatentMDM, is there a matched block- or line-level baseline that helps separate semantic compression from changes in prediction granularity and candidate scoring? These comparisons would materially affect my assessment of novelty and attribution.

Equation (1) appears to place a negative coefficient before the insertion loss while defining (\phi(d,g)=e^g-dg), whereas Appendix D.2 uses a positive coefficient. Which objective was implemented, and is the discrepancy a typographical error?

6 comments

We thank the reviewer for highlighting our contributions. Below, we report the results of the additional experiments suggested by the reviewer.

(On latentMDM) What evidence distinguishes useful nonlocal segment decisions from selecting predictable scaffolding or empty segments? Can the existing generation traces be analyzed by distance from revealed content, excluding empty/EOS-only segments and separating scaffolding from substantive computations?

For LatentMDM, is there a matched block- or line-level baseline that helps separate semantic compression from changes in prediction granularity and candidate scoring? These comparisons would materially affect my assessment of novelty and attribution.

To examine the effect of segmentation, we trained fixed-block variants of LatentMDM that replace newline-based segments with blocks of 8, 16, or 32 tokens. The variants retain the same segment scoring rule and local autoregressive decoding procedure.

ModelSegmentation configurationAccuracy
LatentMDM (125M)newline-based46.17%
LatentMDM (125M)Fixed-block (32)29.03%
LatentMDM (125M)Fixed-block (16)32.45%
LatentMDM (125M)Fixed-block (8)37.45%
ARM (125M)—52.46%
MDM (125M)—26.31%
LatentMDM (537M)newline-based56.33%
ARM (537M)—57.54%
MDM (537M)—33.97%

Newline-based segmentation, the default configuration in our paper, outperforms the tested fixed-block variants by 8.72–17.14%. These results support the benefit of aligning latent prediction units with line boundaries on TinyGSM.

We next examined generation order after excluding empty segments and EOS tokens and analyzing predictable scaffolding, e.g., function headers, docstrings that repeat the question, and imports, separately. We measure the normalized Kendall tau distance D from L2R order, where D = 0 denotes L2R generation and D = 1 denotes the reverse order. All measurements use the full GSM8K test set of 1,319 examples, a sampling temperature of zero, and confidence-based decoding. LatentMDM variants select segments, whereas token-level MDMs select individual tokens.

LatentMDM(125M)LatentMDM(537M)fix8(125M)fix16(125M)fix32(125M)MDM(125M)MDM(537M)
0.37040.36890.09670.04500.02170.04760.1285

Newline-based LatentMDM departs substantially more from L2R generation at both model scales than the fixed-block variants or token-level MDMs.

Interestingly, replacing newline-based segmentation with fixed-size blocks substantially reduces both accuracy and departure from L2R generation, despite retaining the same segment scoring rule and local autoregressive decoder. This effectively supports our design intuition that meaningful segment boundaries help realize the model’s any-order capability.

Next, to investigate how nonlocal commitments may help subsequent generation, we also inspected individual decoding trajectories. Our intuition is that a committed future segment can serve as an anchor, helping the model determine the intermediate computations that connect the revealed left context to that segment. This may enable the model to reason backward from the anchor or develop multiple intermediate steps in parallel.

GSM8K examples #70 (https://ibb.co/Z6g9cStZ) illustrates this behavior. After revealing only the function header, confidence-based LatentMDM selects slot 5. Committing this segment changes the candidate contents of the remaining slots; these updated candidates then remain unchanged throughout subsequent decoding. This behavior is consistent with the anchor helping organize the intervening computations. We further provide 4 additional GSM8K traces: #49 (https://ibb.co/bRGzq3y5) , #155 (https://ibb.co/RTxrKQsr) , #903 (https://ibb.co/5gq64f4f) , #1189 (https://ibb.co/n8B9dfdt) .

The examples illustrate how a nonlocal commitment can inform earlier, unresolved steps. Together with the segmentation ablation, it supports the usefulness of making such commitments accessible through segment-level prediction.

There is no direct evaluation of non-code reasoning where useful future segments and their boundaries are less predictable. The authors acknowledge this limitation, but the demonstrated scope remains substantially narrower than the broader framing around genuine any-order reasoning.

We additionally trained 537M-parameter models for 500k optimizer steps. At this scale, LatentMDM achieves 56.33% accuracy, compared with 57.54% for ARM and 33.97% for MDM. These results extend our evaluation to a larger model scale, although the experiments remain limited to program-based reasoning on TinyGSM. Training a general-purpose language model from scratch would require substantially more compute and data; within the resources available for the rebuttal, we therefore prioritized model scaling and segmentation controls on TinyGSM.

Can the positional-uncertainty diagnosis be supported by controls for token frequency and repeated occurrences, or by a controlled manipulation of placement ambiguity? How should the mass–localization plots be interpreted beyond the inverse relationship already implied by their definitions?

We thank the reviewer for pointing out these two confounds. We agree that aggregate mass measures expected token count and that the definition implies (\mathrm{LOC}\leq\min(1,1/m)). The inverse relationship between mass and localization alone is therefore insufficient evidence of positional uncertainty.

To examine the phenomenon beyond this constraint, we repeated the analysis separately for tokens appearing exactly once or twice in the ground-truth masked region. In particular, we examined restricted mass intervals around these counts: (m\in[0.5,1.5]) for singleton tokens and (m\in[1.5,2.5]) for doubleton tokens. The plot (https://ibb.co/0pdbbzP1) reports results for Dream-Coder and DiffuCoder on HumanEval and MBPP.

Within these groups, we still observe low-localization candidates (green) at comparable mass, while committed tokens (red) tend to concentrate a larger fraction of their mass at the selected position. In particular, singleton tokens exhibit low localization even around (m=1), where the mathematical upper bound permits localization as high as one. Thus, low localization in this region is not forced by the upper bound.

These results support a more precise interpretation of our diagnostic: even after controlling ground-truth occurrence count and restricting predicted mass, a token’s probability can remain dispersed across positions. The observed selection pattern is consistent with confidence-based decoding favoring candidates whose probability is more concentrated at a particular position.

Which controlled results isolate the new FlexMDM components from the original formulation and additional fine-tuning?

Controlled results for the FlexMDM components. We agree that the insertion-and-unmasking mechanism and its training objective are those of Kim et al. [2025a], who adapted a pretrained model to specific tasks: they train LoRA adapters on LLaDA-8B and instruction-tune it separately for each evaluation, GSM8K and single-line HumanEval infilling. What we add is a fully fine-tuned model for general-purpose code generation. We argue that the insertion temperature and the log-space prediction address problems of this setting, and we isolate each below.

Count-preserving insertion temperature. This modification acts only at inference, so Table 6 isolates it on the same trained model, as the reviewer notes. With the default placement, FlexMDM trails Dream-Coder on MBPP at every k. Sharpening where masks are inserted, while keeping how many are inserted, raises MBPP Pass@1 at every temperature in the sweep, from 62.22 to 64.70 at T_ins = 0.6 (Dream-Coder: 64.53), and likewise on MBPP+. Preserving the count matters: rescaling the insertion rates directly would change the expected length, and making the count deterministic collapses the generated length.

Log-space insertion prediction. For this modification, the submission has no controlled comparison against the original formulation, as the reviewer notes. We have now run one, on the same backbone and training data.

Setup. We perform two training runs from Dream-Coder 7B that differ only in how the insertion head outputs the expected count λ: in log space, as in our paper (λ = exp g), or directly, as in Kim et al. [2025a] (λ = softplus g). Everything else is the same and follows the recipe in our paper, including its corpus, batch size 576, learning rates of 1e-5 for the backbone and 2e-5 for the insertion head, bf16 parameters and optimizer states, and the loss λ − x log λ, where x is the true count. The runs also share the seed, the data order, the noise levels drawn at each step, and the initialization, except for the head's output bias, which is set so that both start from the same predicted count. Any difference between the runs therefore comes from the parametrization, not from the fine-tuning. Each run trains for 1,000 steps, with a constant learning rate after a 100-step warmup. We evaluate both runs on 1,362 sequences held out from the training corpus, each corrupted at 25 noise levels in exactly the same way for both.

Results. The two parametrizations differ at the start of generation, where the counts to be predicted are largest. Results at step 1,000 on the held-out data, by diffusion time t:

ttokens insertedlargest gap (99th pct.)predicted / true count: logpredicted / true count: softpluslargest prediction: loglargest prediction: softplusunmasking CE (nats/token): logunmasking CE: softplus
< 0.003< 0.5%4221.060.31305375.376.30
0.003–0.0120.5–2%2811.090.67237375.115.67
0.012–0.042–7%1310.991.03112364.804.89
0.04–0.17–16%710.971.0135254.424.48
0.1–0.316–45%341.021.0221153.253.27
≥ 0.3> 45%91.041.04980.910.91

Predicted / true count: median over held-out sequences of the total predicted insertion count divided by the true number of missing tokens.

Once about 2% of the program's tokens have been inserted, the two heads are equally well calibrated. Before that, a single gap can hold hundreds of tokens, and the softplus head does not reach such counts. Its largest prediction is 37, against 305 for the log head. At t < 0.003 it predicts 29% of the missing tokens in total, against 98% for the log head. The limit is numerical. To output λ in the hundreds, a softplus head needs pre-activations in the hundreds, while a log head needs only log λ ≈ 6. The softplus head's last-layer weights therefore grow until they reach 2⁻⁷, the bf16 magnitude above which an update of the head's learning-rate size is rounded away. At step 1,000, 8% of them sit at that value, and its largest prediction grew only from 29 to 37 between steps 500 and 1,000. The log head reaches its counts through the exponential, with all last-layer weights below 0.0045.

The softplus run's token predictions are also worse in these early states (+0.6 to +0.9 nats per token), even though the parametrization changes only the insertion head. We suspect that the insertion loss, which the softplus head cannot reduce there, pulls on the shared network.

This is a result about our regime: bf16 fine-tuning at the learning rates above. We have not tested fp32 master weights.

The additional fine-tuning. We indeed did not separate insertion itself from the additional training. However, that training is part of the method rather than an extra: Dream-Coder has no insertion head and was never trained to insert tokens, so this stage plays the role of FlexMDM's pretraining, much as Dream-Coder was itself adapted from the autoregressive Qwen2.5-Coder. The MDM continued-fine-tuning baseline the reviewer describes would separate the two. We have not run it, since it needs about 1,150 H100-hours, as much as our own fine-tuning.

Revision. We will add the log-space comparison to Appendix D.1, with the table above, the calibration over training, and its limits: 1,000 training steps and no comparison of generated programs. In Section 4 ("Advances over prior work") and Appendix D.1 we will describe the log-space prediction by what this comparison shows: under our bf16 fine-tuning, predicting in log space lets the insertion head reach the large counts needed at the start of generation, which the softplus head of Kim et al., trained the same way, does not reach. We will also list the MDM continued-fine-tuning baseline as future work and present FlexMDM's downstream results as a benchmark-dependent trade-off rather than a consistent improvement (see our response to Reviewer tAfV).

Equation (1) appears to place a negative coefficient before the insertion loss while defining (\phi(d,g)=e^g-dg), whereas Appendix D.2 uses a positive coefficient. Which objective was implemented, and is the discrepancy a typographical error?

We thank the reviewer for catching this. It is a typo: Eq. (1) should not have a minus sign before the insertion loss. Our implementation adds the insertion loss with a positive weight, as stated in Appendix D.2, and this is the objective used for all reported FlexMDM experiments. The insertion head predicts the log expected insertion count. We will correct Eq. (1); this correction does not change the implementation or the reported results.

Reviewer_tAfVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that masked diffusion models (MDMs) offer an any-order prediction interface but, under confidence-based decoding, collapse to near left-to-right (L2R) inference, and attributes this to positional uncertainty: on a fixed token canvas, the model may know what to generate but not where to place it. The diagnosis is supported by AST-based similarity between confidence-decoded and L2R-decoded programs and by a mass–localization analysis of token posteriors. Two remedies are proposed: FlexMDM, an insertion-based MDM obtained by fine-tuning Dream-Coder 7B for Python, evaluated with tree-based any-order metrics and pass@k on HumanEval/MBPP; and LatentMDM, a 125M masked diffusion model over line-level segment latents with an autoregressive decoder, trained on TinyGSM and evaluated on GSM8K against AR, MDM, DUO and S-FLM baselines.

Critical strengths

Prior work on causal collapse in MDMs mainly reports that confidence-based decoding reduces diversity or yields AR-like token orders. This paper goes one step further and offers a concrete mechanism, positional uncertainty, with a simple diagnostic (aggregate mass m(v;xt)m(v;x_t) versus localization LOC(v;xt)\mathrm{LOC}(v;x_t)) and a structural measurement at the program level rather than the token-order level. The diagnosis directly motivates two remedies that relax the two assumptions behind the bottleneck, fixed canvas and token-level granularity, and both produce measurably different generation behavior: FlexMDM revisits roughly half of sibling branches on HumanEval versus almost none for Dream-Coder, and LatentMDM commits high-mass, low-localization tokens that standard MDMs leave undecoded.

Critical weaknesses

The evidence that FlexMDM's any-order behavior improves code generation is weak and confounded. FlexMDM is clearly worse at low kk (HumanEval pass@1 50.65 vs. 58.65, HumanEval+ 46.61 vs. 53.89), trails slightly on MBPP at k≥2k \ge 2, its clearest gains are at pass@16 on HumanEval/HumanEval+ (about 1.2–2.4 points). At the same time, the comparison changes several factors at once: FlexMDM is further fine-tuned on about 2.6M Python sequences, uses token temperature 0.1 versus 1.0 for Dream-Coder, and uses an insertion temperature chosen per benchmark (0.6 on MBPP, 1.0 on HumanEval). The pass@16 gain may therefore come from the additional fine-tuning data or the decoding configuration rather than from insertion. A control that fine-tunes Dream-Coder on the same corpus with the standard MDM loss and evaluates it under a matched sampling protocol is necessary to attribute the gains to insertion-based any-order inference.

LatentMDM is presented as the main modeling contribution, but it is validated only in a narrow setting: a 125M model trained on TinyGSM, evaluated on GSM8K, with segments defined as lines, at most 16 segments of at most 32 tokens during training and 8 segments at inference. These solutions are short, highly templated programs, so it remains unclear whether segment-level order search scales to longer and less regular code, where the segmentation choice and segment count matter much more. In addition, the advantage over AR and other baselines is reported at matched wall-clock time with batch size 1, a regime that favors LatentMDM's parallel segment decoding. The paper should report accuracy at matched steps or FLOPs and under larger batch sizes.

Questions for authors

The link between positional uncertainty and causal collapse is shown only correlationally (Fig. 3). A direct test is missing: if confidence scores are corrected for positional dispersion on the same fixed canvas (e.g., using m(v;xt)m(v;x_t) or LOC(v;xt)\mathrm{LOC}(v;x_t) in the ranking, similar in spirit to the cross-position reweighting of Patel et al. (2025a) discussed in the appendix), does collapse disappear?

The structural similarity analysis lacks an L2R self-similarity baseline. Each confidence-decoded program is matched against its best of 32 L2R samples, but the paper does not report how similar an independent L2R sample is to the remaining 31. Without this reference, it is hard to judge whether the confidence-vs-L2R scores indicate near-identical structure or simply the ordinary similarity level among valid solutions to the same short HumanEval/MBPP problem.

The tree-based any-order metrics (CBC, RUB, OBW) measure how generation moves across the AST, and insertion mechanically produces non-contiguous drafts. Higher values therefore do not by themselves show useful any-order reasoning. Is there any per-sample relation between these metrics and correctness?

Lines are used as segments. What motivates this choice over syntactic units (e.g., statements or AST subtrees), which would align better with the structural any-order claim? How many TinyGSM examples are truncated by the 16-segment or 32-token limits, and how does performance change with different segment granularities?

6 comments

We thank the reviewer for highlighting our contributions and thoughtful comments. Below, we provide the results of the additional experiments that the reviewer suggested.

W1. FlexMDM claim scope

At the same time, the comparison changes several factors at once, …. The pass@16 gain may therefore come from the additional fine-tuning data or the decoding configuration rather than from insertion.

Confounds in the downstream comparison. We thank the reviewer and agree that Table 5 varies several factors at once, so it cannot by itself attribute FlexMDM's downstream differences to insertion. The extra fine-tuning, however, is a necessary part of the method, not an optional extra: Dream-Coder was never trained to insert tokens and has no insertion head, so the 2.6M sequences serve as FlexMDM's pretraining, much as Dream-Coder was itself adapted from the autoregressive Qwen2.5-Coder.

The control the reviewer describes, Dream-Coder fine-tuned on the same corpus with the standard MDM loss under a matched sampling protocol, would isolate insertion. We have not run it, since it needs about 1,150 H100-hours of fine-tuning, but below we clarify how the sampling settings affect the comparison.

FlexMDM is further fine-tuned on about 2.6M Python sequences, uses token temperature 0.1 versus 1.0 for Dream-Coder, and uses an insertion temperature chosen per benchmark (0.6 on MBPP, 1.0 on HumanEval).

Dream-Coder's temperature. Temperature 1.0 may look like a handicap for the baseline, but its effect cuts both ways. Dream-Coder's official evaluation uses temperature 0.2 on HumanEval and 0.1 on MBPP; the MBPP value equals FlexMDM's token temperature. At these temperatures, Dream-Coder's sampler is nearly deterministic. In 16-sample runs, its samples were byte-identical on 155 of 164 HumanEval problems and 374 of 378 MBPP problems, and no problem had both passing and failing samples. Its Pass@16 therefore equals its Pass@1:

Model, token temperatureHumanEval Pass@1HumanEval Pass@16MBPP Pass@1MBPP Pass@16
Dream-Coder, 0.2 / 0.1 (official setting) ‡63.4163.4166.4066.40
Dream-Coder, 1.0 (Table 5)58.6590.8564.5392.86
FlexMDM, 0.1 (Table 5)50.6592.0764.7091.27

‡ 16 samples per problem, using Table 5's prompts and FlexMDM's length cap and grading. Table 5's Dream-Coder rows instead allow 512 new tokens, as Dream-Coder's own evaluation does. With that budget, a single-sample MBPP run at temperature 0.1 gives a Pass@1 of at least 74.9.

We used temperature 1.0 for Dream-Coder's pass@k rows to obtain diverse samples (Appendix D.3). This costs Dream-Coder at Pass@1, but favors Dream-Coder at large k, where FlexMDM's small HumanEval advantage appears. In addition, on HumanEval, Table 5 already gives Dream-Coder the more favorable of the two prompt formats we tested. That prompt raises its Pass@1 by 19.6 points at temperature 1.0 (see our response to Reviewer FAzX). FlexMDM's token temperature is low because its insertion steps are themselves random (Appendix D.3). Its samples are diverse even at 0.1: 142 of 164 HumanEval problems have both passing and failing samples. Each model is thus sampled with enough diversity for pass@k.

Revised claims. We will scope FlexMDM's claims to what we evaluated:

  • In the abstract and Section 4, we will present FlexMDM's contribution as the any-order decoding it exhibits (Table 4, measured with Dream-Coder at its official temperatures). We will describe its downstream results on the four benchmarks as a benchmark-dependent trade-off between Pass@1 and larger-k performance, rather than a consistent improvement.
  • We will add a limitations paragraph on the scope of these experiments (one 7B base model, function-level Python benchmarks) and list the same-data MDM fine-tuning control as future work.
  • We will state both models' sampling settings and their rationale next to Table 5, and report the low-temperature Dream-Coder rows in the appendix.

W2. LatentMDM’s evaluation scope

LatentMDM is presented as the main modeling contribution, but it is validated only in a narrow setting.

We agree that TinyGSM remains a narrow evaluation setting. However, pretraining a general-purpose language model from scratch would require substantially more compute and data than our current experimental setting. Within the resources available for the rebuttal, we therefore focused on scaling the model and testing alternative segmentation configurations on TinyGSM. In particular, we trained a 537M LatentMDM with capacity for 64 segments of maximum 32 tokens each, alongside same-size baselines, for 500k optimizer steps

In addition, the advantage over AR and other baselines is reported at matched wall-clock time with batch size 1, a regime that favors LatentMDM's parallel segment decoding.

To address this, on the same setup, we extended the comparison to the 537M-scale and also measured accuracy, throughput, and peak allocated GPU memory at batch sizes 1, 4, 16, and 64.

ModelAccuracybs=1bs=4bs=16bs=64
LatentMDM(125M)46.17%1.76 / 0.655.99 / 0.6919.10 / 0.8654.58 / 1.86
ARM(125M)52.46%0.40 / 0.741.22 / 0.833.66 / 1.3812.65 / 3.37
MDM(125M)26.31%0.12 / 1.150.41 / 2.461.06 / 7.691.48 / 28.64
LatentMDM(537M)56.33%1.00 / 2.543.15 / 2.6110.46 / 2.8731.72 / 3.99
ARM(537M)57.54%0.19 / 3.120.62 / 3.431.96 / 4.666.91 / 9.59
MDM(537M)33.97%0.06 / 3.450.21 / 4.770.61 / 10.040.72 / 31.10

All measurements were obtained on a single NVIDIA GB200 GPU; each batch-size cell reports throughput (questions/s) / peak allocated memory (GiB). The accuracies are measured at a temperature of 0.0.

The scaled-up 537M LatentMDM achieves 56.48% accuracy, compared with 57.54% for ARM and 33.97% for MDM, while providing higher throughput and lower peak allocated memory than ARM in the reported efficiency experiments below. The relative memory advantage even grows with batch size. At 125M parameters, LatentMDM uses 87.8% of ARM’s peak allocated memory at batch size 1, decreasing to 55.2% at batch size 64. At 537M parameters, those are 81.4% and 41.6%. Thus, within the tested range, larger batches preserve a substantial throughput advantage while increasing the relative memory savings.

The paper should report accuracy at matched steps or FLOPs and under larger batch sizes.

Regarding evaluation at matched FLOPs, we note that such comparisons are affected by differences in how the architectures perform inference, including KV caching and segment unmasking in LatentMDM. Likewise, an inference step represents different amounts of work across models. For step counts, Table 7 of our manuscript separately examines committing multiple segments per iteration. We therefore focus on measured throughput and accuracy at matched wall-clock time, which directly characterize performance on the evaluated hardware while accounting for caching, memory access, and parallel execution.

Q1.

The link between positional uncertainty and causal collapse is shown only correlationally (Fig. 3). A direct test is missing: if confidence scores are corrected for positional dispersion on the same fixed canvas (e.g., using or in the ranking, similar in spirit to the cross-position reweighting of Patel et al. (2025a) discussed in the appendix), does collapse disappear?

We tested whether PCG-inspired cross-position reweighting can reduce collapse without changing the canvas Patel et al., 2025a. All experiments use Dream-base on 164 HumanEval tasks, with a 256-token canvas and 256 model calls. We consider three different ways to reweight the confidence: (1) cross-position reweighting changes both token probabilities and position selection; (2) soft L2R bias favors earlier positions; (3) ranking-only changes position selection while keeping the original token-sampling rule and applying no soft L2R bias. For each output, we measure AST similarity using the same protocol.

ConditionAccuracyStructural similarity to L2R
Baseline55.49%0.760
Cross-position reweighting52.44%0.723
Soft L2R bias54.27%0.782
Cross-position reweighting + soft L2R bias53.05%0.721
PCG-inspired ranking-only54.27%0.778

The results are mixed; cross-position reweighting reduces similarity from 0.760 to 0.723. The ranking-only variant yields a similarity of 0.778 and accuracy of 54.27%. These experiments therefore do not establish that confidence correction resolves the observed behavior, but neither do they rule out an inference-based remedy.

One possible explanation is that reweighting confidence does not directly resolve the underlying placement ambiguity. For example, a model may assign substantial aggregate probability to a return token while remaining uncertain whether it belongs inside or outside a loop. Cross-position reweighting can change which token or position is prioritized, but it does not directly supply the missing structural context needed to resolve that choice. The model may therefore change its token-generation order while still producing final programs structurally similar to L2R outputs.

Q2.

The structural similarity analysis lacks an L2R self-similarity baseline

We observed very high structural similarity among independently sampled L2R outputs. For each prompt, we independently sampled 32 L2R outputs. We then treated each output as a query, computed its maximum AST similarity to the remaining 31 outputs, and averaged these best-match scores. This leave-one-out comparison provides the requested L2R self-similarity baseline.

ModelDatasetValid ASTDSuccessful ASTD
Dream-baseHumanEval0.94750.9881
LLaDA-8B-InstructHumanEval0.94550.9531
LLaDA-8B-InstructMBPP0.95490.9639
LLaDA-Instruct-JustGRPO-CodeHumanEval0.99660.9968
LLaDA-Instruct-JustGRPO-CodeMBPP0.99390.9904

As shown in the table, L2R self-similarity is high across all settings. In our manuscript, we will include this similarity result within the L2R baseline to further contextualize that confidence-vs-L2R samples are indeed structurally similar.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback and time. Below, we summarize the main concerns and how we addressed them.

Reviewer tAfV

  • FlexMDM’s mixed performance and confounded comparisons. We added low-temperature Dream-Coder results, clarified the sampling settings, and re-scoped FlexMDM’s downstream findings as a benchmark-dependent trade-off.
  • LatentMDM’s limited scale and batch-size-one evaluation. We extended training to 537M parameters with same-size AR and MDM baselines, demonstrating its efficienc3y at a larger scale. We also added segmentation ablations and quantified training-data exclusions. This ablation shows the role of semantic-based segmentation in the latentMDM performance, further justifying our design choice.
  • Whether the diagnostics establish useful any-order inference. We added the requested L2R self-similarity baseline, tested confidence reweighting on a fixed canvas, and analyzed generation-order metrics conditioned on correctness. The same-model LatentMDM ordering ablations provide the direct evidence that adaptive ordering improves performance.

Reviewer 11bT

  • Whether nonlocal generation reflects substantive computation or predictable scaffolding. Our analyses on the latentMDM's decoding order show departures from L2R generation and illustrate how committing a future segment can inform masked segments.
  • Controls for positional uncertainty and the role of segment boundaries. We added singleton/doubleton analyses within restricted mass ranges. These reveal dispersed positional probability even when the bound does not force low localization. We also retrained fixed-block LatentMDM variants: newline-based segmentation achieves 46.17% accuracy versus 29.03–37.45% for the tested fixed blocks, and this further justifies our rationale behind the LatentMDM design.
  • FlexMDM’s incremental contribution and the insertion-loss discrepancy. We separated the inherited insertion mechanism from our extensions and added a controlled log-space-versus-softplus comparison. This supports improved prediction of large insertion counts under our bf16 training setup, with the 1,000-step scope.

Following the manuscript revision, Reviewer 11bT stated that most concerns had been addressed and raised their rating.

Reviewer FAzX

  • Overstated downstream improvements and the HumanEval Pass@1 regression. We added crossed prompt comparisons and an insertion-temperature ablation. The prompt experiment shows that the relative performance depends strongly on Dream-Coder’s prompt format; the temperature ablation shows that sharpening insertion alone does not recover the original HumanEval gap. Our response therefore scopes the results as benchmark- and sampling-dependent trade-offs.
  • Small-scale validation and insufficiently specified headline margins. We extended LatentMDM to 537M parameters and supplied the pass@K and latency values underlying the comparisons. We clarified the difference between stochastic sampling in Figure 5 and greedy decoding in the policy ablation.
  • Whether positional uncertainty extends beyond the original code models. We added the diagnostic for LLaDA 2.0 16B on both HumanEval and GSM8K math. The similar pattern provides evidence beyond the original model scale and Python setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.