acceptodds
Under review as a conference paper at ICLR 2027

The Parallel-Decoding Score of Masked Diffusion Language Models Is Not a Mutual Information

Abstract

Training-free parallel decoders for masked diffusion language models decide how many positions to commit by bounding the dependence among them. The score they bound, , is read as a conditional mutual information, a reading that presumes the model's any-order conditionals are those of a single joint. We give an exact model-internal identity, : is the mutual information of the model's own one-at-a-time rollout and is a self-marginalization defect, the divergence between the marginal the model reports directly and the marginal its rollout induces. The identity splits any set's parallel error into a total correlation plus a sum of defects, and yields three one-sided certificates of incoherence. With estimators whose bias direction is known, the defect's mean is certified positive over pairs on four models and over scattered sets of two to eight positions on three, and on peaked anchors a conservative estimate puts it at a median 0.37–0.45 of the score. A set's parallel error also depends on the order in which it is revealed, by several times replicate noise. Where the two marginals disagree, the rollout's predicts the documents' own tokens better on average, so a parallel step samples from the less accurate one. Inside EB-Sampler's GSM8K decodes on LLaDA-8B-Instruct, half the decodes overrun the entropy allowance while the dependence it is written against stays within it: on the positions measured, the overrun comes from the defect. All code and the records behind every number are in the supplementary material and will be released at https://github.com/xxx/xxx upon acceptance.

Primary area:
generative models
Keywords:
masked diffusion language models, parallel decoding, mutual information, self-marginalization, kl decomposition, any-order conditionals, diffusion llms

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XUMSOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores the problem of parallel decoding in Masked Diffusion Language Models (MDLMs) through the lens of mutual information. The authors argue that MDLMs are not ideal and can provide different marginal distributions for the same token depending on the order of unmasking due to neural approximations. Based on this idea, the authors name this discrepancy the "self-marginalization defect" and explore methods to quantify it. In practice, the authors estimate this "self-marginalization defect" in several contemporary MDLMs and propose a sampler based on mutual information corrected for the "self-marginalization defect".

Critical strengths
  • The observation that MDLM models are not ideal and can induce different joint distributions on tokens depending on the unmasking order is quite interesting and could lead to some useful practical insight.
Critical weaknesses

The paper is very hard to read from start to finish, and that is the main problem.

  • The sections lack motivation and conclusions.
    • For example, Section 3 is more of a list of technical details rather than a motivated methodology connected to the other sections of the paper.
    • Section 2 is very technically complex and very hard to read. It should be simplified and supplemented with useful commentary.
    • In addition, some of the notation is not introduced correctly. In the contributions section, TCTC, Δπ\Delta_\pi, and HH are mentioned, but no definitions or explanations are given. Also, for example, I^cor\hat{I}_{\rm cor} is defined in Proposition 4, but there is no explanation of what it means, why we need to estimate it, or why the whole proposition is introduced for this purpose.
    • Experimental Sections 4 and 5 are borderline unreadable. The experimental results are mostly presented in text rather than in plots or tables. The references to the appendix are overwhelming. Moreover, the main problem is that there is no clear motivation, meaningful data analysis, or conclusions. These sections are more like lists of facts rather than coherent sections of a paper.
    • The conclusion section also does not help much in drawing useful conclusions from the paper; it is a list of facts as well.

The main problem with the paper is its very poor presentation, which makes it almost unreadable and hard to judge. I believe this paper needs to undergo a major revision.

Questions for authors

See weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer XUMS

Thank you for the frank assessment. We agree that the presentation obscured the contribution, and we have rewritten the paper.

What the paper claims, and the evidence.

Training-free parallel samplers rank positions by a dependence score that is read as a mutual information. For a deployed model it is not: it equals the dependence of the model's own one-at-a-time sampler plus a non-negative self-marginalization defect.

  • Proposition 2. The decomposition is exact; it is proved in Appendix A and checked numerically to 10⁻¹⁵ on enumerable models.
  • Table 1. The defect is certified positive on four models, with an estimator whose expectation is at most the true defect.
  • Proposition 3 and Fig. 2. When the defect is large follows from a bound that one additional forward pass can evaluate.
  • Fig. 4. Measured inside EB-Sampler's own decodes, the defect is 42–56% of the cost of the parallel steps by our point estimate, and the sampler's budget does not account for it.

Scope. The paper does not propose a new sampler. The score-based decoders in Appendix N are ablations: they test whether the score can serve as a decoder on its own, and §5.2 reports that it cannot. The deployed sampler we study is EB-Sampler, unchanged.

The readability changes you asked for.

  • Motivation and conclusion per section. §3 opens with the estimation problem it solves, §4 is organised as four questions, and §4.1–4.4 and §5.1 each end with a boxed takeaway.
  • Section 2 simplified.
    • A worked example (Fig. 1) gives each quantity a concrete value.
    • Each proposition is followed by a plain-language reading.
    • Certificates (A) and (B) are condensed into one paragraph.
  • Notation.
    • TC, Δ_π and H are defined before use, and the contributions use no symbols.
    • Î_cor is introduced in §3 together with its purpose: the same forward passes also estimate the dependence alone, and that is what separates dependence from defect.
  • Experiments. Results are in Table 1 and Figs. 1c–5. Full-resolution tables are in Appendix F, and an appendix guide maps each claim to its evidence.
  • Conclusion. One paragraph of findings, then a limitations paragraph.

Soundness.

  • Appendix A gives analytical proofs of the identities and bounds, and exact enumeration on small models separately checks their implementation.
  • The empirical claims are reported with the controls and uncertainty analyses specified for each experiment.
  • Results that did not go our way are reported as such. For example, the MDLM-OWT fine-tunes did not reduce the measured defect (§5.3).

We would be grateful if you could look at the revision. The general response gives a short reading route.

As the discussion period closes soon, we would be grateful if you could look at the revised PDF, since your assessment rested mainly on presentation. The general response gives a ten-minute reading route (Fig. 1 → §2.1 → Fig. 4 → §5.2), and our reply above summarises each claim and its evidence. We are happy to clarify anything in the remaining time.

Reviewer_XUMSOpenReview ↗

Answer to the Rebuttal

I thank the authors for taking the time to answer my questions and for their thorough rework of the paper.

In particular, I thank the authors for effort taken into the paper revision and the revised version is now much more human readable. I can now see the narrative of the paper, the purpose of sections, the defect definition and the measurement of it. However, as also reviewer JUnb noted:

"The blue text shows that basically the whole paper is re-written, and to me it looks like the framing of results is significantly different and would require a fresh review. I hope this hard work pays off in the next submission."

The entire main body of the paper has been rewritten, which constitutes a major revision and would require a fresh review. In addition, the Appendix remains largely unchanged in the revised version, and the presentation issues I noted about the main text in my initial review still seem to persist there. Revising the Appendix for the next submission would therefore also be beneficial.

I'd like to note that paper has its positive moments:

• I think that studying marginal inconsistency with respect to the generation trajectory and its connection to parallel decoding is an interesting and valuable direction.

• Additionally, I find valuable the observations that imperfect conditionals induce different distributions depending on the unmasking order and, in particular, that this self-marginalization defect actually dominates the parallel-decoding score in important regimes.

But, because the paper had very poor readability prior to the rebuttal, has undergone a major revision, and now needs to be reviewed again in full, I recommend that the authors resubmit this work to a future conference. Therefore I keep my initial score.

Reviewer_JUnbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

In Masked Diffusion Language Models, parallel tokens are decoded independently without taking correlations into account, a potential source of error. Many works have naively estimated this dependence via TC. However, another line of work has pointed out that the distribution induced by different orders of decoding is not necessarily consistent, therefore the TC is not well-defined (which distribution?). This paper aims to correct this discrepancy and accurately measure the decoding defect, taking order inconsistency into account. The paper develops a few tractable ways to estimate the defect, and then the results suggest that the part of the error that has been ignored is relatively large. The paper attempts to interpret the source of the defect, and considers implications for sampling.

Edit post-rebuttal:

I really appreciate the work that authors put into this revision. The abstract and contributions were clearer. I liked the intuitive way of phrasing the defect, and organizing around that:

the shift the model makes when told its own top guess.

It looks like the results are organized with a different flow, which I think may help, though some individual paragraphs still look dense.

The blue text shows that basically the whole paper is re-written, and to me it looks like the framing of results is significantly different and would require a fresh review. I hope this hard work pays off in the next submission.

Second rebuttal:

I see your point - this is mainly presentation improvement - not a change in substance. If you can tune some of the remaining readability issues then I think the paper looks quite reasonable. I will change my score to 3.

Critical strengths
  • Background, related work, setup, and methods seemed thorough and clear to me. There are a lot of very recent and relevant related works - I was pleasantly surprised to see most of the ones I knew about. And the papers showing the conditionals need not be consistent with a single joint were represented. I had not seen a discussion that put these two threads of work together before this ICLR.

  • Notation was relatively intuitive, development of main mathematical results made sense to me.

Overall, this seems like a detailed study of parallel decoding error that for the first time characterizes the order inconsistency effects. For the community studying MDLM models, this should be a useful resource.

Critical weaknesses

Readability issues made it difficult to understand the implications

There are difficult-to-read passages that are dense with internal jargon and unintuitive numbers for humans to read.

Inside EB-Sampler’s GSM8K decodes on LLaDA-8B-Instruct, half the decodes overrun the entropy allowance while the dependence it is written against stays within it: on the positions measured, the overrun comes from the defect.

This is the abstract, and I didn't even know what EB-Sampler was at this point, and the meaning of entropy allowance wasn't very clear to me either.

That measurement, with an estimator that bounds ∆ from below and budget, precision and placebo controls: the defect is certified positive over pairs on four models and over sets on three, is a large share of the score by a conservative estimate, marks the one-pass marginal as the less accurate against the text, and makes a set’s parallel error depend on more than the set (§3–4.1).

Really tough for me to parse. Actually all the contributions were hard to parse. The sentences were too dense with jargon. It's ok to use AI for writing help, but I think here that a more human touch in pulling out what is truly important would be beneficial here.

Other locations where I found the writing impenetrable: 90-91, 195-204, 250-258, 274-275, 370-407, 433-442. This isn't even comprehensive. But in the results section the main issue is the unreadability of numeric result discussions. Here's a very typical example.

"it beats the best baseline costing no more at none of ten operating points and a value-permuting placebo at none of five, with gains up to about 0.13 not excluded".

I defy anybody to try to make sense of this. The numbers belong in visualizations, and the discussion should be more high level.

Ultimately, it was hard to understand the implications of the results for samplers, because I couldn't understand what the results discussions were saying.

Minor comments

I wonder if "DUEL: Exact Likelihood for Masked Diffusion via Deterministic Unmasking" could be relevant? (I only vaguely remember, so I could be wrong.) Your fixed order notation made me think of that one.

at an anchor whose entropy is small against the score, the score is almost entirely defect,

Can this be put more intuitively? Maybe even an example? If H(p_i) is the marginal entropy, then I think this is saying that if the first token is deterministic we get all error from the defect? I think that's what the discussion suggests also, and you point out that confidence based methods prefer this, so maybe extra relevant. I'm just trying to see if I read it correctly. I would have been happier to develop the meaning of these tests with toy examples, and have fewer results. This could be related/same as the Ben-Hamu result, where they say something simple: if all the tokens are deterministic except for one then we have no error.

Questions for authors

See weaknesses - I guess a more intuitive discussion of the main results is what I'm looking for.

2 comments
AuthorsOpenReview ↗

Response to Reviewer JUnb

Thank you for the encouraging assessment and for the line-level pointers.

Abstract and contributions. Both are rewritten without undefined terms, and the intuition is now one sentence: "the defect measures how much the model changes its mind when it is told something it already believed."

  • In Fig. 1, a self-consistent model that is 97% sure the anchor is Paris would predict French with probability 0.97×0.90 + 0.03×0.03 ≈ 0.87 in one pass. It predicts 0.60.
  • The abstract now names EB-Sampler as "an entropy-bounded parallel sampler", and §2 explains it and its error budget before the audit in §5.1 uses them.
  • The abstract sentence you quoted now reads: "Measured along the decodes of EB-Sampler, an entropy-bounded parallel sampler, on GSM8K and HumanEval, the defect is, by our point estimate, about half of the local cost of the parallel steps on LLaDA-8B and over a third on Dream-7B, and the sampler's entropy budget does not account for it."

The passages you listed (submitted line numbers → where they are now):

  • l. 90–91 (remark on absolute continuity) → the opening of Appendix A.
  • l. 195–204 (list of controls) → §3 "Controls", one clause per control; details in Appendices C–E.
  • l. 250–258 (confidence, mask-ratio and filter statistics) → §4.2 with Fig. 2. The unfiltered share is a row of Table 1, and the full numbers are in Appendix F.
  • l. 274–275 (regressions of the defect on confidence, and how often D̂ and Î_cor pick different thirds) → one sentence in §4.2 "Nearby positions"; the full paragraph and the former Table 7 (now Table 10) are in Appendix F.
  • l. 370–407 (GSM8K frontier) → one recommendation in §5.2 ("Never use the score alone"); the figure and numbers are in Appendix N.
  • l. 433–442 (shares read during decoding; the two-point probe) → Appendix F, "The share a decoder can read for itself", with details in Appendix O; §5.2's fourth recommendation states the conclusion. The sentence "it beats the best baseline costing no more at none of ten operating points…" is rewritten in plain terms: used as a filter, the defect does not beat the best baseline at equal cost.

Numbers in visualisations. The results are built around Table 1 and Figs. 2–4, and §4.1–4.4 and §5.1 each end with a boxed takeaway. Full-resolution tables are in Appendix F.

Your reading of certificate (C) is right, and we developed it into Proposition 3 (§2.1).

  • As the anchor becomes certain, the dependence vanishes, and both the score and the defect tend to the shift the model makes when told its own top guess.
  • A shift larger than −log w, where w is the anchor's top probability, cannot be dependence alone. This gives a certificate that costs one additional forward pass, with the anchor filled.
  • At confident anchors the shift predicts the measured defect with R² ≥ 0.95 on three models.
  • We now make your connection to Ben-Hamu et al. explicit. Their entropy bound implies that a step in which every position but one is deterministic has no dependence error. In that regime the dependence term vanishes, but the defect need not, and confidence-ranked samplers operate exactly there.

DUEL is now cited and discussed in §6. It shows an order dependence of exact likelihoods over whole sequences, which complements the within-step order dependence we measure. Corollary 1, the whole-decode version of Proposition 1, is in the same spirit.

Fewer results, more toy examples. The paper now leads with the worked example, and secondary results moved to the appendix.

Thank you for reading the revision and for updating your review.

On whether the revision needs a fresh review. The rewrite is in presentation, which all three reviews asked for. The blue marks every sentence we rewrote, so it overstates how much the content changed.

  • Unchanged. The decomposition, the estimators, the four models, the data and the measurements are those of the submission, with the same numbers:
    • Table 1: conservative median shares 0.370, 0.414, 0.431 and 0.450; certified mean defects 0.026, 0.017, 0.013 and 0.013 nats.
    • The reveal-order spreads (0.260, 0.224 and 0.202).
    • The document-token comparison (+0.015 nats).
    • The original 20-problem in-sampler audit (Appendix K).
  • Same order of argument. Decomposition (§2), then measurement (§3–4), then what it means inside a sampler (§5.1).
  • Three additions, each answering a request in the reviews:
    1. Proposition 3, which develops your own reading of certificate (C).
    2. The in-sampler audit extended from 20 to 120 decodes (4UVS: "a small number of GSM8K problems with one main model").
    3. The objective comparison on the enumerable model (4UVS: "where the defect comes from").

On the paragraphs that are still dense. We agree that the second paragraph of §5.1 and parts of §4.2 still carry too many numbers. For the final version we will move those numbers into a small table next to Fig. 4 and keep only the takeaway in the text. If there are other paragraphs you would like us to simplify, we would be glad to know.

We hope this addresses the concern about the scope of the changes, and we would be grateful if you took it into account in your final assessment.

Reviewer_4UVSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper challenges the common interpretation of parallel-decoding dependence scores as conditional mutual information when a masked language model's conditionals are not mutually coherent. Using the model's one-at-a-time rollout distribution, it decomposes the score into genuine mutual information plus a nonnegative self-marginalization defect, and derives related incoherence certificates and a corrected estimator. Measurements on MDLM, LLaDA and Dream find substantial defects and reveal-order dependence under several controls. The paper also audits these effects inside decoding and tests a probe-based selection rule, but the resulting decoder performs substantially worse than EB-Sampler. The main contribution is therefore a diagnosis and measurement framework rather than a decoding method.

Critical strengths
  1. It exposes a hidden assumption behind a large body of parallel-decoding work. Several schedule-level analyses and dependency-based decoders treat the parallel-step error as a conditional total correlation and the pair score as a mutual information. The paper shows exactly when this reading fails, with a short identity that separates the model's own dependence from a defect term, and gives checkable consequences: one-sided incoherence certificates, conditions under which the defect dominates the score at peaked anchors, and the finding that the parallel error of a fixed set depends on the reveal order, well above replicate noise. This changes how guarantees based on coherent conditionals should be read.

  2. The measurement is careful, statistically honest, and reproducible. The authors handle the upward bias of the plug-in KL estimator with a jackknife and a split-sample estimator that is a lower bound in expectation, calibrate a numerical floor, use placebo and replicate-noise nulls, validate the identities on enumerable models, and are explicit about which numbers are certified, conservative or merely estimated. Unfavourable results are reported rather than hidden. Code and records are released.

Critical weaknesses
  1. Most of the paper is spent analysing experimental data, but the analysis does not lead to a good solution. The paper identifies that existing scores and budgets are mis-specified, and then devotes the bulk of its main text and appendices to measuring how large the problem is (defect shares, order dependence, controls, in-sampler audits). It never shows how to do better. It proposes no sampler, and the decoding experiments are all negative or neutral: the decoder built on the score collapses relative to EB-Sampler, the corrected estimator cannot be distinguished from the original score at set level, and using the defect as a filter on EB-Sampler or Fast-dLLM does not beat the best baseline at equal cost. The most decision-relevant audit inside a deployed sampler also rests on a small number of GSM8K problems with one main model. A reader therefore learns that the score is "not a mutual information", but not whether this matters for any practical sampler, nor what should be done differently.

  2. The theoretical contribution is thin relative to the amount of empirical analysis, and the empirical analysis does not yield a sharper understanding. The core algebra is classical (reverse I-projection and compensation identities, as the authors acknowledge), and the remaining theory consists of short consequences such as one-sided certificates; the rest of the paper is empirical measurement. Yet the measurements stay at the level of "the defect is non-zero and sizeable": they do not explain where the defect comes from (the masking objective versus any imperfect fit), nor provide a predictive account of when it is large, and on the toy model the defect share does not shrink with a much better fit. The headline shares also concern a filtered subset of peaked anchors and are lower without the filter. So the work supplies neither a new principled result nor a method that exploits the observation.

Questions for authors
  1. Could the authors give a more intuitive explanation of when and why the self-marginalization defect becomes large?
  2. How should practitioners use this diagnosis to make better parallel-decoding decisions, beyond interpreting existing dependence scores more cautiously?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4UVS

Thank you for the precise summary. We address both weaknesses and both questions.

W1. Does it matter in practice?

The paper is a diagnosis: the score that training-free samplers use is not the mutual information it is read as. Two practical consequences follow.

(i) The entropy budget does not account for the defect. We audited EB-Sampler's own decodes, now 120 of them:

  • LLaDA-8B on GSM8K: 60 problems (was 20).
  • Dream-7B on GSM8K: 40 problems (was 20).
  • LLaDA-8B on HumanEval: 20 problems (new).

At each step we estimate Proposition 1 on at most the first 8 committed positions in confidence order, sum these local costs along the greedy decode, and compare them with the budget for the full committed sets. These sums are not KL divergences between output distributions.

  • Share. By the jackknife, the defect is 0.55 [0.50, 0.60], 0.42 [0.39, 0.45] and 0.56 [0.53, 0.59] of that cost. The conservative estimator is underpowered at 32 rollouts. Re-measured at 128 rollouts, it gives 0.39 [0.24, 0.50] on LLaDA, which excludes zero. On Dream the jackknife gives 0.35 at 128 rollouts and the conservative estimate is not resolved.
  • Overruns. The cost exceeds the budget on 27 of the 120 decodes. The rate depends on the problems: on LLaDA it was 8 of the first 20 and 6 of the next 40. On all 27, the estimated dependence alone stays within the budget (12 of 27 with the conservative defect).
  • New: accuracy.
    • Overrunning decodes are correct less often: 15/27 against 76/93.
    • On LLaDA GSM8K it is 8/14 against 42/46 (Fisher p = 0.007).
    • The 120 decodes cover 80 distinct problems. Pooled, adjusting for forward passes and setting and clustering by problem, the odds ratio is 0.22 [0.07, 0.75] (p = 0.015).
    • The continuous cost/budget ratio has the same sign but is not resolved (p = 0.24).
    • It is an association, and Dream and HumanEval alone are too small to resolve it.

(ii) The score is the right pair cost, not an MI estimate. By Prop. 2(a), D(i→j) is exactly the cost of committing j together with i.

  • Let G be the change in the log-probability of the document's own token at the target when the anchor's true token is revealed. Ranking an anchor's companions by Î_cor forgoes 6–13% more |G| than ranking by D̂.
  • The entropy-difference form I_ent, the training target of a neural MI estimator, is far worse, and it goes negative.
  • At the level of whole sets, D̂ and Î_cor are not separated, and choosing companions by entropy is at least as cheap (Appendices L–M).

§5.2 turns this into four recommendations, each with its caveats. It also states two negative results plainly: the score alone is not a decoder, and the cheap inference-time corrections we tested do not beat the best baseline at equal cost.

W2. When is it large, and where does it come from?

  • When. New Prop. 3 predicts the defect from one additional forward pass.
    • Let w be the anchor's top probability, Sh the shift when the anchor is filled with that token, and h the binary entropy. Then w·Sh − h(w) ≤ Δ ≤ D, and Δ = 0 forces Sh ≤ −log w.
    • At confident anchors, Sh predicts the measured defect with R² = 0.95–0.97 on three models. On the same pairs, confidence features reach at most 0.13.
    • Sh > −log w certifies incoherence on 3–10% of all probed pairs.
    • The share rises from ≈0.2 to ≈1 as anchor entropy falls (Fig. 1c), and it grows as fewer positions remain masked (Fig. 2a).
    • Adjacent pairs have 20–60× the defect of distant ones (Fig. 2c).
  • Where. The masked-diffusion objective fits each masking pattern separately. Nothing ties the prediction with the anchor masked to the average of the predictions with it filled.
    • This matches the toy result you point to: a 600× better fit shrinks the defect and the score together without lowering the share.
    • On the enumerable model, with the same architecture and number of updates, adding that tie halves the defect, lowers the share from 0.20 to 0.14 and improves the conditional fit. It does not improve the sampler's output KL.
    • The MDLM-OWT fine-tunes we tested did not reduce the measured defect, and the term itself did not fall during training. They therefore do not establish whether enforcing the tie on a pretrained model would lower the defect or improve decoding.
  • Filtered subset. Table 1 now shows the unfiltered share (0.18–0.25) beside the filtered one, and §5.1 measures the positions a sampler commits.

Q1 (intuition). The model is 97% sure the anchor is Paris. If its predictions came from one joint distribution, its one-pass prediction for French would be its average over its own guesses for the anchor: 0.97×0.90 + 0.03×0.03 ≈ 0.87. In fact it is 0.60. The gap is the defect.

Q2. See W1 and §5.2–5.3.

As the discussion period closes soon, we would be grateful to know whether our response addresses your two concerns:

  • Whether the defect matters inside a deployed sampler. The audit now covers 120 decodes on two models and two tasks, and decodes that overrun the budget are correct less often.
  • When and why the defect is large. See the new Proposition 3 and the objective comparison on the enumerable model.

We are happy to clarify anything in the remaining time.

I am particularly interested in the generative-perplexity improvement in Figure 5b. However, since the measured defect does not decrease, the improvement appears largely unrelated to correcting the defect. Could the authors clarify how they interpret this result and whether it may instead arise from a generic self-distillation effect?

I would also be interested in seeing SMD tested on larger models such as LLaDA or SDAR, ideally on reasoning and code-generation tasks. In particular, it would be valuable to compare models trained with fully random masking against models trained with block-diffusion objectives, to determine whether the training structure changes the defect or the effectiveness of SMD.

Overall, the encouraging GenPPL results have improved my assessment. I have updated my score from 2 to 3.

Thank you for the follow-up and for raising your score.

How we read the GenPPL result. We agree that it cannot be attributed to the defect, and a generic self-distillation effect is the more likely explanation. Three observations point that way:

  • The measured defect did not decrease (×1.06 and ×1.02 against the matched controls).
  • The gain barely depends on how many positions are committed per step. At temperature 1, SMD's generative perplexity is 16–22% below the control's at every step count from T = 16 to T = 128 (64 down to 8 positions per step), with no trend; at matched entropy it is 12–14% at T = 32 and 9–11% at T = 128. A gain that came from a smaller parallel-step error should shrink markedly as fewer positions are committed per step.
  • SMD costs 0.010 nats per token in NELBO while improving samples, which is the pattern of a regulariser that trades likelihood for sample quality, not of a better fit.

So the result shows that the SMD term changes sampling, not that it does so through the defect. In the final version we will state this reading in §5.3; Appendix P already names self-distillation as the alternative explanation.

Larger models and block-diffusion objectives. We agree these are the right next tests; we could not run them within the discussion period. Two results bear on them:

  • Dream-7B, adapted from an autoregressive model, and LLaDA-8B, trained from scratch, have similar defect shares (0.45 and 0.41, Table 1).
  • In our static comparison, contiguous blocks carry 11–29× the summed defect of scattered sets of the same size (Appendix I), so the positions inside a block are where a block-diffusion model would have to control it.

Whether block-diffusion training changes the defect, and whether SMD then helps on GSM8K and HumanEval with LLaDA-8B or SDAR, remains untested. We will state it as the main open question in §5.3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers. All three found the paper hard to read, and Reviewer 4UVS asked what the diagnosis changes in practice. We have uploaded a revision; changed text is in blue.

A ten-minute reading route.

  1. Fig. 1: a worked example. The model is 97% sure an anchor is Paris, yet being told Paris moves its prediction elsewhere from 60% to 90% on French.
  2. §2.1: when the defect is large, with a certificate that costs one additional forward pass.
  3. Fig. 4: what happens inside EB-Sampler's own decodes.
  4. §5.2: four recommendations for sampler design.

What changed.

  1. The main text is rewritten around that example.

    • §4 is organised as four questions, and §4.1–4.4 and §5.1 each end with a boxed takeaway.
    • Full-resolution tables moved to Appendix F.
    • A guide at the start of the appendix maps each claim to its evidence.
  2. New theory.

    • Proposition 3 bounds the defect by the self-confirmation shift Sh: how far the prediction at the target moves when the anchor is filled with the model's own top guess, which has probability w.
      • The bounds are w·Sh − h(w) ≤ Δ ≤ D, with h the binary entropy.
      • Δ = 0 forces Sh ≤ −log w, so a larger shift certifies incoherence. Checking it costs one additional forward pass, with the anchor filled.
    • Corollary 1 extends Proposition 1 to whole decodes. It is proved in Appendix A and checked numerically, to 4×10⁻¹⁵, on 15 enumerable models.
  3. New experiments.

    • The in-sampler audit now covers 120 decodes of 80 problems (Appendix K): LLaDA-8B on GSM8K (60), Dream-7B on GSM8K (40) and LLaDA-8B on HumanEval (20).
      • By our jackknife point estimate, the defect is 0.55, 0.42 and 0.56 of the measured local cost of EB-Sampler's parallel steps.
      • The conservative estimator is underpowered at 32 rollouts. At 128 rollouts on LLaDA it gives 0.39 [0.24, 0.50], which excludes zero.
      • Decodes whose cost exceeds the budget are correct less often: 15 of 27, against 76 of 93 for the rest. Adjusting for forward passes and setting, with errors clustered by problem, the odds ratio is 0.22 (p = 0.015). This is an association, not a causal claim.
    • The self-confirmation shift on held-out documents, three models (Appendix H). At confident anchors it explains log Δ̂ with R² = 0.95–0.97. On the same pairs, confidence features reach at most 0.13.
    • Training objectives on the enumerable model (Appendix P).
      • A term that ties the two predictions together halves the defect and improves the conditional fit.
      • The fine-tunes of MDLM-OWT we tested with that term did not reduce the measured defect, and the term itself did not fall during training.

We look forward to the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.