- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper shows that two unmasking schedules for masked diffusion models (MDMs) that share the same "local frequencies" (defined as the frequency with which a schedule sees a certain unmasking problem of the form "unmask position z conditioned on positions x and y already being unmasked") and the same predictor (i.e., the trained model) can still yield distinct marginal distributions over generations. In experiments, they approximate this scenario with finite samples, and show the proven effect occurs in practice, under their specific setup.
Critical strengths
The theory seems to be sound, and the results seem correct (as far as I can tell), and could be interesting; however, I don't think the problem is motivated (see below), so the relevance of the results remains questionable.
Critical weaknesses
- Main concern: I think the problem statement of the paper has no practical motivation. The main result says two schedules with the same local problem frequencies can end up generating different marginal distributions. However, the setup under which this occurs assumes a schedule where the next token position to be revealed depends not only on the set of revealed tokens up to that point, but also on the order in which they've been revealed; at least one of the two schedules needs to have this property. Indeed, if both schedules with identical "local frequencies" do not depend on the order in which previously unmasked positions have been unmasked, then they (trivially) lead to the same marginal distribution. I believe this never shows up with MDMs in practice; even the two references the authors give themselves, uniform random sampling in LlaDA [1] and ancestral sampling in MDLM [2], do not use such schedules (these schedules don't depend on the exact order in which previously unmasked tokens have been unmasked). I am not aware of any schedule where this scenario comes up in practice, and the authors don't provide one, either; they only show this happens in toy examples. Furthermore, in practice, one typically uses more sophisticated schedules that depend on model predictions in some form (e.g. via model confidence scores) and that reveal more than one token at a time, which is also not covered by the author's setup. If the authors are able to provide convincing examples/references where this problem setup is actually relevant, I'm happy to reconsider my rating, but I currently believe the problem has no motivation in practice.
- The paper is very difficult to parse; sentences are verbose and syntactically overly complex. A lot of terminology is confusing and does not follow what's commonly used in the literature; example: "Revealed coordinates are never revised, so the terminal assignment retains every intermediate value."; much more natural and in line with common terminology would be something like "The MDM sampler does not remask/resample already unmasked positions." The whole paper reads like this. While I haven't checked with any tools, I'm pretty sure the paper is (mostly) AI generated.
- Some statements in the paper seem inaccurate; e.g., in the first two paragraphs of the introduction alone, it says that at training, one typically "predict[s] one of the remaining masked positions"; I'm not aware of any paper that does that, usually all remaining masked positions are predicted at a training step; and the second paragraph makes it sound like the inference schedule always determines the "local prediction problems" at training; however, usually training uses random masking independent of the inference schedule, and only more recently it has been explored to align the training schedules (approximately) with the inference schedule (e.g. [3]).
[1] Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. Large Language Diffusion Models. arXiv:2502.09992, 2025.
[2] Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and Effective Masked Diffusion Language Models. arXiv:2406.07524, 2024.
[3] Jaeyeon Kim, Jonathan Geuter, David Alvarez-Melis, Sham M. Kakade, and Sitan Chen. Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training. Proceedings of the 43rd International Conference on Machine Learning (ICML), 2026.
Questions for authors
- can you provide examples of schedules actually employed in practice that fit your setup, i.e. two schedules that have the exact same local problem frequencies, that do not depend on model predictions, and where at least one of the two schedules depends on the order in which past tokens have been umasked?
3 comments
W2 & W3: “paper is very difficult to parse” and “Some statements in the paper seem inaccurate”
A standard MDM training forward pass predicts all masked positions and aggregates their token losses; training masks are generally sampled independently of the inference schedule. We will make both points explicit in the Introduction with the following replacement. “A training forward pass predicts all masked positions and aggregates their token-level losses. Each loss term is indexed by the visible context and its target position. Training masks are generally sampled independently of the inference schedule.” Our local prediction term indexes one contribution to that sum. We will present training–inference alignment, including the work you cite by Kim et al. (2026), as an explicit training-design choice.
We will also use the simpler terminology you suggest. The sentence you highlighted will become “The sampler never remasks or resamples a revealed position.” We will explain the four-order example before the formal results, separate assumptions from conclusions in long sentences, and describe the largest TV difference between frequency-matched schedules before introducing its name. These changes will make the setting and the main question easier to follow from the start.
Thank you for your response
I thank the authors for the rebuttal. However, I don't believe my main concerns have been adequately addressed. My concern that the paper lacks motivation, and that their main result is not applicable to MDM samplers in practice, remains. The authors references two samplers: (1) the MDLM sampler; however, this sampler does not make position choices depend on previous reveal order in practice. (2) The paper by Uria et al. (2014); it seems like the authors just retrofitted this paper to my review, but it is not actually about MDM sampling at all, and predates MDMs by many years (while their sampler does seem to have the required order-dependence as far as I can see). Together, this means my main concern has not been sufficiently addressed. I also still believe the paper needs to be reworked in terms of its writing, and encourage the authors to do that. I will maintain my initial rating, and recommend rejection of the paper. I hope the authors understand my decision.
Thank you for your careful review and helpful comments.
W1 & Q1: “the problem statement of the paper has no practical motivation” and “examples of schedules actually employed in practice”
The practical motivation is to determine whether matching the local prediction tasks used during generation is enough to preserve the output distribution. A relevant example in existing MDM practice is finite-step ancestral sampling with parallel reveals, covered by our block extension in Proposition 11. Its diffusion clock carries information beyond the filled set. This connects the question to the MDLM sampler itself and to what local training–inference alignment can guarantee.
In the ancestral sampler of Sahoo et al. (2024, Section 3.2.3), positions are revealed using a value-independent clock, and already revealed tokens are retained. For the time-independent MDLM-OWT predictor, we can omit empty updates and represent the remaining steps as an ordered sequence of blocks. Earlier blocks carry information about how much of the clock remains. To see this directly, take four positions and four equal time intervals, conditional on all positions being revealed before the final cleanup. Each position then independently chooses one of four reveal intervals. After the prefix ({1,2}), the probability that the next block is {3,4} is 3/7; after ({1},{2}), it is 2/3. The filled set is identical. More generally, under the same conditioning, with two positions remaining the probability of revealing them together is (m+2)/(2K−m) after m nonempty blocks in K intervals. The full state (S,t) is Markovian, but the set S alone omits clock information that affects the continuation.
We can preserve every local frequency while removing this information. On the branch that finishes before cleanup, let f(S,B) be the ancestral sampler's frequency of revealing block B from set S. Our matched control chooses
pˉ(B∣S)=∑C=∅f(S,C)f(S,B).
Flow conservation ensures that it visits every block edge with exactly the same frequency. Consequently, it also preserves every token-level frequency ftoken(S,i)=∑B∋if(S,B). Both laws ignore predictions and generated values and use the same parallel block predictor. The baseline is the existing ancestral sampler; the set-only replacement is our control for testing whether its local statistics suffice. Retaining this branch's probability and leaving the complementary cleanup branch unchanged also preserves local frequencies for the full sampler.
Finite-order NADE ensembles provide the single-position counterpart (Uria et al., 2014, Section 4). They select a complete order before generation and retain it. Our libraries {1234,2143} and {1243,2134} illustrate how this operation permits equal edge frequencies with different continuations after the same set. We measured this effect in eight controlled MDLM-OWT comparisons between random 32-order libraries and their set-only replacements. With identical edge frequencies, predictor, and continuation, output TV was 4.36% to 5.69%, with positive simultaneous 95% lower bounds in all eight FP32 comparisons. The order-library measurements quantify the response under learned conditionals, while the ancestral-clock calculation identifies a source of history dependence in an existing MDM sampler.
The relevance to alignment is concrete. Shih et al. (2022) weight conditional prediction losses by inference-time visitation frequencies. Equal local frequencies make this additive objective identical for every predictor. Our results identify when the generation distributions can nevertheless differ, what path information is missing, and how to detect the difference without retraining. Set-only rules, including idealized single-position random LLaDA, form the sufficient case because their edge frequencies recover every transition. Finite-order ensembles and the clock-dependent block formulation explain why the broader question arises. Confidence-based policies require including the prediction-dependent action probabilities. We will bring these examples and distinctions into the Introduction and the discussion beside Definition 1, so the practical question is clear before the formal results.