- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies the cost of sampling several tokens independently in masked diffusion models, under random reveal patterns that are independent of the token values. It shows that the minimum conditional approximation error of one step is an average of the conditional total correlation of the newly revealed tokens (closely related to Lavenant & Zanella and Chen et al.). For a family of independent correlated pairs it computes the output distribution of the exact-marginal sampler in closed form, and shows that the optimal path error decays as 1/T in the number of steps while the output error decays as 1/T². For noisy parity blocks it derives a non-uniform optimal schedule, and it shows that representing the pair family as a finite mixture of product distributions needs exponentially many components, while a suitable ordering allows exact sampling in two rounds. Experiments check the formulas numerically and, on constructed span-infilling texts with MDLM and a Qwen3 diffusion model, compare reveal policies.
Critical strengths
-
A precise and checkable separation between path error and output error. The paper gives exact closed forms for a simple but non-trivial family, and shows that the cumulative conditional (path) error and the error of the final distribution scale differently with the number of steps. This clarifies that step budgets derived from path-level bounds can be conservative for output accuracy, and the formulas are verified numerically.
-
Clean theory that exposes how dependence structure shapes scheduling and representation. For noisy parity blocks all proper marginals are uniform, yet the optimal schedule is non-uniform, which shows that pairwise independence does not determine the optimal allocation of reveals. The component-count lower bound for mixtures of products and the two-round exact sampler are a neat illustration of what independent updates can and cannot represent. The assumptions (random reveal, exact marginals, specific families) are stated honestly.
Critical weaknesses
-
The strict results are limited to toy families and their practical meaning is unclear. The core identity for the single-step error is essentially known (Lavenant & Zanella; Chen et al.), as the paper acknowledges, so the new content lies in the explicit computations for independent pairs and parity blocks and the mixture-component bound. The headline 1/T versus 1/T² separation is proved only for independent correlated pairs, and it comes from the quadratic behaviour of the divergence near the target; the paper gives no argument that it carries over to general dependence structures (for the parity blocks only costs and schedules are derived, not output rates). The analysis also assumes exact marginals and data-independent random reveal, whereas practical samplers choose positions adaptively and learned marginals add an error that accumulates over steps. As a result, the reader learns that such effects exist in a simple model, but not how they should change the design or step budget of a real masked diffusion sampler.
-
There are no results on standard tasks and no improvement is proposed. The paper analyses the problem but proposes no method that can be used on real models, and there are no results on benchmarks such as math or code. The constructive procedures in the paper are existence-style constructions that require known dependency blocks and exact conditionals, and an appendix description of schedule selection from finite samples on tiny binary laws for which no results are reported. The only experiment with pretrained models uses constructed span-infilling texts, and its "dependency-aware" policy uses dependency spans that are supplied to the sampler, i.e. information that is not available in practice; the metric is reference-token NLL under argmax infilling rather than the quality of the generated text. This shows that knowing the dependencies helps, but not that the theory leads to any gain in actual generation, nor how to obtain the dependence information for natural text.
Questions for authors
- Which conclusions from the structured pair and parity examples do the authors expect to generalize to more realistic dependence patterns?
- How should these theoretical results guide the schedule or number of steps for a practical masked diffusion model when the dependence structure is unknown?