- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies questions related to the accuracy of an efficient (parallel) sampling from masked diffusion models (MDM). In particular, the authors ask how large could be the total variation distance from the conditional distribution learned by "heads" of the model to the target one, so that the parallel sampling would still be accurate.
Critical strengths
I cannot identify any strengths of the work. It is written in an obscure, not self-containing way. I had to read the cited works just to get an idea of what the authors intended to do.
Critical weaknesses
The writing prevents to understand what is the problem, contribution and relation to prior works. In particular, the title, the first line of the abstract and the posed question contain "noisy pairs", which is not an established term and requires clarification. There is no explanation for the introduced notation in the background section, what is , what is the teacher ? What is "shallow parallel generation"?
Questions for authors
Could you formulate in your own words what is the problem you want to study, what is the contribution of the paper and how it relates to the previous works?
21 comments
【Part 2】 Rebuttal by Authors
W2. The definitions of , the target , and the conditional heads
R2: is the full random vector, and is the distribution we want the sampler to generate. Specifically,
$ X=(A_1,B_1,\ldots,A_n,B_n), \qquad B_i=A_i\oplus E_i, $
where all and are mutually independent under . Thus consists of independent copies of the pair described above. The main text now calls the target distribution. The appendix explains “teacher” as a synonym for this target.
The supplied joint model is denoted by . Its conditional prediction, or head, for a missing coordinate at visible context is
$ q_{R,i}(a\mid x_S) =\Pr_{R_n}(X_i=a\mid X_S=x_S), \qquad a\in{0,1}. $
We call the predictions coherent because all these heads come from the same joint law . The revised setup introduces the random vector and both distributions before stating their accuracy and sampling properties.
Manuscript changes: Target vector and law in Section 2, Eq. (2) (p. 2); explicit head formula in Section 2 (pp. 2–3); proof terminology in Appendix A (p. 16).
【Part 3】 Rebuttal by Authors
W3. The meaning of “shallow parallel generation”
R3: We mean generating the entire vector within a prescribed budget of sampling rounds. In each round, the sampler chooses a set of missing coordinates, draws their values independently from the supplied heads at the context visible at the start of that round, and keeps the generated values fixed. Later rounds may use those values to choose the next batch; its predictions are the original heads evaluated at the updated context.
The planner may know both and , use private randomness, and adapt to the revealed values. Its choice of positions precedes the new draws. These rules specify the sampling interface and the resource being measured. With the exact target heads, all first bits can be generated in one round and all second bits in another, so the target has an exact two-round sampler.
Manuscript changes: Two-round example in Section 1 (p. 1); formal execution rule in Definition 1, Section 2 (p. 3); an example of position selection using the current predictive distributions in Section 2 (p. 3).
【Part 1】 Rebuttal by Authors
We sincerely thank Reviewer WuCj for reviewing our manuscript and for identifying where the terminology, problem statement, and contribution were difficult to follow. We appreciate the opportunity to clarify these points. Your comments have prompted a substantial revision of the opening exposition. Below, we explain the setup directly and respond separately to the terminology questions, the research problem, the contribution, and the relation to prior work.
All page references below refer to the revised manuscript.
W1. The meaning of “noisy pairs” and the clarity of the title and abstract
R1: Thank you for pointing out that this shorthand needed an explanation. The construction is a pair of correlated bits: draw a fair bit A, then obtain B by flipping A with probability 1/4. Both bits have fair marginals, but they disagree with probability 1/4. Drawing them independently instead makes them disagree with probability 1/2. Drawing A first and then B conditional on A gives the exact joint distribution.
The revised title is “How Model Accuracy Limits Parallel Sampling in Masked Diffusion: Sharp Thresholds for Correlated Bit Pairs.” It identifies the field, sampling problem, and target family directly. The revised abstract starts by explaining parallel generation in masked diffusion, then describes the target in plain language. The Introduction gives the explicit two-bit example before introducing the theoretical question.
Manuscript changes:
Title and abstract (p. 1); elementary example inSection 1 (p. 1); formal pair construction inSection 2, Eq. (2) (p. 2).