- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the problem of how to increase the diversity among a batch of samples/generations in masked dLLMs like LLaDA. They propose a sampler that promotes diversity within a batch by down weighting tokens that other samples in a batch sampled at a given position (Algorithm 1). While the proposed idea is interesting, some theoretical claims are unfounded (Proposition 1, see weakness section below) and experimental evidence is inconclusive about how the proposed approach compares with highly related work on improving diversity in dLLMs [1, 2].
Also, ofc I might be wrong, but the submitted manuscript feels AI generated in a lot of parts (introduction, entire Section 2). Why submit such a manuscript to a conference when the peer-review system is already so overloaded? :(
Critical strengths
- The setting of studying diversity in dLLMs is interesting and important in the light of recently discovered Flexibility Trap [3].
Critical weaknesses
- The writing is quite bad. For example introduction really doesn't read smoothly and it in many parts it provides too much information for introduction without first providing the right context. Some sentences/parts read like AI slop imo, for example: Ensembles without randomness. At temperature zero a symmetric start-of-step sampler is degenerate: its K samples coincide, so the vote cannot beat a single decode. Reading the peers within the step can make the paths differ.
- There is no background section on dLLMs or sampling in dLLMs. Some background info is scattered around the introduction and then Section 2 jumps right into the proposed method...
- The paper seems to claim that if we generate K samples/generations in dLLMs at then we get no diversity between K samples. But that's not true or I am misunderstanding the claims (Proposition 1)? Because even for greedy decoding (T=0) we can vary the generation order in dLLMs which will results in different generations for the same prompt? just take LLaDA ckpt from HuggingFace and generate with random unmasking strategy. see also see also in [1]
- Section 2.2 also feels AI generated, it's hard to read through it in details
- Experiments are weak. Main experimental figure is Figure 2 where they do not compare with highly related works like ODD [2] or TLC/TCT [1]. Also showing results only up to k=10 is not enough, at least k=16 or k=32 would be nice. And at least one more model besides LLaDA/LLaDA-1.5, it could be Dream/iLLaDA/SDAR/LLaDA-2 etc. Because of this limited experiments scope, it's impossible to judge the proposed method effectiveness at this point
Questions for authors
See weaknesses above.
- Also do I understand correctly that you assume that each sample in a batch unmasks same number of tokens in each sampling steps? Could you comment on how your proposed sampler is compatible with adaptive samplers like Fast-dLLM [4] for which the number of unmasked tokens can vary between sampling steps?
[1] Olausson et al. A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models
[2] Lamont et al. Free Lunch for Pass@k? Low Cost Diverse Sampling for Diffusion Language Models
[3] Ni et al. The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models
[4] Wu et al. Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding
We address the reviewers requests below. Several listed weaknesses concern material already in the submitted paper; we give the location in each case, and report the requested experiments.
W1 / W2. Writing; background
Response: We revise the introduction so that the setting (masked diffusion decoding, self-consistency, why K samples duplicate) precedes the method, and we add a background section on masked diffusion decoding, the confidence commit rule, pure diffusion versus blocks, and self-consistency. The quoted paragraph is Proposition 1 in two sentences; we unpack it. In Section 2.2 one sentence will introduce each result.
W3. Proposition 1 and random unmasking order
Response: Proposition 1 does not make the claim attributed to it. Its hypothesis is that each path's commit is one deterministic function of the canvases read at the start of the step; it then states that such a pool holds one distinct sample. A random unmasking order is not such a function. It is randomness applied to each path independently of its peers, the same class as token temperature (Section 2.3), and every temperature-zero statement in the paper concerns deterministic commit rules. The distinction is measurable: at matched disagreement, peer-blind randomness costs 7 to 13 per-sample points relative to the peer statistic (Table 3). We move the hypothesis into the statement and add the random-order case as a remark; the proposition and its proof are unchanged.
W4a. ODD and TLC/TCT
Response: ODD is compared throughout: at both step sizes in the main comparison (Table 2), at its published temperature with paired intervals (Table 12), and in pass@k (Table 16). Figure 2 isolates the coverage gap against self-consistency; we add ODD to it. For the rebuttal we also ran ODD under pure diffusion with the same EOS correction as SR (LLaDA-8B-Instruct, GSM8K, 1280 evaluations). SR leads ODD at both strengths, and its overhead over SC is a fraction of ODD's (cost: H200, K=10, blocks of 32):
TLC/TCT is cited in the introduction and related work. It samples which positions each path commits, independently of its peers, so it is a peer-blind sampler, and its authors explicitly report that this diversity does not reliably improve the self-consistency vote. SR couples the paths, which is why it yields distinct samples at temperature zero.
W4b. K beyond 10
Response: Table 6 (Appendix C.3) reports K up to 32. Coverage rises with K and the vote is flat; the paper states this as a limit.
W4c. Models beyond LLaDA
Response: We add Dream-v0-Instruct-7B (Qwen2.5-initialised, independent of LLaDA); LLaDA-1.5 is Table 11.
SR beats SC at matched temperature on both metrics (significant over 1200 pairs) and stays ahead of SC at its best temperature: the LLaDA block pattern. lambda=32 was picked on 200 tuning problems disjoint from evaluation.
Q1. Equal counts; Fast-dLLM
Response: Equal counts are not assumed. Within a step the paths commit in sequence, and the peer count at position j updates whenever a peer commits j, so the number of positions a path commits does not enter. Under a confidence threshold as in Fast-dLLM, each path commits its own set and the penalty applies unchanged. We ran only the fixed-count schedule and state adaptive schedules as untested.
We hope these corrections and experiments allow a reassessment of the soundness and informativeness scores.