- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how token predictions evolve during continuous diffusion language model (CDLM) sampling. The authors present a framework that categorises the prediction dynamics into commitment, revisitation, and exploration. Experiments across four existing CDLMs show that token probability distributions can repeatedly move away from and return toward earlier predictions, even after the top-1 token stabilises. The authors associate this behaviour with candidate competition and cross-position feedback. Based on these motivations, they propose velocity blending to control sampling updates as a training free control to reduce sampling time. While the paper studies a timely problem, I have several concerns regarding motivation and evaluation.
Critical strengths
- The paper investigates a timely and relatively underexplored problem on the prediction dynamics in CDLMs, which provides interesting insights into how token predictions evolve during sampling.
- The proposed framework offers a structured way to examine prediction changes beyond top-1 token stability by distinguishing prediction dynamics into exploration, revisitation, and commitment.
- Diverse experiments and empirical observations were conducted on four existing CDLMs.
Critical weaknesses
- While the paper presents several studies and motivates revisitation as potentially inefficient, it does not provide direct evidence that positions exhibiting more revisitations contribute less to generation quality or require unnecessary computation. In fact, revisitation may reflect useful refinement of dependencies between tokens. The inconsistent quality effects of velocity damping discussed in Lines 427-429 further weaken this motivation.
- Additionally, the authors define revisitation using a four-step window and fixed thresholds (Line 198). However, it is unclear whether this detects meaningful changes in token predictions or simply small fluctuations in probability distributions. Potentially need to compare with simpler measures, such as changes in token confidence or entropy, and demonstrate what additional information it provides.
- Table 1 appears to be difficult to interpret and understand, the caption and discussion in Sec 5.2 requires further and more intuitive explanation. For example, it is unclear why late(tail) merging changes only 2.40% of tokens in LangFlow but 61.51% in Plaid, despite having similar runtime reductions. It is recommended to explain why the method behaves so differently across models and evaluate its performance using generation quality rather than primarily comparing outputs with the original sampler.
- While the paper is structurally clear, the content is genuinely difficult to follow. It is suggested for the authors to provide more intuitive explanations when introducing key concepts, such as commitment, revisitation, and exploration in Section 3.2. Similarly, Section 5.2 would benefit from a deeper discussion of the insights and implications of the results, rather than merely reporting statistics.
Questions for authors
- Currently, the connection between section 4 and 5 is relatively weak, could the authors elaborate on how the proposed framework and empirical findings translate into practical improvements in CDLM sampling?
- Shown in the experiments, the four CDLMs appears to have quite different prediction dynamics. Could the authors provide some insights on which architectural or sampling characteristics contribute to these differences?
- How should the relationship between generation quality and commitment be interpreted?
7 comments
W3: Table 1; LangFlow 2.40% vs. Plaid 61.51%
Thank you for pointing this out; you are right that this difference needed more explanation.
For LangFlow's 2.40% against Plaid's 61.51%, our analysis is that the difference comes from when positions commit, as the submission noted (lines 469–471; see Figure 3). Tail skips late steps, which leaves the output unchanged only where positions have already settled. By late sampling, LangFlow's output is already settled, whereas Plaid's is not: at 90% of sampling progress, 43.0% of Plaid positions are still exploring or revisiting, against 9.2% in LangFlow (lines 264–265). Merging the same late steps therefore changes Plaid's output but not LangFlow's, although both save a similar share of calls.
New experiments. Following your comment, we tested this explanation and evaluated generation quality directly; Section 5.2 now summarizes both (lines 469–476), and Appendices F.7 and F.8 give the details.
- Noise control. Without any merging, keeping Plaid's sampling schedule and redrawing only the noise of the second half of sampling changes 75.3% of its tokens (Appendix F.7), so Plaid's late steps still decide its output.
- Generation quality. As you suggest, we now evaluate quality on new test sets rather than agreement with the native output. On 500 new XSum and 500 new WMT14 items, Tail with a quarter fewer calls matches the native sampler (ROUGE-L 26.50 against 26.47; BLEU 23.75 against 23.67) and beats a uniform grid with the same calls by 0.58 ROUGE-L and 0.78 BLEU, with BERTScore and COMET agreeing (Appendix F.8). With a stronger scoring model and MAUVE, Tail on unconditional ELF also matches the native sampler (Qwen2.5-7B NLL ; MAUVE 0.86 against 0.85), whereas Head and Uniform fall to MAUVE 0.31 and 0.61.
In short, the gap is explained by when positions commit, and Tail keeps the native quality on new test sets.
W4.1: Intuition for the modes
Thank you for this suggestion on the writing. The introduction already presented the three modes informally with Figure 1 (lines 68–73); following your suggestion, Section 3.2 now also gives plain-language definitions next to the formal ones (lines 200–202):
- Exploration: the position has not yet settled on its final token.
- Commitment: its top-1 token is final and stays final.
- Revisitation: its distribution moves away from an earlier state and comes back, with or without a top-1 change.
Section 3.2 thus now gives the intuition you asked for alongside the formal definitions.
W4.2: Discussion in Section 5.2
Thank you for the reminder. In the submission, Section 5.2 reported the results concisely for lack of space. With the additional page, Section 5.2 now ends with a paragraph on what the results imply (lines 492–497): where steps can be saved is set by commitment rather than by time, and preserving the native output is a different goal from improving it. As you suggested, Section 5.2 now discusses what the results mean rather than only reporting them.
Q1: From Section 4 to Section 5
Thank you for this question. Overall, Section 4 measures when and how predictions settle, and Section 5 tests what this implies for sampling. We will try our best to make the link clearer, to answer your question:
How Section 4 leads to Section 5. The submission opens Section 5 with two questions that follow from Section 4 (lines 372–374): whether late updates, which act mostly on committed positions, can be omitted with little change to the output, and whether revisitation responds when consecutive velocities are blended. Each experiment builds on one observation of Section 4:
- Late updates act mostly on committed positions (Section 4.1). Section 5 merges late intervals: at calls in LangFlow, ELF, ELF-XSum, and German-to-English, Tail changes 0.3–5.1% of tokens, whereas Head changes 5.5–74.1%.
- Positions commit at model-specific times (Section 4.1). The same merge is not safe in Plaid, whose output is still being decided late in sampling.
- Revisitation is competition among a few recurring candidates (Section 4.2). Damping velocity changes at revisiting positions changes revisitation locally, without a consistent quality gain yet.
The findings translate into practice through commitment: once positions have committed, later updates can be merged, saving a quarter of model calls at native quality where commitment is reached early, as for the deterministic samplers on unconditional generation, summarization, and translation (Appendix F.8). How far the findings carry over to faster or better samplers in general remains open, and the current approach (commitment-based merging) is a first step in that direction.
What the revision adds. Section 4.1 now states the share of late updates that act on committed positions, which is what Section 5 builds on (lines 269–270), and Section 5.2 now ends with a paragraph on what the results imply for sampling (lines 492–497).
Thank you for a careful and expert review. We ran the comparison with simpler measures you suggest, quality evaluations on new test sets, and a noise control that explains the Plaid gap; Appendix F reports them. We have updated the paper, and line numbers below refer to the revised version.
W1: Revisitation may be useful refinement rather than waste
Yes, we agree that revisitation can be useful refinement. At the same time, revisitation does consume computation, since every back-and-forth update costs a model evaluation, and this is why we study it; the paper does not treat it as waste (lines 509–511 and 532–533). What our results separate is when it happens:
The efficiency result rests only on the second case. Because late updates act mostly on committed positions, merging them saves a quarter of model calls (24% of wall-clock time in ELF) while changing only 0.3–5.1% of tokens in LangFlow, ELF, ELF-XSum, and German-to-English (Table 1). Revisitation also consumes model evaluations, which is why the paper characterizes it in detail; which part of it can be saved is what Section 5 tests rather than assumes. The damping result you cite is consistent with this: damping revisitation directly does not reliably improve quality. In short, revisitation before commitment can be useful refinement, as you suggest, and the saving we report comes only from the calls after commitment.
W2: Meaningful changes or small fluctuations; simpler measures
We did not use confidence or entropy because they answer a different question: they describe one distribution at one step, whereas revisitation asks whether a distribution returns toward its own earlier state. A return can leave confidence and the top-1 token unchanged, and a large change in entropy need not be a return. We ran the comparison you suggest on the native runs of all four models (new Appendices F.1 and F.2):
Small fluctuations. The detector is designed to ignore them, and new checks confirm this:
Together, these checks show that the detector measures substantial returns toward an earlier state, which entropy and confidence do not capture.