acceptodds
Under review as a conference paper at ICLR 2027

Explore, Revisit, or Commit: Prediction Dynamics in Continuous Diffusion Language Models

Abstract

Continuous diffusion language models (CDLMs) refine continuous states without committing to tokens, so although every sampling step costs a full model evaluation, what each step contributes remains unclear. We address this question by studying prediction dynamics, how CDLM updates form and revise token predictions, through a framework that assigns each position at each step to commitment, revisitation, or exploration. Applying it to four representative CDLMs (LangFlow, Plaid, ELF, and Cola-DLM), we find that revisitation, in which the predicted distribution moves away from an earlier state and then back toward it, is widespread: it persists after the top-1 token stabilizes, and additional sampling steps advance commitment without removing it. We hypothesize that revisitation stems from candidate multiplicity and cross-position dependence, and our experiments support both sources: revisitation often involves competition among a few candidates, with the same rival returning at the same position, and in LangFlow and ELF, freezing the prediction memory of other positions changes how a revisiting position is revised. Building on these findings, we propose velocity blending, a training-free sampling control that blends consecutive velocities in two ways: late interval merging saves about a quarter of sampling time while changing far fewer native tokens than early merging, whereas velocity damping has setting-dependent effects on revisitation and quality. Taken together, our results separate observing, altering, and exploiting prediction dynamics.

Primary area:
generative models
Keywords:
continuous diffusion language models, generation dynamics, generative models, mechanistic analysis, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KJbqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how token predictions evolve during continuous diffusion language model (CDLM) sampling. The authors present a framework that categorises the prediction dynamics into commitment, revisitation, and exploration. Experiments across four existing CDLMs show that token probability distributions can repeatedly move away from and return toward earlier predictions, even after the top-1 token stabilises. The authors associate this behaviour with candidate competition and cross-position feedback. Based on these motivations, they propose velocity blending to control sampling updates as a training free control to reduce sampling time. While the paper studies a timely problem, I have several concerns regarding motivation and evaluation.

Critical strengths
  1. The paper investigates a timely and relatively underexplored problem on the prediction dynamics in CDLMs, which provides interesting insights into how token predictions evolve during sampling.
  2. The proposed framework offers a structured way to examine prediction changes beyond top-1 token stability by distinguishing prediction dynamics into exploration, revisitation, and commitment.
  3. Diverse experiments and empirical observations were conducted on four existing CDLMs.
Critical weaknesses
  1. While the paper presents several studies and motivates revisitation as potentially inefficient, it does not provide direct evidence that positions exhibiting more revisitations contribute less to generation quality or require unnecessary computation. In fact, revisitation may reflect useful refinement of dependencies between tokens. The inconsistent quality effects of velocity damping discussed in Lines 427-429 further weaken this motivation.
  2. Additionally, the authors define revisitation using a four-step window and fixed thresholds (Line 198). However, it is unclear whether this detects meaningful changes in token predictions or simply small fluctuations in probability distributions. Potentially need to compare with simpler measures, such as changes in token confidence or entropy, and demonstrate what additional information it provides.
  3. Table 1 appears to be difficult to interpret and understand, the caption and discussion in Sec 5.2 requires further and more intuitive explanation. For example, it is unclear why late(tail) merging changes only 2.40% of tokens in LangFlow but 61.51% in Plaid, despite having similar runtime reductions. It is recommended to explain why the method behaves so differently across models and evaluate its performance using generation quality rather than primarily comparing outputs with the original sampler.
  4. While the paper is structurally clear, the content is genuinely difficult to follow. It is suggested for the authors to provide more intuitive explanations when introducing key concepts, such as commitment, revisitation, and exploration in Section 3.2. Similarly, Section 5.2 would benefit from a deeper discussion of the insights and implications of the results, rather than merely reporting statistics.
Questions for authors
  1. Currently, the connection between section 4 and 5 is relatively weak, could the authors elaborate on how the proposed framework and empirical findings translate into practical improvements in CDLM sampling?
  2. Shown in the experiments, the four CDLMs appears to have quite different prediction dynamics. Could the authors provide some insights on which architectural or sampling characteristics contribute to these differences?
  3. How should the relationship between generation quality and commitment be interpreted?
7 comments

Thank you for a careful and expert review. We ran the comparison with simpler measures you suggest, quality evaluations on new test sets, and a noise control that explains the Plaid gap; Appendix F reports them. We have updated the paper, and line numbers below refer to the revised version.

W1: Revisitation may be useful refinement rather than waste

Yes, we agree that revisitation can be useful refinement. At the same time, revisitation does consume computation, since every back-and-forth update costs a model evaluation, and this is why we study it; the paper does not treat it as waste (lines 509–511 and 532–533). What our results separate is when it happens:

  • Before commitment, revisitation decides the token: these episodes often end with the rival (34–61% in LangFlow, Plaid, and ELF; lines 302–304). This is the refinement you describe.
  • After commitment, revisitation almost never changes the output: the position returns to its original token in at least 96% of cases across the four models (lines 302–304), and in 97% on summarization and translation (new analysis, now in Appendix D.2).

The efficiency result rests only on the second case. Because late updates act mostly on committed positions, merging them saves a quarter of model calls (24% of wall-clock time in ELF) while changing only 0.3–5.1% of tokens in LangFlow, ELF, ELF-XSum, and German-to-English (Table 1). Revisitation also consumes model evaluations, which is why the paper characterizes it in detail; which part of it can be saved is what Section 5 tests rather than assumes. The damping result you cite is consistent with this: damping revisitation directly does not reliably improve quality. In short, revisitation before commitment can be useful refinement, as you suggest, and the saving we report comes only from the calls after commitment.

W2: Meaningful changes or small fluctuations; simpler measures

We did not use confidence or entropy because they answer a different question: they describe one distribution at one step, whereas revisitation asks whether a distribution returns toward its own earlier state. A return can leave confidence and the top-1 token unchanged, and a large change in entropy need not be a return. We ran the comparison you suggest on the native runs of all four models (new Appendices F.1 and F.2):

ModelDetected windows with no top-1 change (%)Windows with a large entropy change that are not detected (%)AUC of simple measures for predicting detection
LangFlow98.098.20.47–0.57
Plaid81.795.80.39–0.57
ELF74.999.60.48–0.66
Cola-DLM89.199.30.35–0.44
  • The first column shows that most revisitation is invisible to the top-1 token, and therefore to measures based on confidence or top-1 changes.
  • The second column shows that large entropy changes are almost never returns: entropy changes whenever probability moves, in any direction.
  • The third column tests four single-step measures (change in entropy, change in top-1 confidence, step size, and number of top-1 changes). An AUC of 0.5 means chance, and none exceeds 0.66, so these measures cannot tell which windows contain a return.

Small fluctuations. The detector is designed to ignore them, and new checks confirm this:

  • An excursion counts only if it exceeds a floor of ϵ=10−3\epsilon=10^{-3} in Hellinger distance over the full vocabulary; this floor was already part of the submitted detector (lines 209–213).
  • Revisitation remains in all four models when the floor is raised tenfold, when the return must also be large in absolute terms, and under a strict configuration that requires both a large excursion and a near-complete return (new; Appendix F.1; also see in our response to Reviewer M1rM, W1.1).

Together, these checks show that the detector measures substantial returns toward an earlier state, which entropy and confidence do not capture.

W3: Table 1; LangFlow 2.40% vs. Plaid 61.51%

Thank you for pointing this out; you are right that this difference needed more explanation.

For LangFlow's 2.40% against Plaid's 61.51%, our analysis is that the difference comes from when positions commit, as the submission noted (lines 469–471; see Figure 3). Tail skips late steps, which leaves the output unchanged only where positions have already settled. By late sampling, LangFlow's output is already settled, whereas Plaid's is not: at 90% of sampling progress, 43.0% of Plaid positions are still exploring or revisiting, against 9.2% in LangFlow (lines 264–265). Merging the same late steps therefore changes Plaid's output but not LangFlow's, although both save a similar share of calls.

New experiments. Following your comment, we tested this explanation and evaluated generation quality directly; Section 5.2 now summarizes both (lines 469–476), and Appendices F.7 and F.8 give the details.

  • Noise control. Without any merging, keeping Plaid's sampling schedule and redrawing only the noise of the second half of sampling changes 75.3% of its tokens (Appendix F.7), so Plaid's late steps still decide its output.
  • Generation quality. As you suggest, we now evaluate quality on new test sets rather than agreement with the native output. On 500 new XSum and 500 new WMT14 items, Tail with a quarter fewer calls matches the native sampler (ROUGE-L 26.50 against 26.47; BLEU 23.75 against 23.67) and beats a uniform grid with the same calls by 0.58 ROUGE-L and 0.78 BLEU, with BERTScore and COMET agreeing (Appendix F.8). With a stronger scoring model and MAUVE, Tail on unconditional ELF also matches the native sampler (Qwen2.5-7B NLL +0.016 [−0.009,+0.042]+0.016\ [-0.009,+0.042]; MAUVE 0.86 against 0.85), whereas Head and Uniform fall to MAUVE 0.31 and 0.61.

In short, the gap is explained by when positions commit, and Tail keeps the native quality on new test sets.

W4.1: Intuition for the modes

Thank you for this suggestion on the writing. The introduction already presented the three modes informally with Figure 1 (lines 68–73); following your suggestion, Section 3.2 now also gives plain-language definitions next to the formal ones (lines 200–202):

  • Exploration: the position has not yet settled on its final token.
  • Commitment: its top-1 token is final and stays final.
  • Revisitation: its distribution moves away from an earlier state and comes back, with or without a top-1 change.

Section 3.2 thus now gives the intuition you asked for alongside the formal definitions.

W4.2: Discussion in Section 5.2

Thank you for the reminder. In the submission, Section 5.2 reported the results concisely for lack of space. With the additional page, Section 5.2 now ends with a paragraph on what the results imply (lines 492–497): where steps can be saved is set by commitment rather than by time, and preserving the native output is a different goal from improving it. As you suggested, Section 5.2 now discusses what the results mean rather than only reporting them.

Q1: From Section 4 to Section 5

Thank you for this question. Overall, Section 4 measures when and how predictions settle, and Section 5 tests what this implies for sampling. We will try our best to make the link clearer, to answer your question:

How Section 4 leads to Section 5. The submission opens Section 5 with two questions that follow from Section 4 (lines 372–374): whether late updates, which act mostly on committed positions, can be omitted with little change to the output, and whether revisitation responds when consecutive velocities are blended. Each experiment builds on one observation of Section 4:

  • Late updates act mostly on committed positions (Section 4.1). Section 5 merges late intervals: at 0.75×0.75\times calls in LangFlow, ELF, ELF-XSum, and German-to-English, Tail changes 0.3–5.1% of tokens, whereas Head changes 5.5–74.1%.
  • Positions commit at model-specific times (Section 4.1). The same merge is not safe in Plaid, whose output is still being decided late in sampling.
  • Revisitation is competition among a few recurring candidates (Section 4.2). Damping velocity changes at revisiting positions changes revisitation locally, without a consistent quality gain yet.

The findings translate into practice through commitment: once positions have committed, later updates can be merged, saving a quarter of model calls at native quality where commitment is reached early, as for the deterministic samplers on unconditional generation, summarization, and translation (Appendix F.8). How far the findings carry over to faster or better samplers in general remains open, and the current approach (commitment-based merging) is a first step in that direction.

What the revision adds. Section 4.1 now states the share of late updates that act on committed positions, which is what Section 5 builds on (lines 269–270), and Section 5.2 now ends with a paragraph on what the results imply for sampling (lines 492–497).

Reviewer_rS3hOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes the prediction behaviors of several existing continuous diffusion language models. The predictions at each position and sampling step are categorized into three classes: exploration, revisitation, and commitment, based on whether the predicted token has stabilized compared to the final output and whether the predicted distribution resembles an earlier state. This paper proposes quantitative measures to distinguish these three types of predictions and profiles the predictions of existing continuous diffusion language models. The results show that revisitation is prevalent across these continuous diffusion language models, which can cause unnecessary inference overhead. This paper proposes two hypotheses to explain revisitation: candidate competition and the influence of updates at other positions. It further proposes a controlled sampling method, velocity blending, to improve sampling efficiency and mitigate revisitation.

Critical strengths

This paper characterizes the prediction behaviors of continuous diffusion language models and provides an empirical analysis of several existing models. Their findings might help develop better sampling algorithms

Critical weaknesses
  • The writing caused me a lot of confusion while following this paper, and I feel it should be substantially improved. I specify a few, but not limited to them:
    • Section 3.1: observation n∈Nn \in \mathcal{N}. It's not clear at first glance whether nn denotes a predicted object or an index. For an index, maybe it's better to use \mathbb{N}, the natural numbers, rather than \mathcal. The distinction between nn and kk is also unclear. There is no nn or NN symbol in Figure 2
    • Section 3.2: "Consequently, after stable commitment, a position can alternate between commitment and revisitation". This needs more explanation, especially if it's an intended design choice
    • Table 1: would be better to bold the key entries you want to show. Now it looks a bit messy. Some entries retain the 0 before the decimal point, while others do not
  • I don't think the prediction behaviors (exploration, revisitation, etc.) are unique to continuous diffusion language models. Discrete diffusion language models with uniform noise (Generalized Interpolating Discrete Diffusion) or remasking (the ones you cited) exhibit the same phenomena. So the findings and the hypotheses proposed in this work on "revisitation" are kind of trivial to me. I understand the 'continuous vs discrete' difference, but besides that, regarding the patterns or dynamics of model predictions, what makes the findings of this work novel/interesting? Can you expand more on the connection/difference compared with existing studies from discrete diffusion language models?
  • The experimental results are not strong enough to support the claims made in this paper. I can see that the fixed tail merging reduces the sampling time, but it has worse quality in some cases. The changes in revisitation are also mixed, with no clear improvement.
Questions for authors

Please see Weaknesses

7 comments

Thank you for the review and in particular for your summary. It states the three modes, the two criteria that define them, and our two hypotheses exactly as we intended, and we are glad that the core of the paper came across. We address your three concerns below; line numbers refer to the revised version.

W1: Writing issues

Thank you for these concrete suggestions. We have revised each of these places, and we have also gone through the whole paper again to correct and polish it. For your convenience, we clarify each point here.

  • nn, N\mathcal N, NN. nn is an index over the ordered readouts of a run, and NN is its last value, the terminal observation. In the submission, N\mathcal N denoted the set {1,…,N}\lbrace 1,\dots,N\rbrace, which we used to shorten the formulas. Following your suggestion, and to remove the ambiguity you point out, the revision drops the separate set symbol: Section 3.1 now reads "observation n=1,…,Nn=1,\dots,N, an index over the ordered diagnostic readouts of a run", so nn is written explicitly as an integer index. We have also added a notation table at the start of Appendix C.
  • Figure 2 has no nn or NN because it shows solver updates, indexed by kk up to KK. To reduce the confusion further, we have updated Figure 2's caption, which now states that nn counts the readouts of pp collected along these updates. Figure 3, which plots against nn, now labels its axis as the normalized observation index, and its caption writes the sample size as "64 runs per model" rather than reusing nn.
  • Alternation between commitment and revisitation. Yes, this is intended. The submission already stated the defination (line 203) and this consequence (lines 231–233); following your suggestion, the revision adds an explanation right after it (lines 233–235), which we summarize here. Taken literally, the two conditions can overlap; this follows from the design, in which the two modes look at different things. Commitment is a property of the top-1 token: a position is committed from the observation after which its top-1 token no longer changes. Revisitation is a property of the whole distribution: it is detected when the distribution moves away from an earlier state and comes back, which can happen while the top-1 token stays the same. For the analysis, we want every position to carry exactly one mode at each observation, so revisitation takes priority where the two overlap.
    • For example, a committed position is also revisiting when its final token loses some probability to a rival and then regains it, without the rival ever becoming top-1. Its label then reads commitment, revisitation during the return, and commitment again; this is the alternation in Section 3.2. The top-1 token stays final throughout.
  • Table 1. Thank you for pointing this out; bold now marks the key entries, as you suggest, and the table now uses a more consistent number format.

W2: What is new relative to discrete diffusion

We agree that some of these behaviors are not unique to CDLMs, and the submission already discussed them in discrete diffusion: remasking and deferred commitment reopen or postpone token decisions (lines 130–132), revision policies such as STaRR and Posterior Refinement decide which tokens to change (lines 149–151), and we contrasted CDLMs only with samplers that retain decoded tokens (lines 76–77). The main difference from these studies is what is measured: they follow the decoded top-1 token, whereas we follow the whole predicted distribution, so we can see a position revise its prediction even while its top-1 token stays the same (Section 3.2). What is new is therefore the dynamics beneath the top-1 token, and the paper reports three findings about them that a token-level record cannot show:

  1. Updates at committed positions still do work. In the last quarter of sampling, 56–95% of position updates act on committed positions, and they carry 28–77% of the probability movement (lines 259–260).
  2. Revisitation is competition among a few candidates. Lost probability flows to a single rival, and the same rival returns at the same position (lines 302–307). Episodes that begin before commitment end with the rival in 34–61% of cases in LangFlow, Plaid, and ELF, whereas those that begin after commitment return to the leading token in at least 96% (lines 299–300).
  3. It depends on other positions. Holding the prediction memory of the other positions changes which way a revisiting position revises (lines 351–354). Conversely, a committed position, whose top-1 token no longer changes, still shapes the rest of the sequence through its memory: perturbing only that memory changes predictions at other positions, including distant ones, and the effect grows over the next four predictions while the position's own top-1 stays the same (lines 355–357; Appendix F.5).

Findings 2 and 3 support the two sources of revisitation proposed in the introduction. They matter in a CDLM because its next update uses the full predicted distribution through the continuous state and prediction memory (lines 77–80), whereas remasking and uniform-noise samplers carry forward only the selected tokens (lines 130–133).

W2, continued. New experiment on discrete models. If discrete diffusion models also revisit, what is specific to CDLMs? To answer this, we ran our detector, unchanged, on four discrete diffusion models, MDLM, ReMDM, DUO, and GIDD (Appendix F.3; lines 267–269), and asked two questions for each model:

  1. Does the denoiser's predicted distribution revisit?
  2. Does what the sampler carries to the next step revisit?

In a CDLM the two are the same, because the whole predicted distribution enters the next update. In a discrete model they can differ, because the sampler draws tokens from the prediction and carries only those tokens forward.

ModelSamplerRevisitation in the denoiser's prediction (% of windows)Revisitation in what is carried to the next step (% of windows)
MDLMmasking3.35>0.000
ReMDMremasking4.330.140
DUOuniform noise3.360.006
GIDDmask + uniform1.840.133
LangFlowCDLM6.13same as prediction
PlaidCDLM4.01same as prediction
ELFCDLM1.55same as prediction
Cola-DLMCDLM1.37same as prediction
  • The discrete denoisers revisit about as often as the CDLMs, so the phenomenon itself is shared, as you say.
  • Almost none of it reaches the next step, because the sampler keeps only the sampled tokens.
  • Likewise, in the discrete denoisers 15–22% of the probability movement happens after the top-1 token has become final, but in the distribution from which their samplers draw the next state, at most 2% does.
  • Both results hold at 128 sampling steps.

Refinement beneath a stable token is therefore discarded by a discrete sampler and reused by a CDLM; this is what is specific to CDLMs, and why the dynamics beneath the top-1 readout matter in them.

Relation to existing studies of discrete diffusion. The table compares them on five points.

StudyWhen the decoded token stops changingFull predicted distributionRevision beneath a stable top-1 tokenChanges the sampler
Remasking (ReMDM, ReMeDi, CORE)✗✗✗✓
Early convergence (LEAP)✓✗✗✓
Intermediate answers (Time Is a Feature)✗✗✗✓
Temporal emergence (Lu, 2026)✓✗✗✗
This paper✓✓✓✓
  • Existing studies follow the decoded token or a scalar summary of one distribution, such as confidence (lines 143–148), so they record when a token stops changing; none compares a distribution with its own earlier state, which is what revisitation measures.
  • This matters in practice: 75–98% of the windows our detector flags contain no top-1 change (Appendix F.2), so a token-level record would miss them.
  • We build on these studies rather than replace them: we adopt the commitment measure of Lu (2026) and add revisitation beneath it.
  • For a CDLM, fixing a token once its top-1 stabilizes, as early-convergence detection does in discrete samplers, would also cut off the 4–27% of probability movement that follows stabilization and that the sampler carries forward.

We also revised our draft. Section 1 now states that discrete samplers also overwrite tokens and their denoisers revisit, and what is specific to CDLMs; Section 2 contrasts our measurement with token-level analyses and cites UDLM, DUO, and GIDD; Section 4.1 summarizes the discrete reference.

  • UDLM. Schiff, Y., Sahoo, S. S., Phung, H., Wang, G., Boshar, S., Dalla-torre, H., de Almeida, B. P., Rush, A., Pierrot, T., & Kuleshov, V. (2025). Simple guidance mechanisms for discrete diffusion models. In International Conference on Learning Representations.
  • GIDD. von Rütte, D., Fluri, J., Ding, Y., Orvieto, A., Schölkopf, B., & Hofmann, T. (2025). Generalized interpolating discrete diffusion. In International Conference on Machine Learning.
Reviewer_M1rMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies prediction dynamics in continuous diffusion language models through three modes: exploration, revisitation, and commitment. Across LangFlow, Plaid, ELF, and Cola-DLM, it measures distributional changes after top-1 stabilization and candidate competition in certified contests. Context interventions in LangFlow and ELF examine the influence of surrounding predictions. It proposes velocity blending: interval merging skips model evaluations, and velocity damping smooths successive updates. Evaluations include observation-cadence controls and paired sampling comparisons for unconditional and conditional generation. Fixed late merging saves about a quarter of sampling time while preserving native outputs better than early merging. Damping has less consistent effects on revisitation and generation quality.

Critical strengths

1. Motivation & Usefulness

  • Understanding each sampling step's contribution is important. Continued distributional change after top-1 stabilization distinguishes token stability from the end of refinement (Table 5).
  • Late merging preserves native outputs better than early merging at the same model-call budget. For ELF-XSum, Tail changes 1.17% of token slots versus Head's 15.72% (Table 1). Both use 49 evaluations, with measured runtime near 0.75× native (Table 10).

2. Related Works & Novelty

  • Distribution-level revisitation usefully extends token-change tracking. It captures returns toward earlier predictions even when the leading token remains unchanged.
  • The certified-contest analysis identifies recurring rivals in LangFlow, Plaid, and ELF. Context interventions in LangFlow and ELF show that surrounding prediction memory affects subsequent revision direction.

3. Experiments & Robustness

  • The experiments cover four CDLMs with different sampling mechanisms. Matched-cadence comparisons help separate changes in sampling budget from changes in observation frequency.
  • Independent confirmation panels and reported online observation costs make the practical efficiency comparisons more informative.

4. Presentation

The paper is generally easy to follow. Figures explain the prediction modes, and mixed damping results are clearly reported.

Critical weaknesses

The empirical characterization and mixed intervention results are useful. I lean toward weak acceptance. My concerns are the robustness and interpretation of the measured dynamics.

1. Methodology

  1. The detector uses W = 4, an excursion floor of 10^-3, and a relative-recovery threshold of 1/3. The cadence controls and large-contest analysis are useful, but do not establish overall prevalence sensitivity to these choices. Relative recovery can be large for small eligible excursions, and longer windows can raise the score (Appendix C.2, lines 1255–1263). Higher floors and several window spans within each model would strengthen the prevalence estimates. Cross-model rates should remain tied to the reported configurations and readout channels.
  2. Context interventions support the narrower claim that surrounding prediction memory redirects revisions at detected revisiting positions. All six intervals for next-window revisitation changes include zero (Appendix D.5, lines 1494–1496). This does not establish the absence of an effect, but leaves a minor gap to the introduction's source hypothesis (lines 79–88). Matched non-revisiting controls or interventions before episode onset measuring later returns could distinguish that hypothesis from general context sensitivity.

2. Experiments & Robustness

  1. Appendix B.1 (lines 1069–1072) reports only 4–39 distinct tokens in LangFlow's 128-token outputs. The paper appropriately limits its claims to this configuration. Testing at the published 1,024-token OWT length would help establish whether the candidate-competition and feedback findings extend beyond this highly repetitive setting.
  2. Section 5.2 (lines 432–434) explains Plaid's high token mismatch through its larger late exploration-plus-revisitation share. Tail changes 61.51% of native token slots, versus Head's 90.14% (Table 1). However, the grids do not preserve the same stochastic noise path (Appendix E.2, lines 1532–1535). Changed noise may also contribute to native mismatch; its contribution is not measured. A coupled-noise comparison or narrower attribution would strengthen this explanation. This concern leaves the reported timing and quality results intact.

3. Presentation & Reproducibility

  1. Table 1 presents ΔO across different position sets, time windows, and panels. Appendix E.4 discloses selected positions for pulses and all generated positions for schedules (lines 1701–1702). Some combined-policy results use only 16 or 32 runs and different windows (lines 1713–1727). Paired intervals are reported in the appendix, but Table 1 needs scope labels and references to those intervals. Its entries are not directly comparable measures of revisitation reduction.
Questions for authors
  • How stable are prevalence estimates under higher excursion floors and several window spans within each model?
  • Can matched controls or interventions before episode onset connect the supported revision-direction effect to a source of revisitation, beyond general context sensitivity? Do LangFlow's findings persist at the published generation length?
  • Does Plaid's high mismatch under late merging persist when noise paths are coupled across grids? This would help distinguish omitted-update effects from changes in sampling noise.
5 comments

Thank you for the careful reading and the insightful questions. We have updated the paper, and line numbers below refer to the revised version. We ran the checks you asked for: the detector configurations, the matched control, the 1,024-token LangFlow panel, and a noise coupling for Plaid. The findings hold in each.

W1.1 and Q1: Prevalence depends on the detector configuration

This is a fair concern, and one we were aware of while writing the paper: before fixing the detector, we ran many such configurations on development runs, and we chose W=4W=4, ϵ=10−3\epsilon=10^{-3}, and O>1/3O>1/3 for the following reasons.

  • The window is short enough for the shortest schedule yet long enough to hold a departure and a return.
  • The floor removes excursions too small to matter, including those with a large relative recovery (lines 213–222).
  • The threshold requires the distribution to come back at least a third of the way toward the earlier state. Throughout the paper we use this single configuration, so that every comparison within a model uses the same detector.

To address your concern, we have now run the detector under these alternative configurations on the 64 runs per model; the table below gives representative settings.

Window detection rate (%)LangFlowPlaidELFCola-DLM (block 0)
Paper (W=4W=4, ϵ=10−3\epsilon=10^{-3}, O>1/3O>1/3)6.134.011.551.37
Threshold O>0.2O>0.29.328.422.071.93
Threshold O>0.7O>0.71.000.460.590.41
Window W=3W=34.162.070.710.50
Window W=8W=89.159.135.124.55
Floor ϵ=10−2\epsilon=10^{-2}2.833.511.081.06
Absolute recovery ≥0.01\geq 0.011.613.151.000.94
Absolute recovery ≥0.05\geq 0.050.301.600.690.72
Strict (E>0.03E>0.03, O>0.7O>0.7)0.220.250.350.31

Changing these settings does not produce an abrupt change in any of the four models: as you note, longer windows contain more references and raise the rate, and each setting moves the rate in the same direction in all four, and revisitation remains in every model under all 32 combinations we tested (0.06–19.0% of windows). The configuration therefore changes how much revisitation is counted, but none of the paper's conclusions is overturned or reversed. As you suggest, we compare levels across models only under the stated configuration and readout channel; the conclusions rest on the presence and structure of revisitation. New Appendix F.1 reports all of them.

W1.2 and Q2: Context interventions and the source hypothesis

You are right that the submitted interventions support the narrower claim: holding the memory of the other positions changes the direction in which a revisiting position revises (Section 4.3), but they do not show that this dependence produces revisitation. On the six intervals you cite (submitted Appendix D.5): they measure revisitation in the next window, which mostly covers steps after the return has begun and is shared by all branches, so it can hardly register a change; the primary outcome, the direction of revision, is significant in both models. To test the source hypothesis more directly, we ran the matched non-revisiting control you suggest and one further experiment.

Matched control (new; Appendix F.5). We perturb only the prediction memory of one committed position and compare the predictions at all other positions with a sham that leaves this memory unchanged. The other positions change, even far from the perturbed one: at positions 17 or more tokens away, by 9.93 [9.25,10.64]9.93\ [9.25,10.64] (ELF) and 3.46 [3.04,3.92]3.46\ [3.04,3.92] (LangFlow) TV$\times10^{3}$ above sham. The response is the same whether the perturbed position is revisiting or is a matched position that is not, so this dependence is general: every position depends on the others through their memory.

Prediction memory sets how much revisitation occurs (new; Appendix F.6). When ELF's prediction memory is two steps out of date, its window rate rises from 1.55% to 5.55%; when the memory is switched off, it rises to 2.32%.

Together, these results support the source hypothesis in the form the data allow: every position depends on the others through prediction memory, and the state of this memory changes how much revisitation occurs.

W2.1 and Q2: LangFlow at 128 tokens

In the paper we ran LangFlow at 128 tokens, the output length of our unconditional panels; we have now also run it at 1,024 tokens, where its outputs are no longer repetitive and the findings of Sections 4.1 and 4.2 hold.

LangFlow, native sampling128 tokens1,024 tokens
Distinct tokens per sample22.0399.5
Repeated 4-grams (%)63.711.9
Window rate, default (%)6.1319.4619.46
Window rate, strict (%)0.221.781.78
Movement after top-1 stabilization (%)26.641.941.9
Contests with one dominant rival (%)86.084.684.6
Same rival recurs (%)76.176.576.5

Single-rival concentration and rival recurrence are nearly unchanged, revisitation remains under both the default and the strict detector, and 41.9% of probability movement follows top-1 stabilization. The panels differ in length and number of calls, so we compare presence and structure, not levels. The Section 4 findings also do not rest on LangFlow alone: candidate concentration and rival recurrence hold in Plaid and ELF, and the cross-position effect (W1.2) holds in ELF. In a new merging experiment at this length, merging also behaves as in Table 1: on 32 runs at 768 of 1,024 calls, Tail changes 2.6% of tokens, against 51.3% for Head and 40.5% for Uniform. The revision reports LangFlow at both lengths. We are also running the cross-position (feedback) experiment at 1,024 tokens, and will try our best to report it before the end of the discussion period.

W2.2 and Q3: Plaid's mismatch versus its noise path

Thank you for this question; you are right that the noise path contributes, and we now measure how much.

Explanation in the submission. Section 5.2 attributed Plaid's large mismatch to when its positions commit (lines 472–474): many of Plaid's positions are still undecided late in sampling (43.0% at 90% of sampling, against at most 9.2% in the other models), so a late change, to the updates or to the noise, can alter its output.

New experiments (Appendix F.7).

  • Coupled noise. An exact coupling is not available: it requires the noise of two fine steps to combine into the noise of the merged step, and on Plaid's sampling schedule they do not (in our check, 93 of 96 merged intervals fail the variance check at tolerance 10−610^{-6}). We therefore use an approximate coupling, in which each merged step reuses the normalized combination of the two fine-step noises it replaces, keeping the marginal of the merged step. With this coupling, Tail changes 35.9 [32.1,39.6]35.9\ [32.1,39.6]% of tokens instead of 61.5%, and Head 54.5 [49.7,59.2]54.5\ [49.7,59.2]% instead of 90.1%: the noise path contributes, as you suspected, and the gap between late and early merging remains. The remaining changes fall on positions that have not committed when merging begins (42.2% of them change under Tail, against 3.3% of committed positions), and merging only the last quarter or eighth of the calls lowers the change to 11.9% and 4.9%.
  • Noise only, no merging. Keeping Plaid's sampling schedule and initial state and redrawing only the noise of the second half of sampling, the part that Tail changes, changes 75.3 [73.6,77.0]75.3\ [73.6,77.0]% of tokens (the 64 documents of Table 1, two redraws each); a full redraw changes 99.1%. The first half of Plaid's sampling thus fixes only about a quarter of its tokens.
  • Later starting points. We move the start of the noise replacement and of the Tail merge later in sampling:
Start call kk (of 128)Positions still uncommitted at kkTokens changed, noise replaced from kkOf these, among committed positionsTokens changed, Tail merge from kk
6482.4%75.3%17.7%61.5%
9649.0%46.1%10.8%31.5%
11229.4%28.5%7.5%17.6%

The share of tokens that change tracks the share of positions not yet committed, committed positions keep their token far more often, and merging later behaves the same way. In the native runs, only 56% of Plaid's late updates act on committed positions, against 89–95% in the other three models (Section 4.1).

Where positions have committed by then, in LangFlow, ELF, ELF-XSum, and German-to-English, the same late merge changes only 0.3–5.1% of tokens, as the submitted Table 1 already shows. In short, the noise path explains part of Plaid's mismatch, and the rest comes from positions that have not yet committed when merging begins.

W3: Table 1 scope labels

Thank you for this suggestion: Table 1, which collects separate experiments in one grid to give an overview, now carries the labels and references you ask for.

  • Scope labels. Rows that apply a one-call pulse are marked †; their ΔO\Delta O covers the selected positions in the first window after the pulse, whereas unmarked rows cover all generated positions. Section 5.2 states that ΔO\Delta O is comparable within each group and not between them.
  • References to the intervals. A note under the table points to the notes on the sampling-control tables in Appendix E.4, which now open with the scope of each row group and list where the paired intervals are reported.
  • Sample sizes. For the combined policies, Table 12 (Appendix E.4) reports the 64-run extensions next to the primary estimates. For the merge-only rows, we are running the full 64-run panels with independent checks, and will try our best to report them before the end of the discussion period.

Thank you again. We believe these additions address each of your concerns, and we hope they strengthen your assessment of the paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers and the AC

We thank the reviewers and the AC for their time and constructive feedback. All changes in the revised PDF are marked in blue.

Summary

As the reviewers' words, the paper "analyzes the prediction behaviors of several existing continuous diffusion language models" and categorizes its predictions "into three classes: exploration, revisitation, and commitment," with "quantitative measures to distinguish these three types of predictions" (rS3h). "The results show that revisitation is prevalent across these continuous diffusion language models" (rS3h): "token probability distributions can repeatedly move away from and return toward earlier predictions, even after the top-1 token stabilises" (KJbq). The paper "proposes two hypotheses to explain revisitation: candidate competition and the influence of updates at other positions" (rS3h), and in our experiments in sampling, "fixed late merging saves about a quarter of sampling time while preserving native outputs better than early merging" (M1rM).

1. Strengths recognized by the reviewers

  • An important and underexplored question. Understanding what each sampling step contributes "is important" (M1rM), the paper "investigates a timely and relatively underexplored problem" (KJbq), and "the differences between continuous and discrete diffusion make this worth studying" (M1rM).
  • A framework beyond top-1 stability. The three modes offer "a structured way to examine prediction changes beyond top-1 token stability" (KJbq).
  • Diverse experiments. "Diverse experiments and empirical observations were conducted on four existing CDLMs" (KJbq).
  • Practical relevance. The findings "might help develop better sampling algorithms" (rS3h).

2. Concerns addressed during the discussion

M1rM writes that "most of my initial concerns are now addressed," and KJbq that the added experiments "strengthen the empirical analysis." The new analyses address each concern:

  • Robustness of revisitation (M1rM W1.1, KJbq W2). Revisitation appears in every model under all detector settings, and no single-step measure captures it.
  • Cross-position dependence (M1rM W1.2). Perturbing the memory of one committed position changes predictions far from it.
  • Where merging is safe (M1rM W2.2, KJbq W3). Merging changes tokens mainly where positions have not yet committed, which explains Plaid's exception.
  • Quality on new data (rS3h W3, KJbq Q3). On new summarization and translation test sets, late merging keeps native quality with a quarter fewer steps.
  • What is specific to CDLMs (rS3h W2, M1rM). Discrete diffusion models also revisit, but only a CDLM passes the full predicted distribution to its next update.

3. Revisions following the reviewers' suggestions

  • One line of argument (KJbq). The paper now follows one chain, from research question to interpretation.
  • Clearer definitions and positioning (rS3h, M1rM). Plain-language definitions, and an Introduction that states what the analysis adds beyond discrete diffusion.
  • New theory (M1rM, KJbq). Appendix C shows when revisitation cannot change the token and why merging is safe at committed positions.

We thank the reviewers again for helping us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.