acceptodds
Under review as a conference paper at ICLR 2027

Clock Diffusion: Efficient Semi-Autoregressive Continuous Diffusion Language Models

11 views

Abstract

Recent works on continuous diffusion for discrete data have demonstrated performance on par with comparable discrete diffusion models. However, these continuous counterparts lack key features that are essential to practical use as language models, namely variable-length generation and support for a key-value cache, and they still lag behind the frontier of autoregressive and discrete diffusion quality. In this work, we address these limitations. We do so by introducing a model parameterization that uses position-dependent noise schedules to define semi-autoregressive (SAR) continuous diffusion language models (DLMs). Together with efficient training and sampling algorithms, we call this framework **Clock Diffusion**, and we present two special cases of our method: block and sliding window generation. We then define **ClockDLMs**, a family of Gaussian DLMs based on sliding window Clock Diffusion that attain state-of-the-art diffusion likelihood bounds on OpenWebText, even beating the performant block SAR discrete diffusion models. ClockDLMs trained on TinyGSM also substantially outperform continuous baselines on the GSM8K benchmark and match and exceed comparable SAR discrete diffusion models. Finally, building on our parameterization, we propose more efficient samplers that we dub **Cache Grab**, which adapt techniques from accelerated inference in discrete diffusion, such as committing tokens whose probabilities exceed a confidence threshold and self-speculative decoding, further improving our models' quality and efficiency.

Primary area:
generative models
Keywords:
Continuous diffusion for discrete data, Semi-Autoregressive, KV Cache

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_qUKkOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Clock Diffusion, a semi-autoregressive framework for continuous diffusion language models based on position-dependent noise schedules. The authors further introduce ClockDLMs and Cache Grab samplers to improve generation quality and inference efficiency. Experiments across multiple benchmarks demonstrate the effectiveness of the proposed framework.

Critical strengths

The paper presents a comprehensive framework for continuous diffusion language models, covering model formulation, efficient training, and accelerated inference. The experimental evaluation is also extensive. The strong results across different benchmarks demonstrate the effectiveness of the proposed framework.

Critical weaknesses
  1. Limited evaluation at larger scales. The experiments focus on relatively small models. It remains unclear whether the proposed framework can maintain its advantages when scaled to larger language models.
  2. Limited practical efficiency evaluation. The paper mainly evaluates inference efficiency using NFE, without sufficient wall-clock latency and throughput measurements. These results would help establish whether the reported computational savings translate into practical speedups.
Questions for authors

It would be valuable to provide a more comprehensive evaluation of practical inference efficiency, including wall-clock latency and throughput. In particular, a comparison with recent discrete or hybrid diffusion approaches such as TiDAR would help clarify the performance and efficiency trade-offs between continuous and discrete diffusion language models. It is okay if continuous diffusion does not outperform these methods, as it is still an emerging research direction. If computational resources are limited, it is also acceptable not to conduct larger scale experiments.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qUKk

We thank the reviewer for their positive comments and for their constructive feedback. Below we address the specific issues raised. Please note that all paper revisions discussed below have been incorporated into an updated PDF posted here and are marked in blue font.


W1: Limited evaluation at larger scales

Thank you for this suggestion. To evaluate our method at larger scales, we used the released 0.5B and 7B checkpoints of Lafa, from the recent work “Large Flow Language Models” (Chavan et al., 2026). Lafa models are block semi-autoregressive (SAR) continuous flow language models, adapted from Qwen2.5-Instruct, with a block size of 4. From each released checkpoint, we ran two fine-tuning (FT) runs that differ only in the clock and training mask:

  • Block FT: Lafa's original block SAR clocks and training mask.
  • Sliding-window FT: our sliding-window clocks and training mask.

Sliding-window FT matches or outperforms block FT on all six model/benchmark pairs, with gains of up to 9.1 points (7B, IFEval), indicating that the advantage of our sliding-window clocks persists at the 7B scale and carries over to a continuous language model beyond our ClockDLMs.

0.5B Lafa Model

HumanEval (↑\uparrow)GSM8K (↑\uparrow)IFEval (↑\uparrow)
Lafa (Chavan et al., 2026)21.325.043.6
+ Block FT27.4426.0043.07
+ Sliding-window FT27.4430.7147.69

7B Lafa Model

HumanEval (↑\uparrow)GSM8K (↑\uparrow)IFEval (↑\uparrow)
Lafa (Chavan et al., 2026)63.469.560.1
+ Block FT68.2969.6066.73
+ Sliding-window FT73.7870.9675.79

Experimental setup notes:

  • Both FT runs train for 5k steps on Dolci-Instruct-SFT, the dataset used to train Lafa, with the noise schedule frozen and learning rates 5× lower than Lafa's (2×10−52\times10^{-5} for 0.5B and 2×10−62\times10^{-6} for 7B).
  • Both FT variants are decoded with block/window size 4 and 4 Euler steps per block/window
  • All evaluations use this block/window size and 4 euler steps of denoising per block/window.
  • The first row of each table is taken from Table 5 of Chavan et al. (2026) (flow decoding, block size 4, 4 Euler steps, prefix guidance g=8g=8, inference noise scale 0.1).

W2: Report additional throughput metrics other than average NFEs/sequence

This suggestion is well taken. In the revised appendix, we added versions of Figures 4, 5, and 6 that use tokens per forward (↑\uparrow) and tokens per second (↑\uparrow) on the horizontal axes, in place of the NFEs/seq (↓\downarrow) used in the main-text figures. These are Figures 9, 10, and 11 in the new Appendix G.4 (“Additional Throughput Metrics”), which also defines both metrics and the measurement setup (batch size 1, single H100, bfloat16).

Against the diffusion baselines, the trends of the original figures carry over to both metrics.

For example, at S=32S=32 and N=4N=4, ClockDLM reaches 53.8% accuracy with 3.9× fewer NFEs than BD3LM at comparable accuracy (52.9%; Figure 4), which translates into 2.6× higher tokens per second (529 vs. 202; Figure 9).

We note that the wall-clock gains are smaller than the NFE savings, however, because each ClockDLM decoding step (model forward plus sampler update) currently takes about 12 ms, vs. about 6 ms for AR and 8–9 ms for BD3LM. As a result, at S=32S=32 and N=8N=8, for example, ClockDLM uses 3.4× fewer NFEs than AR but is only 1.7× faster in tokens per second (270 vs. 158), at 3.8 points lower accuracy (60.5% vs. 64.3%). We expect optimized kernels and inference serving to reduce the per-step overhead and widen these gains.

The Cache Grab samplers translate their NFE savings into wall-clock speedups over standard ClockDLM decoding (Figure 10). Cache Grab-Confidence (η=0.7\eta=0.7) increases tokens per second by 2.9–4.2×, with accuracy changes between −2.4 and +0.4 points. Cache Grab-SSD, with the jointly trained models of Appendix G.3, increases tokens per second by 4.6–5.4× while also improving accuracy, and every Cache Grab-SSD configuration exceeds AR in both accuracy and tokens per second (1.9-2.3×).


Q1: How does ClockDLM compare to works, such as TiDAR in terms of efficiency-performance trade-offs?

Thank you for suggesting this comparison. For a controlled comparison with TiDAR (Liu et al., 2025), we are currently conducting the following experiment:

  • Initializing TiDAR from our AR baseline trained on TinyGSM, so that it shares our training data and model scale
  • Fine-tuning for an additional 25k steps following the TiDAR training recipe.
  • We will report GSM8K accuracy vs. tokens / forward and tokens / second for TiDAR models and for ClockDLMs with Cache Grab-SSD

We will post the results in this thread before the end of the discussion period as soon as they are available. Please let us know if this intended experiment matches your expectations.

Thank you for the detailed response and additional experiments,and I have raised my score accordingly.

AuthorsOpenReview ↗

Follow up: TiDAR experiment results

As promised in our response to Q1, here are the results of the controlled comparison with TiDAR (Liu et al., 2025).

ModelSSGSM8K acc. (%) (↑\uparrow)Tokens/forward (↑\uparrow)Tokens/s (↑\uparrow)ms/forward (↓\downarrow)
AR baseline–64.30.991586.3
TiDAR461.63.565097.0
TiDAR862.05.848057.3
TiDAR1660.07.5510097.5
TiDAR3260.07.058868.0
ClockDLM + Cache Grab-SSD1665.94.0534711.7
ClockDLM + Cache Grab-SSD3267.64.3937111.8
ClockDLM + Cache Grab-SSD6464.83.6730212.2
  • Accuracy: Every ClockDLM + Cache Grab-SSD configuration (64.8–67.6%) is more accurate than every TiDAR model (60.0–62.0%) and than the AR baseline, whereas TiDAR falls 2-4 points below the AR model it was initialized from.
  • Throughput: TiDAR is currently 1.4-2.7× faster in tokens per second than our fastest Cache Grab-SSD configuration (S=32S=32), for two reasons. First, its forward passes are cheaper (7.0-8.0 ms vs. 11.7-12.2 ms), consistent with the per-step cost of our current implementation discussed in W2. Second, at S≥8S \ge 8, TiDAR produces more tokens per forward (5.8-7.6 vs. 3.7-4.4) because it drafts and verifies in a single forward pass, whereas Cache Grab-SSD uses separate passes. At matched S=32S=32, these two factors contribute 1.5× and 1.6×, respectively (2.4× overall).

Thank you for your continued engagement with our work and for raising your score.

Sincerely,

The Authors

Reviewer_4LD8OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Clock Diffusion, a modern framework for semi-autoregressive continuous diffusion language models. Building on existing approaches to blockwise and sliding-window diffusion, the authors adapt these ideas to modern continuous diffusion language models, enabling variable-length generation and efficient KV caching. They introduce sampling methods from discrete diffusion that lead to faster inference and performance improvements. The authors validate their method on OpenWebText, TinyGSM/GSM8K and a summarization task - a standard set of benchmarks - with state-of-the-art empirical performance.

Critical strengths

Bringing sliding window diffusion to modern-day continuous diffusion language models is a valuable contribution. It brings practical benefits and strong empirical performance as seen on the benchmarks. The paper is clear, comprehensive and well-written.

Critical weaknesses

I haven't identified any critical weaknesses that support rejection.

Questions for authors
  • I wonder about the reasoning of the authors to not include gen. PPL as a metric. I understand it is a flawed metric, yet it is reported by most methods so I think the comparison would be worthwhile.
  • I noticed that OWT performance of block diffusion seems relatively insensitive to block size, is there anything we might infer from that?
  • How should one select the block/window size in practice? A compute–quality comparison across sizes, and ideally between blockwise and sliding-window generation, would be very informative.
  • Could the authors elaborate on when and why sliding-window diffusion is preferable to blockwise diffusion? While sliding-window generation often performs better, the improvements do not appear entirely consistent across tasks and window sizes. Understanding whether its benefit comes primarily from staggered noise levels, more progressive left-to-right conditioning, or other factors would help clarify the contribution of the proposed scheduling strategy.
  • The OpenWebText MAUVE results deteriorate sharply at low NFE budgets, is this because of a minimum number of steps that are required within each window/block? This relates back to the earlier question: if SAR has a lower number of required steps times for reasonable generative performance than synchronous times that would be a weakness worth making explicit.

Additional feedback

While the authors are careful to identify where they build on existing work and the related works section is thorough, I think the introduction could more explicitly acknowledge existing SAR continuous DLMs and asynchronous schedules.

Set Diffusion is compared against in the benchmarks and introduces some similar ideas for discrete diffusion, I'd argue that this work should also be discussed in the Related Work section.

The main text could explain more explicitly that sliding-window training uses sorted random local times, rather than only the evenly spaced deterministic schedule, to support Cache Grab's variable-stride sampling (if I'm understanding this correctly).

6 comments
AuthorsOpenReview ↗

Response to Reviewer 4LD8 (1/3)

We thank the reviewer for their positive assessment of our work and for the valuable questions and feedback, which we address below. Please note that all paper revisions discussed below have been incorporated into an updated PDF posted here and are marked in blue font.


Q1: Report Gen PPL for OWT models

In the table below we provide Gen PPL under GPT-2 Large (and entropy) results for models trained on OWT. The entropy results are needed, because as noted in prior works, Zheng et al., 2025 and Pynadath et al., 2026, Gen PPL is not meaningfully interpretable in isolation.

At each decoding budget, ClockDLM has higher Gen PPL but also higher entropy. At matched entropy, as recommended by Pynadath et al. (2026), ClockDLM’s Gen PPL is comparable to RePlaid’s (38.9 vs. 38.8 at entropy ≈5.47; 28.3 vs. 29.6 at ≈5.38; 23.0 vs. 24.6 at ≈5.33)..

NFEs per seq. (ClockDLM / RePlaid)ClockDLM Gen PPL (↓\downarrow)ClockDLM Entropy (↑\uparrow)RePlaid Gen PPL (↓\downarrow)RePlaid Entropy (↑\uparrow)
67 / 651080.55.9157.65.56
135 / 129430.15.8138.85.47
271 / 257115.55.6129.65.38
543 / 51353.35.4924.65.33
1087 / 102538.95.4621.45.28
2174 / 204928.35.3819.25.22
4348 / 409723.05.3317.35.18

ClockDLM (S=64S=64) uses N=4,8,…,256N = 4, 8, \ldots, 256 denoising steps per token and RePlaid uses T=64,128,…,4096T = 64, 128, \ldots, 4096 steps; these are the runs in Figure 8, with sampling settings as in Appendix G.1.


Q2: What conclusions can be drawn from relative insensitivity to block size for B-ClockDLMs on OWT PPL?

We agree that block-model perplexity varies relatively little across the tested sizes, although as with sliding window ClockDLMs and with BD3LM, smaller blocks consistently improve PPL. Given that we also jointly learn the noise schedule for each model and that the training dynamics differ per SS, it is difficult to draw strong conclusions about the relatively less steep drop in PPL for B-ClockDLMs compared to other semiAR models.

AuthorsOpenReview ↗

Response to Reviewer 4LD8 (2/3)

Q3: How should block/window size be selected, given the quality-compute trade-off?

Thank you for this question. Below we summarize results across window sizes SS and per-token budgets NN, for both block and sliding-window (SW) generation. The key takeaways are as follows:

  • Default to sliding window. It is generally more accurate than block (this holds for all configurations where N≥4N \ge 4) and uses fewer NFEs/seq (see Table A below).
  • Give each token at least 4 denoising steps. At N=2N=2, sliding-window accuracy drops sharply (e.g., from 53.8% to 19.4% at S=32S=32), and this is the only regime where block is more accurate (Table A). If the budget is tight, use a larger window rather than fewer steps per token (see Table B below).
  • Shrink the window as the budget grows. Smaller windows add sequential conditioning and win once the budget is large enough; larger windows win at small budgets because they give each token more denoising steps for the same NFEs/seq. On GSM8K, the most accurate choices are S=64S=64 at ≈15 NFEs/seq, S=32S=32 at ≈25–50, S=8S=8 at ≈80, and S=4S=4 at ≈160 (Table B).

Table A: Block vs. sliding window (SW) on GSM8K, for varying window size SS and per-token budget N≤SN \le S

SSNNBlock acc. (%)Block NFEs/seqSW acc. (%)SW NFEs/seq
4263.0120.960.381.7
4463.8200.965.7161.5
8256.661.946.242.3
8459.7102.661.382.7
8861.5183.963.8165.0
16245.033.032.423.2
16453.554.357.843.8
16857.596.759.186.1
161658.6181.360.0172.6
32229.018.719.414.5
32442.629.853.824.0
32848.052.860.547.2
321651.098.459.194.4
323254.3190.159.6188.1
64215.611.33.910.9
64427.617.935.614.8
64836.531.449.927.8
641642.158.354.055.2
643246.9112.954.7109.8
646449.1220.155.6219.1

NN denotes denoising steps per token. Bold marks the higher accuracy and the lower NFEs/seq in each row. All configurations use diffusion-only training and the standard sampler (no Cache Grab).

Table B: Sliding-window GSM8K accuracy (%), with NFEs/seq in parentheses, for varying window size SS, grouped by approximate budget

Approx. NFEs/seqN/SN/SS=4S=4S=8S=8S=16S=16S=32S=32S=64S=64
≈151/16–––19.4 (14.5)35.6 (14.8)
≈23–281/8––32.4 (23.2)53.8 (24.0)49.9 (27.8)
≈42–551/4–46.2 (42.3)57.8 (43.8)60.5 (47.2)54.0 (55.2)
≈82–1101/260.3 (81.7)61.3 (82.7)59.1 (86.1)59.1 (94.4)54.7 (109.8)
≈160–220165.7 (161.5)63.8 (165.0)60.0 (172.6)59.6 (188.1)55.6 (219.1)

Each row groups configurations with the same ratio of denoising steps per token to window size (N/SN/S), which gives approximately matched NFEs/seq; within a row, NFEs/seq grow with SS by up to ≈35%. Bold marks the most accurate configuration in each row. Same training and sampler as Table A.

AuthorsOpenReview ↗

Response to Reviewer 4LD8 (3/3)

Q4: What is driving sliding window improvements over block counterparts and in what settings are these improvements present?

As discussed above, sliding window is generally preferable to the block parameterization, and our main hypothesis is that, as the reviewer suggested, the gains are coming from the “more progressive left-to-right conditioning”. That is, within any given active set of tokens, the sliding window has a stronger left-to-right bias than block versions. The importance of this inductive bias is underscored by the AR results on GSM8K and by the fact that as SS decreases, i.e., models become more like AR, performance improves.

Table 1 of the paper (OpenWebText perplexity) for S≤16S \le 16, and the fact that sliding-window ClockDLMs outperform their block counterparts on GSM8K at every window size once N≥4N \ge 4 (Table A), support this hypothesis. The advantage is not uniform, however: at S=64S=64 the block variant is slightly better on OpenWebText perplexity (20.9 vs. 21.1) and on CNN/DailyMail (34.3 vs. 34.0 ROUGE-1). The GSM8K N=2N=2 case, where block is ahead at every window size (Table A), is consistent with our hypothesis: with only two denoising steps per token, the staggered noise levels within a window become coarse and resemble the noising pattern of a block SAR model.


Q5: Why do MAUVE results deteriorate in the low NFE regime?

As discussed in Appendix G.1, the deterioration stems from the small per-token budgets in this regime, and the data show a clear threshold: for ClockDLM (S=64S=64), MAUVE jumps from 0.01 at N=16N=16 denoising steps per token (271 NFEs/seq) to 0.89 at N=32N=32 (543 NFEs/seq). RePlaid shows a similar threshold between T=64T=64 (0.37) and T=128T=128 (0.79) steps. Per token, ClockDLM therefore needs fewer steps than RePlaid, but because a 1024-token sequence spans 16 windows of S=64S=64, each step per token costs ~17 NFEs/seq. We hypothesize that open-ended generation is more sensitive to this than GSM8K because it lacks a conditioning prompt, and we have made this limitation explicit in the Limitations paragraph of Section 6 (see the updated PDF, in blue).

Additional comments

More clearly acknowledge existing SAR continuous DLMs and asynchronous schedules in the introduction.

This feedback is well taken. We have revised our manuscript to more explicitly highlight ideas discussed in the “Related works” section in the introduction as well (see updated PDF in blue font).

Discuss Set Diffusion in Related works section

Thank you for this suggestion. We have included the following in our revised manuscript:

Recently, Arriola et al. (2026) extended block diffusion to a more flexible set formulation. Using a noising schedule biased toward left-to-right decoding, they derive a sliding window formulation that can be seen as a masked diffusion analog of our continuous method.

Move explanation of the timestep sampling for sliding window training from appendix to main text

We agree that this is an important implementation detail. We moved this detail to the main paper of our revised manuscript (see the updated Cache Grab section in blue font).


References

Pynadath, Patrick, Jiaxin Shi, and Ruqi Zhang. "Generative frontiers: Why evaluation matters for diffusion language models." arXiv preprint arXiv:2604.02718 (2026).

Zheng, Kaiwen, et al. "Masked diffusion models are secretly time-agnostic masked models and exploit inaccurate categorical sampling." International Conference on Learning Representations. Vol. 2025. 2025.

Reviewer_KmUxOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce Clock Diffusion, a semi-autoregressive continuous diffusion framework for language. Position-dependent local noise schedules let tokens denoise from left to right, enabling variable-length generation and key-value (KV) caching.

Combining sliding-window clocks with Gaussian noise on token embeddings, the authors introduce ClockDLMs. On OpenWebText, they report the best diffusion likelihood bound in their comparison, including lower perplexity than the cited block discrete model BD3LM. This perplexity remains higher than their autoregressive baseline.

To accelerate inference, the paper introduces Cache Grab samplers, which adapt confidence-threshold early commitment and self-speculative decoding. The paper reports that the SSD variant exceeds its autoregressive baseline on GSM8K. That result uses a separately trained model with a joint autoregressive and diffusion loss, whereas confidence-based early commitment does not exceed the autoregressive baseline.

Critical strengths

The paper targets a concrete gap: recent full-sequence continuous diffusion language models do not support variable-length generation or a KV cache. Clock Diffusion is built around that limitation.

The framework is worked out end to end. Local clocks cover both block and sliding-window generation, with a variational bound, a vectorized training mask, and a sampler that writes finished tokens into the KV cache. For the sliding window, the cache can be updated without waiting for the whole window.

Cache Grab-Confidence gives a real efficiency gain. On GSM8K at S=32, it cuts function evaluations from about 188 to about 60 while accuracy stays near 60%. The paper also adapts self-speculative decoding, and reports a further reduction in function evaluations for that variant.

Critical weaknesses

The core mechanism is adapted from prior work. Local clocks follow the position-dependent schedules of Diffusion Forcing and Rolling Diffusion, which the paper distinguishes as prior work on continuous data such as video. Block generation, the vectorized 2L training layout, and KV caching follow block discrete diffusion. Gaussian diffusion on token embeddings follows Plaid and RePlaid, and Cache Grab adapts confidence-based commitment and self-speculative decoding. The language bound, sliding-window mask, and cache updates for continuous models are useful, but the framework is compositional.

The broader empirical headlines are not tied to one matched comparison. The abstract says ClockDLMs match and exceed comparable semi-autoregressive discrete models on GSM8K, but in Table 2 the diffusion-only model is below the retrained block-discrete baseline. Figure 1 attributes autoregressive-beating GSM8K accuracy to Cache Grab, whereas that result is the self-speculative setting. Appendix F states that this setting uses a model retrained with a joint autoregressive and diffusion loss. Table 8 also shows that a smaller window, decoded with the standard sampler and without Cache Grab, can exceed the autoregressive baseline. The conclusion’s state-of-the-art claim for downstream benchmarks is similarly broader than the summarization results, where discrete diffusion remains ahead.

Questions for authors

The abstract says ClockDLMs match and exceed comparable semi-autoregressive discrete models on GSM8K. In Table 2, diffusion-only ClockDLM is below the retrained block-discrete baseline. Which models count as comparable, and does this claim include that baseline?

Figure 1 credits Cache Grab with beating the autoregressive baseline on GSM8K. Adding only Cache Grab-Confidence to the diffusion-only model does not beat that baseline. The result that does is Cache Grab-SSD, on a model retrained with a joint autoregressive and diffusion loss (Appendix F). Table 8 also shows that a smaller window can beat the same baseline with the standard sampler and no Cache Grab. What is actually responsible for exceeding the autoregressive baseline?

2 comments
AuthorsOpenReview ↗

Response to Reviewer KmUx (1/2)

We thank the reviewer for recognizing the importance of our contribution to the continuous diffusion for language literature. Below we address the specific issues raised. Please note that all paper revisions discussed below have been incorporated into an updated PDF posted here and are marked in blue font.


W1: Clock Diffusion composes different elements of prior works into a single framework

While our method does indeed draw from several prior works, our paper provides clear extensions over each, which we detail more fully here:

Diffusion Forcing / Rolling Diffusion

We build on their position-dependent noise schedules, with Rolling Diffusion being the closest predecessor to our sliding-window formulation. However, we extend these works in three important respects:

  • Likelihood formulation: Rolling Diffusion states the standard ELBO decomposition but focuses its derivation on the diffusion loss, treating reconstruction as negligible. We explicitly derive the reconstruction contribution for discrete tokens at their respective completion times and establish its continuous-time limit. Diffusion Forcing derives a reconstruction term conditioned on the token’s noisy observation and a recurrent state summarizing preceding observations, as they focus on recurrent neural network (RNN) backbones. We instead derive reconstruction at each token’s completion time, conditioned on its clean prefix and partially noisy subsequent tokens and establish its continuous-time limit.
  • Vectorized training: Rolling Diffusion trains on individual windows. Diffusion Forcing’s main experiments use an RNN. Its transformer extension trains each token on noisy past tokens at independently drawn noise levels, so it is almost never trained on a fully clean prefix. In contrast, we process multiple denoising groups and their respective clean prefixes in one forward pass, obtaining diffusion supervision at every token position in a sequence, while also training the model to condition on clean prefixes.
  • KV caching: We provide a Transformer sampling algorithm that caches completed tokens while the remaining window continues denoising, and fuse cache updates with the subsequent denoising forward.

These developments turn the position-dependent scheduling framework from these prior works into a continuous diffusion language model with sequence-level vectorized training, KV caching, and likelihood-based evaluation.

Block Diffusion

Our local clocks recover BD3LM’s block factorization and 2L2L vectorized training as a special case (with Gaussian rather than masked corruption; Appendix B), and the same formalism yields staggered sliding windows. Our extension to sliding-window ClockDLMs is important, as these outperform our block variant at most GSM8K budgets and on OpenWebText perplexity for S≤16S \le 16.

Plaid / RePlaid

While we instantiate our ClockDLM models using RePlaid as a backbone, this is a design decision, not an integral component of our method. Clock Diffusion is compatible with other continuous diffusion for language paradigms, e.g., those that directly apply Gaussian noise to one-hot embeddings, however we elected to build on RePlaid since it was the current state-of-the-art full sequence continuous diffusion language model.

AuthorsOpenReview ↗

Response to Reviewer KmUx (2/2)

W2/Questions: Clarify the empirical results headlines and which models and comparisons support each claim

Below we provide more explicit details about the evidence of each claim. We have revised the manuscript to make these distinctions explicit (abstract, Figure 1 caption, contribution 4, the Section 4 takeaway, Section 4.2, and the conclusion; all in blue). We thank the reviewer for this useful feedback.

Claim: ClockDLMs match and exceed comparable semi-autoregressive discrete models on GSM8K

“Comparable” refers to the BD3LM and Set Diffusion baselines evaluated at matched block/window sizes and decoding budgets.

At matched block/window sizes, Figure 4 demonstrates that ClockDLMs have higher accuracy at almost all decoding budgets relative to both Block discrete diffusion (BD3LM) and the recent Set Discrete Diffusion, with performance gains of more than 10 accuracy points at low-to-moderate budgets (e.g., +19.6 points over BD3LM at ~50 NFEs/seq for S=32S=32)”.

At the largest budget (N=SN=S), BD3LM remains slightly ahead of diffusion-only ClockDLM (S=32S=32: 61.8% vs. 59.6%; S=64S=64: 58.0% vs. 55.6%), while ClockDLM still clearly outperforms Set Diffusion. Relative to previous continuous diffusion models, e.g., RePlaid, which reaches at most 44.5%, ClockDLM closes most of the gap to SAR discrete diffusion. We have revised the abstract and Section 4.2 to state this exception explicitly.

Claim: ClockDLM+CacheGrab exceeds AR on GSM8K

The reviewer is correct: the result that exceeds AR is Cache Grab-SSD on the jointly trained model, and this is the result the abstract and introduction reference. The accuracy gain comes from joint AR + diffusion training, while Cache Grab-SSD provides the efficiency: by construction, Cache Grab-SSD commits exactly the tokens the jointly trained model would produce under greedy AR decoding, so its 67.6% at S=32S=32 is that model’s AR-mode accuracy, reached with 35.5 instead of 158.4 NFEs/seq (4.5× fewer than AR). Consistent with this, joint training also improves standard diffusion decoding at every window size, by 0.9–5.7 points (Table 8). We have revised the Figure 1 caption, abstract, contributions, and Section 4.2 to attribute this result to Cache Grab-SSD with joint training.

As the reviewer notes, at S=4S=4 even diffusion-only training exceeds the AR baseline (65.7% vs. 64.3%). With N=SN=S, however, decoding at S=4S=4 uses about as many NFEs as AR (161.5 vs. 158.4 NFEs/seq) and so offers little parallelism; we therefore emphasize larger windows, where decoding is parallel.

Summarization results

We have narrowed the downstream claims to GSM8K. The conclusion now reads “We attained SoTA diffusion likelihood on OpenWebText and, on GSM8K, outperformed SAR discrete diffusion at all but the largest decoding budgets, with Cache Grab-SSD exceeding the AR baseline,” and the Section 4 takeaway is scoped the same way. On summarization, SAR discrete diffusion remains ahead, as the reviewer notes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.