acceptodds
Under review as a conference paper at ICLR 2027

Time-Anchored Diffusion Language Models: Latent-Space Caching for Fast Generation

Abstract

Recent work on anchored diffusion language models improves denoising by shaping an intermediate latent space with supervised important-token targets. In this work, we introduce time-based (self-supervised) anchoring, which learns and reuses latent anchors without requiring such targets. Our key observation is that anchors encode persistent properties of the clean sequence, such as its semantic intent, global structure, or intermediate plan. Although their hidden representations become stale as the token canvas evolves, their semantic content remains useful across nearby diffusion times. This is implemented through a two-stage architecture consisting of a relatively expensive anchor network that generates the latent cache state and a lightweight denoising network that intelligently combines the cached latent state with the current state at each reverse step using a fusion module. This gives anchoring a latent-space caching interpretation: the anchor network is evaluated periodically, while its cached representation is reused across multiple reverse steps. We instantiate this framework as TADM:Post-train, which time-anchorizes pretrained DLMs, and TADM:Pretraining, which learns time-based anchors during pretraining. Applied to DiffusionGemma-26B, TADM:Post-train improves throughput by approximately 49% to 79% on several math, code, and STEM benchmarks (GSM8K, AIME26, GPQA-Diamond, LiveCodeBench-v6, HumanEval). TADM:Pretraining reduces Transformer-layer computation by up to 38% relative to a standard single-stage DLM, achieves up to 73% higher measured throughput than ADLM.

Primary area:
generative models
Keywords:
Diffusion Models, Time-Anchoring, Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v7BUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TADM is proposed to mitigate the high computational cost of diffusion language models by enabling intermediate transformer layers to be skipped. Motivational insight of this paper is that there are stable token representations between denoising steps despite the bi-directional attention which can be cached and reused in the later steps.

The original model is divided into a shared embedding network, an anchor network and denoiser network. The anchor network is executed only periodically and is skipped in the remaining steps. TADM post training trains the fusion module to reflect current state to the anchored hidden state to make the resulting predictions closely follow the original output. TADM pretraining trains the model backbone and the fusion network allowing the anchor representations to be learned to help the prediction.

TADM posttraining was evaluated on DiffusionGemma and achieved up to 1.79$\times$ throughput while preserving the model accuracy. TADM pretraining was evaluated using a model trained from scratch and compared with prior works.

Critical strengths
  • Not only skipping intermediate layers, TADM trained fusion module allowing current state to be reflected to the cached anchors.
  • Experiments are done on the state of the art diffusion model.
Critical weaknesses
  • K could not be reduced enough to make a clear improvement in TADM pretraining compared to ReMDM.
  • The effectiveness of fusion module is not validated through ablation study.
Questions for authors
  • Compared to Es-dLLM (Zhu et al. ICLR 2026), is TADM more effective in terms of accuracy and throughput?
  • How are the accuracy and throughput results with the fusion module?
  • Do you need to tune K and the layer distribution according to the models and the output lengths?
  • In Table 1, how can some results from K=2 shows higher accuracy than K=1 cases?
  • How did you set the number of steps (48) per canvas?
7 comments

Dear Reviewer v7BU,

Thank you for highlighting our fusion module helps in reflecting the current state to the cached anchors and that our experiments have been performed on state of the art diffusion models. We have revised our paper with the new experiments incorporating your suggestions (written in green). We have addressed your following concerns below:

Q1) K could not be reduced enough to make a clear improvement in TADM pretraining compared to ReMDM.

A1) Thank you for bringing this up. We observe that at matched compute TADM is slightly better than ReMDM in terms of Generative Perplexity but we do not see the same trend in MAUVE. We have re-evaluated MAUVE for ADLM, ReMDM and TADM using the exact same reference samples for a fairer comparison. It is known that MAUVE can be sensitive to the number of samples per batch despite having the same total number of samples[1,2,3]. We previously reported MAUVE for ADLM and ReMDM from their respective papers while providing the new numbers only for TADM. For ensuring a fair comparison (comparing all three in an identical setting), we report updated MAUVE scores in Appendix D, Table 7, and plot MAUVE against inference throughput in Figure 2. At higher denoising-step budgets (T), TADM with (K=4) achieves both higher MAUVE and higher throughput than ReMDM, demonstrating the practical benefit of latent-anchor reuse.

Q2) The effectiveness of fusion module is not validated through ablation study

A2) We have added two fusion ablations for TADM post-training in Appendix D to answer the reviewer’s concern. Table 9 varies the paired-attention rank, and Table 10 compares paired attention with a trained gated residual MLP at the same rank. At (K=3), paired attention achieves higher mean accuracy on all five benchmarks relative to residual MLP, with gains of 24.44% on AIME26 and 11.81% on LiveCodeBench-v6. This comparison shows the benefit of paired attention over a simpler learned correction at longer anchor reuse intervals.

Q3) Compared to Es-dLLM (Zhu et al. ICLR 2026), is TADM more effective in terms of accuracy and throughput?

A3) Thank you for this great question. We have added a comparison of ES-dLLM with TADM on LLaDA, an 8B masked diffusion large language model in Appendix D, Table 8. We observe that TADM could not beat ES-dLLM in terms of throughput, however, combining the two methods gives a much higher inference speedup. With the same confidence-aware parallel decoder, TADM + ES-dLLM (Zhu et al., ICLR 2026) at (K=2) achieves higher throughput than either method alone. Adding TADM to ES-dLLM increases throughput by 17% on GSM8K and 25% on MATH-500, with mean accuracy changes of −0.07% and +0.80%, respectively. These results show that TADM complements ES-dLLM, delivering additional throughput at comparable task accuracy.

Q4) How are the accuracy and throughput results with the fusion module?

A4) In TADM pretraining, Appendix D, Figure 2 and Table 7 show the trade-off between generation quality and throughput. At (T=2048) and (K=2), TADM achieves a MAUVE score of 0.766 versus 0.776 for ADLM while generating 33% more tokens per second (32.23 versus 24.32). This shows that TADM can reuse latent anchors while retaining comparable generation quality.

In TADM post-training, Appendix D, Table 10 compares paired attention with a trained gated residual MLP at the same rank. At (K=3), paired attention achieves higher mean accuracy on all five benchmarks. On AIME26 and LiveCodeBench-v6, it improves mean accuracy by 24.44 and 11.81 percentage points, respectively, over the MLP while retaining 1.52× and 1.65× throughput relative to native DiffusionGemma with the same 48-step budget highlighting the effectiveness of our fusion module.

Q5) Do you need to tune K and the layer distribution according to the models and the output lengths?\

A5) Thank you for asking this interesting question. We did not tune the anchor refresh interval (K) or the layer split separately for each model or output length.

Q6) In Table 1, how can some results from K=2 shows higher accuracy than K=1 cases?

A6) At (K=2), reusing the anchor can change the model’s predictions and send the stochastic sampler along a different denoising trajectory. As a result, (K=2) can yield higher mean accuracy than (K=1) on some benchmarks. We added confidence intervals to Table 1 to show the uncertainty across sampling seeds. We interpret these occasional gains as variation across decoding configurations.

Q7) How did you set the number of steps (48) per canvas?

A7) We used the official sampling configuration from Diffusiongemma which uses 48 steps per canvas

Concluding Remarks:\large \text{Concluding Remarks:}

Thank you again for your thoughtful feedback and for taking the time to evaluate our revision. We hope the new results and clarifications fully address your concerns. If they have positively changed your assessment of the paper, we would be grateful if you considered updating your score accordingly. Please let us know if we can address any remaining questions you may have during the discussion phase.

References

[1] Pillutla et al. (2021). MAUVE: Measuring the Gap Between Neural Text and Human Text Using Divergence Frontiers. NeurIPS, (Appendix D.3, Fig. 7)

[2] Pillutla et al. (2023). MAUVE Scores for Generative Models: Theory and Practice. JMLR, (Section 8)

[3] Kornhardt et al. (2026). SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. arXiv:2610.06513, (Section 3.2.2, Fig. 2, and Appendix E.2.2)

Thank you for your efforts spent on this revision. I have several more questions.

Q1') In most experiments K is set to 1, 2, 3 which too small to say anchored methods are valid approach for diffusion language models (Needs full computation every 2 or 3 steps). In terms of 'computation pruning', there are a number of cache based works (Fast-dLLM (ICLR '26), DyLLM (ICML '26), dLLM-Cache (ICML '26) ...). Considering batched situation, TADM's throughput is not likely to scale as cache based works do.

Q3') Combining TADM with cache based methods is a valid direction. But combining pruning methods might improve accuracy in some cases, but there is a high possibility to further drop the model performance. ES-dLLM also skips layers for unimportant tokens. Authors should clarify TADM's distinct contribution beyond this work. Specifically, which limitations of ES-dLLM does TADM addresses, and what additional benefits does it offer in terms of generation quality and computational efficiency?

Q5') What I meant by 'tuning' is that optimal K couldn't be set across tasks. In practical use, K should be tuned. As increasing K is critical in some cases, K should be set conservatively which reduces the effect of TADM.

Reviewer_kFH2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Time-Anchored Diffusion Models (TADM), which periodically compute an expensive latent representation and reuse it across denoising steps. A lightweight current-state pathway and learned fusion module correct the cached representation before denoising. The framework supports both post-training adaptation and pretraining from scratch. On frozen DiffusionGemma-26B, training a 3.1M-parameter fusion module yields 1.49×–1.79× throughput under fixed-step evaluation, with small changes in benchmark accuracy. Smaller-scale pretraining experiments examine the generation-quality and computation trade-off across anchor refresh intervals.

Critical strengths
  1. A useful approach to learned temporal reuse. The method combines stale deep representations with inexpensive features of the current canvas. Post-training explicitly exposes the fusion module to model-generated rollout states, addressing the mismatch between ordinary training corruption and cached inference. The paired-attention construction also preserves the original representation exactly when the anchor is fresh.

  2. A substantial large-model demonstration. Adapting a frozen 26B model through only 3.1M trainable parameters is practically attractive. The measured throughput improvements across six benchmarks are meaningful. Comparisons with naive stale-anchor reuse further show that learned correction improves robustness on difficult tasks, rather than merely reducing computation.

Critical weaknesses
  1. The advantage over optimized inference alternatives remains insufficiently established. The post-training experiments disable adaptive stopping and use 48 steps per canvas. This fairly isolates per-step savings, but does not establish the improvement over the released model’s adaptive inference configuration. Comparisons with existing cache accelerators are also missing. Furthermore, the naive-reuse comparison changes training and fusion together; it does not separately establish the contributions of rollout training, distillation, and the paired-attention correction.

  2. Accuracy preservation does not fully capture the quality trade-off. Table 6 reports that, at refresh interval three, corruption increases from 11.11% to 20.00% on AIME26 and from 13.90% to 21.71% on LiveCodeBench, despite similar answer accuracy. These results materially qualify the quality-preservation message. The pretraining speedup over ADLM also comes with a substantial MAUVE gap: at 2,048 steps, self-supervised TADM scores 0.650 versus ADLM’s 0.788. Quality-matched comparisons would better establish the practical benefit.

Questions for authors
  1. How much speedup remains when adaptive stopping is enabled for both DiffusionGemma and TADM? How does TADM compare with an existing cache accelerator under matched hardware and quality?

  2. Can you report quality–latency frontiers that include corruption rates alongside task accuracy, and MAUVE alongside pretraining throughput? How sensitive are the corruption measurements to the judge, and are they supported by manual checks?

  3. Can component ablations separate stale-anchor training, rollout-based state construction, distillation, and the paired-attention fusion design? In particular, would a simpler trained residual correction achieve comparable results?

2 comments

Dear Reviewer kFH2,

Thank you for highlighting our method as a useful approach for temporal reuse and addressing the train/inference mismatch. We appreciate that you found the large-model demonstration practically attractive and our method is robust on difficult tasks. We have revised our submission with new experimental results (written in green) incorporating your suggestions. We have addressed your following concerns below:

Q1) Critical Weaknesses: The advantage over optimized inference alternatives remains insufficiently established. The post-training experiments disable adaptive stopping and use 48 steps per canvas. This fairly isolates per-step savings, but does not establish the improvement over the released model’s adaptive inference configuration. Comparisons with existing cache accelerators are also missing.

Q4) Questions for authors: How much speedup remains when adaptive stopping is enabled for both DiffusionGemma and TADM? How does TADM compare with an existing cache accelerator under matched hardware and quality?

A1,4) Thank you for raising these points. As the reviewer has correctly pointed out, we have disabled adaptive stopping and used 48 steps per canvas (maximum denoising budget as per the released sampler configuration of DiffusionGemma) for DiffusionGemma and TADM post-training to isolate the throughput gain from latent-cache reuse. To address your concern about why we disabled it, we believe that a direct comparison over adaptive stopping between DiffusionGemma and our post-training framework is unfair because of how DiffusionGemma was trained [1]. We summarize our findings here:

DiffusionGemma was trained in two stages. Stage 1 used supervised fine-tuning (SFT) to adapt a pretrained autoregressive model to a diffusion language model. The resulting model performed poorly in the few-step regime and required a maximum budget of 192 steps for high-quality generation [1, Section 5]. Stage 2 used sampler distillation and reinforcement learning (SD-RL) to compress denoising trajectories, making adaptive stopping effective in the few-step regime. LoRA fine-tuning of the released checkpoint on PubMedQA without SD-RL increased the effective denoising step count from 18.09 to 31.57 [1, Appendix C, Table 6], showing that adaptive stopping is sensitive to training. A fair comparison under adaptive stopping therefore requires accounting for SD-RL in TADM, for which the DiffusionGemma team has not released an implementation.

We address this concern by applying TADM post-training to LLaDA, an 8B masked diffusion language model trained without step distillation. We use a confidence-aware parallel decoder for adaptive stopping that decodes all tokens with confidence above 0.9, so the step budget is adaptive. At (K=3), TADM increased throughput by 41% on GSM8K and 42% on MATH-500 over LLaDA with the same decoder, with mean accuracy changes of +0.15% and −0.80%, respectively (Appendix D, Table 8).

Regarding missing cache accelerators, we note that the demonstrations of training-free cache accelerators that reuse states across denoising steps within the active canvas focus on the masked diffusion language model (whereas DiffusionGemma is a uniform diffusion language model). Therefore, we use the post-trained LLaDA and combine it with ES-dLLM [2] (Zhu et al., ICLR 2026), a state of the art training-free inference accelerator. Adding TADM (at K=3) improves throughput over ES-dLLM under single-token confidence decoding by 55% on GSM8K and 42% on MATH-500, with mean accuracy change of -0.76% and -0.60% respectively, under matched hardware. Together these results demonstrate that TADM can complement cache accelerators and adaptive samplers.

Q2) Furthermore, the naive-reuse comparison changes training and fusion together; it does not separately establish the contributions of rollout training, distillation, and the paired-attention correction.

A2) The revised paper now includes ablations on the variation of fusion rank and architecture (Appendix D, Table 9 and 10 respectively), effect of distillation (Appendix D, Table 11) and finally, comparison between rollout training and analytical construction of stale anchor pairs (Appendix D, Table 12).

Q3) Accuracy preservation does not fully capture the quality trade-off. Table 6 reports that, at refresh interval three, corruption increases from 11.11% to 20.00% on AIME26 and from 13.90% to 21.71% on LiveCodeBench, despite similar answer accuracy. These results materially qualify the quality-preservation message. The pretraining speedup over ADLM also comes with a substantial MAUVE gap: at 2,048 steps, self-supervised TADM scores 0.650 versus ADLM’s 0.788. Quality-matched comparisons would better establish the practical benefit.

A3) Thank you for bringing this up. We have added accuracy and corruption vs throughput plot (Appendix D, Figure 3) for illustrating the quality-preservation message. For pretraining, we re-evaluated MAUVE for ADLM, ReMDM, and TADM using the exact same samples for a fair comparison. It is known that MAUVE can be sensitive to the number of samples per batch, despite having the same total number of samples [3,4,5]. We previously reported MAUVE scores for TADM only for completion and took the ADLM and ReMDM scores directly from their respective papers. For ensuring a fair evaluation (comparing all three in an identical setting), we have plotted MAUVE vs throughput (Appendix D, Figure 2) to establish the practical benefits of our method.

Q5) Can you report quality–latency frontiers that include corruption rates alongside task accuracy, and MAUVE alongside pretraining throughput? How sensitive are the corruption measurements to the judge, and are they supported by manual checks?

A5) We have added the requested frontiers as suggested in Appendix D. Figure 2 plots MAUVE against generation throughput for TADM pretraining. Figure 3 plots task accuracy and corruption rate against generation throughput for TADM post-training.

To address your question about the judge, we have only measured the corrections using Qwen3-30B-A3B-Instruct-2507, a powerful model, as the judge. We did conduct manual spot checks to measure the quality of the judge's accuracy and we found that it is accurate, further reinforcing its judgements. We have added two examples which were marked corrupt by the judge:

Ex1. Repeated code

#Wait, I should just provide the code once.

def double_the_difference(lst): return sum(x**2 for x in lst if isinstance(x, int) and x >= 0 and x % 2 != 0)

def double_the_difference(lst): return sum(x**2 for x in lst if isinstance(x, int) and x >= 0 and x % 2 != 0)

This function keeps on repeating again and again

Ex2. Repeated tokens

#_,,,,,,,,,,,,1,1,11,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1

This particular response collapses to these repetitions without recovering from it.

Q6) Can component ablations separate stale-anchor training, rollout-based state construction, distillation, and the paired-attention fusion design? In particular, would a simpler trained residual correction achieve comparable results?

A6) We added four TADM post-training ablations in Appendix D as suggested by the reviewer. Table 9 varies the paired-attention rank, Table 10 compares paired attention with a trained gated residual MLP at the same rank, Table 11 ablates knowledge distillation and Table 12 studies the effect of rollout construction versus analytical constructed stale anchor pairs. Table 10 answers the specific question the reviewer has raised about residual correction versus paired attention correction. This table indicates that at (K=3), paired attention achieves higher mean accuracy than the gated residual MLP on all five benchmarks. The largest gains are 24.44% on AIME26 and 11.81% on LiveCodeBench-v6, showing that the choice of fusion architecture matters when anchors are reused for longer.

Concluding Remarks:\large \text{Concluding Remarks:}

Thank you again for your thoughtful feedback and for taking the time to evaluate our revision. We hope the new results and clarifications fully address your concerns. If they have positively changed your assessment of the paper, we would be grateful if you considered updating your score accordingly. Please let us know if we can address any remaining questions you may have during the discussion phase.

References

[1] DiffusionGemma Team et al. (2026). DiffusionGemma Technical Report. arXiv preprint arXiv:2608.00146

[2] Zhu et al. (2026). ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping. ICLR

[3] Pillutla et al. (2021). MAUVE: Measuring the Gap Between Neural Text and Human Text Using Divergence Frontiers. NeurIPS, (Appendix D.3, Fig. 7)

[4] Pillutla et al. (2023). MAUVE Scores for Generative Models: Theory and Practice. JMLR, (Section 8)

[5] Kornhardt et al. (2026). SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. arXiv:2610.06513, (Section 3.2.2, Fig. 2, and Appendix E.2.2, Figure 13)

Reviewer_G75AOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Diffusion LMs need many full-network passes, making them slower than autoregressive decoding. This paper starts from ADLM (2025), where an “anchor network” first predicts informative tokens and then a denoiser conditions on them. Whereas ADLM needs supervised anchor targets, this paper instead makes the anchor a learned latent that is computed rarely and reused, providing up to a 38% reduction in Transformer-layer computation relative to a standard single-stage DLM and up to a 73% throughput gain over ADLM.

The heart of the method: "two-stage architecture consisting of a relatively expensive anchor network that generates the latent cache state and a lightweight denoising network that intelligently combines the cached latent state with the current state at each reverse step using a fusion module," and "the anchor network is evaluated periodically, while its cached representation is reused across multiple reverse steps."

Critical strengths

The idea is simple and well-motivated---desirable properties! The on-policy rollout training addresses the train/inference mismatch directly.

Critical weaknesses

The gain over a matched-compute single-stage model is marginal. K=2 matches ReMDM’s 12-layer compute, and the two are about equal there.

No confidence intervals in results. No clean factorial ablation independently varying stale-anchor training and fusion.

I consider likely impact to be good, but moderate. The post-training recipe is cheap and practical for existing diffusion LMs, and a 1.5–1.8× gain from training only a 3.1M-parameter fusion module on a frozen 26B model is nice to see. But step distillation, adaptive sampling and KV-style caches offer competing or larger gains, and the paper doesn't show how these compose with TADM. Step distillation and adaptive sampling attack the same budget from the other side. Total cost is roughly "number of steps" * "cost per step" and TADM only reduces the second factor, while step distillation and early stopping reduce the first. The paper asserts the methods are “complementary” but doesn't test that. It would be nice to see that shown, because there is a reason to doubt it. For example, TADM’s own tables suggest the benefit shrinks as steps get fewer. Its premise is that the useful semantic information in the anchor changes slowly enough across nearby diffusion steps to allow reuse.

Questions for authors

There is other work in the cross-step hidden-state literature: Loopholing (Jo et al., 2025), MetaState (Xia et al., 2026) and Soft-Masked Diffusion (Hersche et al., 2025). It would be interesting to hear how TADM compares conceptually (and perhaps experimentally, if you think it worthwhile).

2 comments

Dear Reviewer G75A,

Thank you for highlighting our idea as simple and well-motivated and our on-policy training directly addresses the train/inference mismatch. We have revised the draft with new experimental results (written in green) incorporating your suggestions. We address your following concerns below:

Q1) The gain over a matched-compute single-stage model is marginal. K=2 matches ReMDM’s 12-layer compute, and the two are about equal there

A1) Thanks for bringing this up. At matched compute, TADM is slightly better than ReMDM in terms of Generative Perplexity especially at higher T, but we do not see the same trend for MAUVE. We have re-evaluated MAUVE for ADLM, ReMDM and TADM using the exact same reference samples for a fairer comparison. It is known that MAUVE can be sensitive to the number of samples per batch despite having the same total number of samples [1,2,3]. We previously reported MAUVE for ADLM and ReMDM from their respective papers while providing the new numbers only for TADM. For ensuring a fair comparison, we report updated MAUVE scores in Appendix D, Table 7 (comparing all three in an identical setting), and plot MAUVE against inference throughput in Figure 2 establishing the practical benefit of latent-anchor reuse.

Q2) No confidence intervals in results. No clean factorial ablation independently varying stale-anchor training and fusion.

A2) The revised paper now includes confidence intervals for task accuracy, ablations on different fusion rank and architecture in Appendix D, Table 9 and 10, respectively and assess the contribution of rollout-based training with analytically constructed stale-anchor pairs (Appendix D, Table 12) on LLaDA, a well known 8B diffusion large language model keeping the fusion architecture and training objective fixed (Appendix D, Table 12)

Q3) But step distillation, adaptive sampling and KV-style caches offer competing or larger gains, and the paper doesn't show how these compose with TADM. Step distillation and adaptive sampling attack the same budget from the other side. Total cost is roughly "number of steps" * "cost per step" and TADM only reduces the second factor, while step distillation and early stopping reduce the first. The paper asserts the methods are “complementary” but doesn't test that. It would be nice to see that shown, because there is a reason to doubt it. For example, TADM’s own tables suggest the benefit shrinks as steps get fewer. Its premise is that the useful semantic information in the anchor changes slowly enough across nearby diffusion steps to allow reuse

A3) Thank you for this insightful question. We agree that reducing the number of steps and reducing the cost per step can interact with each other. TADM provides a way of reducing the cost per step, whereas, step distillation and adaptive sampling reduces the number of steps. To address your concern about step distillation, our post-training framework is applied to the released DiffusionGemma checkpoint, which has already undergone sampler distillation and Reinforcement Learning [4], a step-distillation technique. The results in Table 1, therefore, show that TADM can accelerate an already step-distilled model.

Regarding KV-style caching, we note that the demonstrations of training-free KV-style cache accelerators that reuse states across denoising steps within the active canvas focus on the masked diffusion language model (whereas DiffusionGemma is a uniform diffusion language model). Therefore, we post-train LLaDA using our framework and combine it with ES-dLLM [5] (Zhu et al., ICLR 2026), a state of the art training-free inference accelerator. Adding TADM (at K=3) improves throughput over ES-dLLM under single-token confidence decoding by 55% on GSM8K and 42% on MATH-500, with mean accuracy change of -0.76% and -0.60% respectively.

We further address your concern about adaptive sampling, by using a confidence-aware parallel decoder, which decodes all tokens above 0.9 confidence, on TADM post-trained LLaDA. At K=3, we achieve an inference speedup of 41% on GSM8K and 42% on MATH-500 over base LLaDA with this adaptive sampler with a mean accuracy change of +0.15% and -0.80% respectively. Finally, with parallel decoding + K=2 TADM + ES-dLLM, we get a speedup of 17% on GSM8K and 25% on MATH-500 relative to ES-dLLM with the same sampler. Mean accuracy changes by -0.07% and +0.80% respectively. In comparison with LLaDA + parallel decoder, we get an impressive speedup of 563% on GSM8K and 476% on MATH-500 with a mean accuracy change of -2.42% and -2% respectively. Together these results demonstrate that TADM can complement step distillation, adaptive samplers and cache accelerators.

Q4) There is other work in the cross-step hidden-state literature: Loopholing (Jo et al., 2025), MetaState (Xia et al., 2026) and Soft-Masked Diffusion (Hersche et al., 2025). It would be interesting to hear how TADM compares conceptually (and perhaps experimentally, if you think it worthwhile).

A4) Thank you for pointing out these connections; we have now updated the manuscript with additional related work in Appendix B.3. We summarize these connections below. TADM shares a core idea with all three methods: a denoising step can use continuous information computed at an earlier step alongside the current discrete token canvas. Loopholing passes a continuous latent representation through a deterministic pathway. MetaState recurrently updates a fixed-size memory from backbone activations. Soft-Masked Diffusion carries forward information about previous token predictions through the embeddings of positions that remain masked.

In TADM, the information carried forward is a deep intermediate hidden state from the most recent anchor refresh. TADM holds this state fixed between refreshes and uses lightweight fusion to adapt it to the current canvas. The cited methods use cross-step information to enrich denoising. TADM uses it to reduce per-step computation by skipping the anchor layers on reuse steps.

Regarding experimental evaluation, we can do it either with pre-training or post-training. For TADM pretraining, it would be interesting to perform a controlled comparison across refresh intervals (K) testing whether adding cross-step feedback preserves generation quality at longer reuse intervals and improves throughput at matched quality. However, due to training time, we are unable to do this within the response phase. We will be happy to perform this experiment for a future revision. In TADM post-training, this latent reuse operates alongside DiffusionGemma’s native self-conditioning. Like soft masking, DiffusionGemma converts previous token predictions into an embedding-space signal. It applies a feedforward network to the probability-weighted token embeddings and supplies the result to the next denoising step. Our post-training results show that TADM can accelerate a model with this prediction feedback already in place.

Concluding Remarks:\large \text{Concluding Remarks:}

Thank you again for your thoughtful feedback and for taking the time to evaluate our revision. We hope the new results and clarifications fully address your concerns. If they have positively changed your assessment of the paper, we would be grateful if you considered updating your score accordingly. Please let us know if we can address any remaining questions you may have during the discussion phase.

References

[1] Pillutla et al. (2021). MAUVE: Measuring the Gap Between Neural Text and Human Text Using Divergence Frontiers. NeurIPS, (Appendix D.3, Fig. 7)

[2] Pillutla et al. (2023). MAUVE Scores for Generative Models: Theory and Practice. JMLR, (Section 8)

[3] Kornhardt et al. (2026). SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. arXiv:2610.06513, (Section 3.2.2, Fig. 2, and Appendix E.2.2, Figure 13)

[4] DiffusionGemma Team et al. (2026). DiffusionGemma Technical Report. arXiv preprint arXiv:2608.00146, (Section 5)

[5] Zhu et al. (2026). ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping. ICLR

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.