- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TADM is proposed to mitigate the high computational cost of diffusion language models by enabling intermediate transformer layers to be skipped. Motivational insight of this paper is that there are stable token representations between denoising steps despite the bi-directional attention which can be cached and reused in the later steps.
The original model is divided into a shared embedding network, an anchor network and denoiser network. The anchor network is executed only periodically and is skipped in the remaining steps. TADM post training trains the fusion module to reflect current state to the anchored hidden state to make the resulting predictions closely follow the original output. TADM pretraining trains the model backbone and the fusion network allowing the anchor representations to be learned to help the prediction.
TADM posttraining was evaluated on DiffusionGemma and achieved up to 1.79$\times$ throughput while preserving the model accuracy. TADM pretraining was evaluated using a model trained from scratch and compared with prior works.
Critical strengths
- Not only skipping intermediate layers, TADM trained fusion module allowing current state to be reflected to the cached anchors.
- Experiments are done on the state of the art diffusion model.
Critical weaknesses
- K could not be reduced enough to make a clear improvement in TADM pretraining compared to ReMDM.
- The effectiveness of fusion module is not validated through ablation study.
Questions for authors
- Compared to Es-dLLM (Zhu et al. ICLR 2026), is TADM more effective in terms of accuracy and throughput?
- How are the accuracy and throughput results with the fusion module?
- Do you need to tune K and the layer distribution according to the models and the output lengths?
- In Table 1, how can some results from K=2 shows higher accuracy than K=1 cases?
- How did you set the number of steps (48) per canvas?
7 comments
Q5) Do you need to tune K and the layer distribution according to the models and the output lengths?\
A5) Thank you for asking this interesting question. We did not tune the anchor refresh interval (K) or the layer split separately for each model or output length.
Q6) In Table 1, how can some results from K=2 shows higher accuracy than K=1 cases?
A6) At (K=2), reusing the anchor can change the model’s predictions and send the stochastic sampler along a different denoising trajectory. As a result, (K=2) can yield higher mean accuracy than (K=1) on some benchmarks. We added confidence intervals to Table 1 to show the uncertainty across sampling seeds. We interpret these occasional gains as variation across decoding configurations.
Q7) How did you set the number of steps (48) per canvas?
A7) We used the official sampling configuration from Diffusiongemma which uses 48 steps per canvas
Thank you again for your thoughtful feedback and for taking the time to evaluate our revision. We hope the new results and clarifications fully address your concerns. If they have positively changed your assessment of the paper, we would be grateful if you considered updating your score accordingly. Please let us know if we can address any remaining questions you may have during the discussion phase.
References
[1] Pillutla et al. (2021). MAUVE: Measuring the Gap Between Neural Text and Human Text Using Divergence Frontiers. NeurIPS, (Appendix D.3, Fig. 7)
[2] Pillutla et al. (2023). MAUVE Scores for Generative Models: Theory and Practice. JMLR, (Section 8)
[3] Kornhardt et al. (2026). SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics. arXiv:2610.06513, (Section 3.2.2, Fig. 2, and Appendix E.2.2)
Thank you for your efforts spent on this revision. I have several more questions.
Q1') In most experiments K is set to 1, 2, 3 which too small to say anchored methods are valid approach for diffusion language models (Needs full computation every 2 or 3 steps). In terms of 'computation pruning', there are a number of cache based works (Fast-dLLM (ICLR '26), DyLLM (ICML '26), dLLM-Cache (ICML '26) ...). Considering batched situation, TADM's throughput is not likely to scale as cache based works do.
Q3') Combining TADM with cache based methods is a valid direction. But combining pruning methods might improve accuracy in some cases, but there is a high possibility to further drop the model performance. ES-dLLM also skips layers for unimportant tokens. Authors should clarify TADM's distinct contribution beyond this work. Specifically, which limitations of ES-dLLM does TADM addresses, and what additional benefits does it offer in terms of generation quality and computational efficiency?
Q5') What I meant by 'tuning' is that optimal K couldn't be set across tasks. In practical use, K should be tuned. As increasing K is critical in some cases, K should be set conservatively which reduces the effect of TADM.
Dear Reviewer v7BU,
Thank you for highlighting our fusion module helps in reflecting the current state to the cached anchors and that our experiments have been performed on state of the art diffusion models. We have revised our paper with the new experiments incorporating your suggestions (written in green). We have addressed your following concerns below:
Q1) K could not be reduced enough to make a clear improvement in TADM pretraining compared to ReMDM.
A1) Thank you for bringing this up. We observe that at matched compute TADM is slightly better than ReMDM in terms of Generative Perplexity but we do not see the same trend in MAUVE. We have re-evaluated MAUVE for ADLM, ReMDM and TADM using the exact same reference samples for a fairer comparison. It is known that MAUVE can be sensitive to the number of samples per batch despite having the same total number of samples[1,2,3]. We previously reported MAUVE for ADLM and ReMDM from their respective papers while providing the new numbers only for TADM. For ensuring a fair comparison (comparing all three in an identical setting), we report updated MAUVE scores in Appendix D, Table 7, and plot MAUVE against inference throughput in Figure 2. At higher denoising-step budgets (T), TADM with (K=4) achieves both higher MAUVE and higher throughput than ReMDM, demonstrating the practical benefit of latent-anchor reuse.
Q2) The effectiveness of fusion module is not validated through ablation study
A2) We have added two fusion ablations for TADM post-training in Appendix D to answer the reviewer’s concern. Table 9 varies the paired-attention rank, and Table 10 compares paired attention with a trained gated residual MLP at the same rank. At (K=3), paired attention achieves higher mean accuracy on all five benchmarks relative to residual MLP, with gains of 24.44% on AIME26 and 11.81% on LiveCodeBench-v6. This comparison shows the benefit of paired attention over a simpler learned correction at longer anchor reuse intervals.
Q3) Compared to Es-dLLM (Zhu et al. ICLR 2026), is TADM more effective in terms of accuracy and throughput?
A3) Thank you for this great question. We have added a comparison of ES-dLLM with TADM on LLaDA, an 8B masked diffusion large language model in Appendix D, Table 8. We observe that TADM could not beat ES-dLLM in terms of throughput, however, combining the two methods gives a much higher inference speedup. With the same confidence-aware parallel decoder, TADM + ES-dLLM (Zhu et al., ICLR 2026) at (K=2) achieves higher throughput than either method alone. Adding TADM to ES-dLLM increases throughput by 17% on GSM8K and 25% on MATH-500, with mean accuracy changes of −0.07% and +0.80%, respectively. These results show that TADM complements ES-dLLM, delivering additional throughput at comparable task accuracy.
Q4) How are the accuracy and throughput results with the fusion module?
A4) In TADM pretraining, Appendix D, Figure 2 and Table 7 show the trade-off between generation quality and throughput. At (T=2048) and (K=2), TADM achieves a MAUVE score of 0.766 versus 0.776 for ADLM while generating 33% more tokens per second (32.23 versus 24.32). This shows that TADM can reuse latent anchors while retaining comparable generation quality.
In TADM post-training, Appendix D, Table 10 compares paired attention with a trained gated residual MLP at the same rank. At (K=3), paired attention achieves higher mean accuracy on all five benchmarks. On AIME26 and LiveCodeBench-v6, it improves mean accuracy by 24.44 and 11.81 percentage points, respectively, over the MLP while retaining 1.52× and 1.65× throughput relative to native DiffusionGemma with the same 48-step budget highlighting the effectiveness of our fusion module.