- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TIDE is the first study of distillation-based model compression (as opposed to step reduction) for diffusion large language models (dLLMs). The authors identify three obstacles that make dLLM compression a cross-architecture problem: timestep-dependent teacher reliability, context scarcity from heavy masking, and tokenizer/vocabulary mismatch. They propose a regime-adaptive framework: TIDAL (dual-axis scheduling over diffusion timestep and training progress) and CompDemo (mask splitting with two complementary teacher forward passes) for shared-tokenizer pairs; Reverse CALM (chunk-level BCE with reversed direction, exploiting the vanishing gradient coefficient when teacher probability approaches 0.5 to filter alignment noise) for cross-tokenizer pairs. Experiments distill WeDLM-8B (dense) and LLaDA2.0-mini (16B MoE) into a 0.6B BD3LM student. Naive distillation does not help (KL -2.12, forward CALM -0.42); TIDE improves over naive by +3.00/+1.95 and over the non-distilled student by +0.88/+1.53 on the eight-benchmark average, with gains concentrated on cross-tokenizer GSM8K (+6.68) and code; memory is reduced 11-22x relative to the teachers.
Critical strengths
The problem framing is genuinely novel - dLLM compression distillation has not been studied before. The regime-adaptive design is well motivated, and the gradient analysis for Reverse CALM (mode-seeking behavior plus dual-end noise filtering) provides theoretical support. Two independent pipelines with different teachers, tokenizers, and naive baselines reduce the risk of a single-run artifact. Paired McNemar tests with Bonferroni correction are used. The limitations section is exceptionally honest: the authors explicitly acknowledge that the AR baseline is faster and more accurate, that gains are narrow, that ablations are only suggestive, and that no training-seed variance is estimated.
Critical weaknesses
-
Effect sizes are narrow and statistical power is weak. The headline +0.88/+1.53 average gains over the non-distilled student are concentrated on cross-tokenizer GSM8K (+6.68) and code; MMLU, HellaSwag, and MATH move by less than a point. All configurations are single-seed, and the authors state they do not estimate variance across training seeds. Ablation differences (0.08-0.26 on average) are explicitly acknowledged to be at the scale of evaluation-side variation (re-evaluation alone moves the average by up to 0.07). HumanEval gains amount to 3-4 problems out of 164, too few to be individually significant. Without seed variance, the component-level contributions of TIDAL and CompDemo are not established.
-
Practical value is undercut by the AR baseline. A same-size AR model (Qwen3-0.6B) is both faster (51.3 vs 41.0 tokens/s) and more accurate (40.9 vs 34.20 average, by reported scores). The deployment positioning acknowledges that the practical benefit is the 11-22x memory reduction and targets memory-constrained or batched settings - but no batched-throughput comparison against AR is provided. The central deployment claim is untested in exactly the regime where it would matter.
-
Ablation budget and coverage are insufficient to support the regime-adaptive design. Component ablations run at 3 epochs (vs 10 for main runs), only in the shared-tokenizer pipeline, and only for TIDE-Interp. The cross-tokenizer pipeline - which carries the main GSM8K gain and the Reverse CALM contribution - has no component-level ablation. The central claim is regime-adaptive matching, but the evidence is: shared-tokenizer prefers TIDE-Interp by 2.76, while the two cross-tokenizer configurations differ by only 0.37, a gap within the noise floor the authors themselves identify.
Questions for authors
-
Can you provide multi-seed results (at least 3 seeds) for the main configurations? Without training-seed variance, the +0.88/+1.53 average gains and the 0.08-0.26 ablation differences cannot be distinguished from run-to-run noise.
-
Can you provide a batched-throughput comparison against the same-size AR model? The deployment claim rests on memory-constrained and batched settings, but this is exactly where no data is provided.
-
Can you provide cross-tokenizer component ablations (TIDAL alone, Reverse CALM alone, CompDemo alone) at the full 10-epoch budget? The regime-adaptive claim needs direct evidence in the pipeline where the main gain lives.