acceptodds
Under review as a conference paper at ICLR 2027

Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models

Abstract

Diffusion large language models (dLLMs) support parallel decoding and bidirectional context, but competitive dLLMs have 8–16B parameters, which limits deployment. Knowledge distillation is a natural remedy, yet existing dLLM methods only reduce inference steps: the distilled model is faster but no smaller. Distillation for model compression, routine for autoregressive (AR) LLMs, has not been studied for dLLMs. Because teacher and student differ in backbone, attention, and tokenizer, compressing a dLLM is a cross-architecture problem with three obstacles: timestep-dependent teacher reliability, context scarcity from heavy masking, and tokenizer/vocabulary mismatch. We present TIDE, a regime-adaptive framework whose components are matched to the teacher–student gap: TIDAL (timestep-aware scheduling) and CompDemo (context enrichment) for shared-tokenizer pairs, and Reverse CALM (noise-filtering chunk-level alignment) for cross-tokenizer pairs. TIDE distills 8B-dense and 16B-MoE teachers, one per run, into a 0.6B student that needs 11–22× less memory than the teachers. Naive distillation does not help: it causes negative transfer for the shared-tokenizer pair and gives no gain for the cross-tokenizer pair. TIDE improves on it in both pipelines (+3.00 and +1.95 on the eight-benchmark average) and ends above the non-distilled student (+0.88 and +1.53), most clearly on GSM8K (+6.68 in the cross-tokenizer pipeline) and less clearly on code. The reported scores of a same-size AR model remain higher (40.9 vs. 34.20 average). To our knowledge, this is the first evidence that a large dLLM can be compressed across heterogeneous architectures. Code is available at https://anonymous.4open.science/r/tide-5E36/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion large language models, cross-architecture distillation, knowledge distillation, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uiXTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TIDE is the first study of distillation-based model compression (as opposed to step reduction) for diffusion large language models (dLLMs). The authors identify three obstacles that make dLLM compression a cross-architecture problem: timestep-dependent teacher reliability, context scarcity from heavy masking, and tokenizer/vocabulary mismatch. They propose a regime-adaptive framework: TIDAL (dual-axis scheduling over diffusion timestep and training progress) and CompDemo (mask splitting with two complementary teacher forward passes) for shared-tokenizer pairs; Reverse CALM (chunk-level BCE with reversed direction, exploiting the vanishing gradient coefficient when teacher probability approaches 0.5 to filter alignment noise) for cross-tokenizer pairs. Experiments distill WeDLM-8B (dense) and LLaDA2.0-mini (16B MoE) into a 0.6B BD3LM student. Naive distillation does not help (KL -2.12, forward CALM -0.42); TIDE improves over naive by +3.00/+1.95 and over the non-distilled student by +0.88/+1.53 on the eight-benchmark average, with gains concentrated on cross-tokenizer GSM8K (+6.68) and code; memory is reduced 11-22x relative to the teachers.

Critical strengths

The problem framing is genuinely novel - dLLM compression distillation has not been studied before. The regime-adaptive design is well motivated, and the gradient analysis for Reverse CALM (mode-seeking behavior plus dual-end noise filtering) provides theoretical support. Two independent pipelines with different teachers, tokenizers, and naive baselines reduce the risk of a single-run artifact. Paired McNemar tests with Bonferroni correction are used. The limitations section is exceptionally honest: the authors explicitly acknowledge that the AR baseline is faster and more accurate, that gains are narrow, that ablations are only suggestive, and that no training-seed variance is estimated.

Critical weaknesses
  1. Effect sizes are narrow and statistical power is weak. The headline +0.88/+1.53 average gains over the non-distilled student are concentrated on cross-tokenizer GSM8K (+6.68) and code; MMLU, HellaSwag, and MATH move by less than a point. All configurations are single-seed, and the authors state they do not estimate variance across training seeds. Ablation differences (0.08-0.26 on average) are explicitly acknowledged to be at the scale of evaluation-side variation (re-evaluation alone moves the average by up to 0.07). HumanEval gains amount to 3-4 problems out of 164, too few to be individually significant. Without seed variance, the component-level contributions of TIDAL and CompDemo are not established.

  2. Practical value is undercut by the AR baseline. A same-size AR model (Qwen3-0.6B) is both faster (51.3 vs 41.0 tokens/s) and more accurate (40.9 vs 34.20 average, by reported scores). The deployment positioning acknowledges that the practical benefit is the 11-22x memory reduction and targets memory-constrained or batched settings - but no batched-throughput comparison against AR is provided. The central deployment claim is untested in exactly the regime where it would matter.

  3. Ablation budget and coverage are insufficient to support the regime-adaptive design. Component ablations run at 3 epochs (vs 10 for main runs), only in the shared-tokenizer pipeline, and only for TIDE-Interp. The cross-tokenizer pipeline - which carries the main GSM8K gain and the Reverse CALM contribution - has no component-level ablation. The central claim is regime-adaptive matching, but the evidence is: shared-tokenizer prefers TIDE-Interp by 2.76, while the two cross-tokenizer configurations differ by only 0.37, a gap within the noise floor the authors themselves identify.

Questions for authors
  1. Can you provide multi-seed results (at least 3 seeds) for the main configurations? Without training-seed variance, the +0.88/+1.53 average gains and the 0.08-0.26 ablation differences cannot be distinguished from run-to-run noise.

  2. Can you provide a batched-throughput comparison against the same-size AR model? The deployment claim rests on memory-constrained and batched settings, but this is exactly where no data is provided.

  3. Can you provide cross-tokenizer component ablations (TIDAL alone, Reverse CALM alone, CompDemo alone) at the full 10-epoch budget? The regime-adaptive claim needs direct evidence in the pipeline where the main gain lives.

Reviewer_E24SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TIDE for compressing large diffusion language models into smaller students across heterogeneous architectures. It addresses three challenges: timestep-dependent teacher reliability, limited context under heavy masking, and tokenizer mismatch. The method is evaluated by distilling an 8B dense teacher and a 16B mixture-of-experts teacher, separately, into a 0.6B block-diffusion student. Across eight benchmarks, the proposed configurations improve the average score over naive distillation and exceed the non-distilled student. The student also requires much less inference memory than the teachers. However, the evaluation covers only one student initialization and two teacher–student pairs, with no repeated training runs to establish the robustness of the gains.

Critical strengths
  1. The paper addresses an important and underexplored compression problem. Distilling 8B and 16B teachers into a 0.6B student is a meaningful step toward making dLLMs more accessible under memory constraints.

  2. The paper provides a technically motivated, regime-adaptive framework with comparative validation. TIDE explicitly addresses teacher reliability, context scarcity, and tokenizer mismatch. Evaluations across two distinct teacher–student pipelines show improvements over naive distillation and the non-distilled student, with paired significance tests supporting gains on several individual benchmarks.

Critical weaknesses
  1. The paper provides limited evidence for the robustness and generality of the reported improvements. Each configuration is trained only once, and the evaluation uses a single 0.6B student initialization and two teacher–student pairs. The improvements over the non-distilled student are modest on the eight-benchmark average (+0.88 and +1.53 points), with gains concentrated on GSM8K. The ablation differences are also small and explicitly acknowledged as potentially within run-to-run variation. Repeated runs and broader model coverage would help establish whether the gains are reproducible and generalizable.

  2. The practical advantage over existing small language models remains insufficiently established. Although the student substantially reduces memory requirements relative to the large dLLM teachers, the reported same-size autoregressive model is both more accurate (40.9 versus 34.2 average) and faster in single-stream inference. The paper does not establish whether the compressed dLLM offers a meaningful throughput or latency advantage in realistic batched-serving settings.

Questions for authors
  1. Have the authors run any repeated training experiments with different random seeds for the main configurations? Whether the 0.88- and 1.53-point improvements over the non-distilled student are reproducible, given the absence of training-run variance estimates?

  2. The shared-tokenizer ablations show relatively small average differences, and the cross-tokenizer ablation is missing. What evidence supports attributing the main improvements to the proposed components individually, particularly reverse CALM's noise-filtering mechanism?

  3. Are there results to show throughput and latency under batched serving or longer-generation workloads against a same-size autoregressive model? Such results would help clarify when the compressed dLLM offers a practical advantage beyond reduced memory consumption.

Reviewer_3h5tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes knowledge distillation for compressing diffusion LLMs (dLLMs) into much smaller students, whereas prior relevant works only reduce denoising steps at a fixed model size. The motivation of the work points out a cross-architecture problem with three issues: timestep-dependent teacher reliability, context scarcity under heavy masking, and tokenizer mismatch. The contributions are TIDE, which combines a dual-axis interpolation schedule over diffusion timestep and training progress (TIDAL), two complementary teacher passes that each reveal half of the masked tokens (CompDemo), and a reverse chunk-level BCE objective for cross-tokenizer pairs (Reverse CALM). The student needs 11–22× less memory than the teachers but remains slower; however, it reports lower accuracy compared to the autoregressive model Qwen3-0.6B (40.9 average).

Critical strengths

The main strength of the paper is the relevancy of the paper. The topic of distillation/compression of dLLMs is both significant and novel to the research community.

Critical weaknesses
  • The two reference points are the AR model and the non-distilled BD3LM. However, BD3LM is also the initialization from which the distilled students are trained for 10 additional epochs. A fairer comparison would be BD3LM trained for the same 10 extra epochs with standard supervised fine-tuning (cross-entropy only, no teacher). Without this control, it is hard to tell how much of the improvement comes from distillation and how much simply comes from additional training. The authors acknowledge this in Sec. 3.1 and Sec. 4. The naive-distillation baseline holds training length fixed, but it is not a teacher-free control. This matters all the more because some of the reported gains over the baseline are quite small.

  • The choice between TIDE-Rev and TIDE-Interp seems to be made after looking at the test results. No validation split is mentioned. In the cross-tokenizer setting, the two differ by only 0.37 points. This gives the impression that there is no clear principle for choosing one configuration over the other, beyond trying both and keeping the better one.

  • Table 9 shows that in some cases the teachers perform surprisingly poorly under the evaluation protocol. For example, LLaDA2.0-mini scores 1.79 on MMLU-Pro, which is even below the 0.6B student. The authors attribute this to truncation or answer-extraction issues. If the teachers tend to produce outputs so long that they do not reach a final answer within the token budget, distillation could transfer this tendency to the student.

  • I’m not sure what the AR model is meant to show. It is never used as a teacher, so the method is not trying to match its distribution. Its scores are taken from prior work rather than evaluated with the same harness. As it stands, it does not work as a baseline for the method. The configuration in Table 6 (number of sampling steps equal to the number of generated tokens) does not reflect a realistic use of a dLLM. It removes the main advantage of diffusion models over AR models, namely parallel decoding.

  • The “memory is the main benefit” argument, along with the throughput comparison, essentially compares a 0.6B model with an 8B/16B model. Those advantages come from model size, not from the proposed method, and the student remains far from the teachers’ quality. I don’t think this comparison supports the contribution.

  • Finally, the distilled model is both slower and less accurate than the same-size AR model (Qwen3-0.6B) from which the student was originally derived. Even allowing for the fact that the AR accuracy numbers come from a different harness, the gap is large (40.9 vs. 34.20). The speed gap (51.3 vs. 41.0 tokens/s) comes from the authors’ own measurements. It is not clear to me why one would use this approach rather than simply training or fine-tuning Qwen directly in a more standard way.

Questions for authors
  • Can you report BD3LM fine-tuned for the same 10 additional epochs with cross-entropy only (no teacher), under the same protocol? This would directly affect my assessment of whether the gains come from distillation.

  • How was the configuration for each regime (TIDE-Interp vs. TIDE-Rev) selected? Was any held-out validation data used, or only the test results in Table 1?

  • Do the distilled students produce longer outputs or more truncated, unextractable answers than the non-distilled BD3LM? For example, what are the output-length statistics and extraction-failure rates per task?

  • How do the distilled students compare with the non-distilled BD3LM when decoding several tokens per step (fewer sampling steps than tokens)?

  • In what concrete setting would a practitioner prefer the distilled 0.6B dLLM over the 0.6B AR model it was derived from? A batched-throughput comparison against AR would help here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.