acceptodds
Under review as a conference paper at ICLR 2027

Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLM

Abstract

Trajectory distillation is a promising training-based approach for accelerating diffusion large language models (dLLMs) by adapting them to generate more tokens per forward pass. However, existing methods construct trajectories from the base model alone and supervise masked tokens without distinguishing their decoding horizons, which limits the knowledge transferred through distillation and yields a suboptimal accuracy–parallelism trade-off. Enriching the knowledge carried by these trajectories and tailoring supervision to temporal decoding dynamics remain open challenges. Inspired by learning with privileged information, and motivated by our observation that the teacher–student distributional gap varies systematically with a token's time to reveal, we propose TAD, a Temporal-Aware trajectory self-Distillation framework. TAD conditions a frozen teacher on target responses available only during training to construct informative decoding trajectories. It then partitions masked tokens into near and distant subsets according to their temporal distance to decoding, applying hard cross-entropy supervision to near tokens for confident parallel decoding and soft distribution matching to distant tokens to avoid premature decisions while preserving future-planning knowledge. Across mathematical reasoning and code generation benchmarks, TAD improves the average accuracy of LLaDA-8B-Instruct from 46.1% to 51.5% and Dream from 58.9% to 61.2%. Its speed-oriented configuration further achieves the highest average Accuracy Under Parallelism scores of 256.6 and 193.9 on the two backbones respectively, which validate that TAD establishes a superior accuracy-parallelism frontier, simultaneously enhancing task accuracy and decoding throughput. Our code is available at https://anonymous.4open.science/r/TAD-B9D0/https://anonymous.4open.science/r/TAD-B9D0/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Large Language Models, Distillation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_K5EcOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TAD trains diffusion language models using teacher trajectories guided by reference answers. It applies hard supervision to tokens decoded soon and soft supervision to those decoded later. Experiments on LLaDA and Dream show gains in average accuracy and decoding efficiency over the base models on math and code tasks.

Critical strengths
  1. The method uses different training targets depending on when each token will be decoded. This provides a simple way to balance speed and accuracy.
  2. Results across two model families support the main claim that TAD can improve average accuracy and decoding efficiency together.
Critical weaknesses
  1. The temporal split follows the teacher’s decoding order, but its suitability for the student’s parallel decoding remains unclear.
  2. Table 3 supports using soft KL for distant tokens. However, the paper does not directly show that the gains come from preserving uncertainty in these tokens.
Questions for authors

How are invalid teacher trajectories filtered? Does this include checking final answers or running code tests before using the trajectory tokens as hard targets?

3 comments

We sincerely thank the reviewer for the positive and encouraging assessment of our work. We are particularly grateful for the recognition that TAD leverages "different training targets depending on when each token will be decoded" to provide "a simple way to balance speed and accuracy."

We also thank the reviewer for raising these thoughtful questions and we address them below.

W1: Suitability of the temporal split for the student's parallel decoding.

We agree that the key consideration is whether the temporal partition derived from the teacher's reveal order is also meaningful for the student's prediction and parallel decoding behavior. Importantly, our temporal partition is not based solely on the teacher's decoding order. As shown in Figure 3(b), we explicitly measure the student's confidence in predicting teacher trajectory tokens at different temporal distances. We observe that the student's confidence is high for near tokens and progressively decreases as the teacher's reveal distance increases. This indicates that tokens revealed earlier by the teacher are also those that the student can predict more confidently, whereas distant tokens remain less predictable to the student. Therefore, the teacher's temporal order serves as a reliable proxy for how confidently the student can predict each token.

W2: Whether soft KL benefits from preserving uncertainty in distant tokens.

To directly verify that the gains of soft KL come from preserving uncertainty in distant tokens, we analyze the prediction statistics on 100 sampled trajectories. First, we measure the entropy of the teacher's output distributions. Distant tokens show much higher uncertainty in the teacher (average entropy of 2.770) than near tokens (0.844). This confirms that the teacher's predictions for distant tokens are inherently uncertain and should not be collapsed into one-hot targets. Second, we compare students trained with Hard-Hard (HH, applying hard cross-entropy to all tokens) and Hard-Soft (HS, our TAD method) on distant tokens using four metrics:

(1) Student Predictive Entropy: The average entropy of the student's output distributions, measuring prediction uncertainty.

(2) Mean Confidence: The average maximum softmax probability, indicating how peaked the predictions are.

(3) Teacher–Student KL: The KL divergence between the teacher's and student's output distributions.

(4) Token Agreement: The top-1 matching rate between the student's predicted tokens and the teacher's trajectory tokens.

Metric (Distant Tokens)Hard-HardHard-Soft
Student Predictive Entropy2.1722.844
Mean Confidence0.4920.355
Teacher–Student KL ↓1.4070.702
Token Agreement ↑0.2020.299

Despite producing more concentrated predictions, HH exhibits approximately 2× higher KL divergence and 9.7 percentage points lower token agreement than HS. This suggests that hard-only supervision can lead to overconfident and poorly aligned predictions on distant tokens, whereas soft KL better preserves the teacher's predictive information. These results support our motivation that distant tokens benefit from preserving the teacher's predictive distributions rather than being directly collapsed into hard one-hot targets. We will incorporate these additional analyses into the revised manuscript to better explain the rationale behind TAD's hard–soft supervision design.

Q1: Filtering of invalid teacher trajectories.

We apply task-specific validation to filter invalid teacher trajectories before using them for student training. For math tasks, we check whether the final answer extracted from each generated trajectory is correct by comparing it against the reference answer. Trajectories with incorrect final answers are discarded. For code generation tasks, we execute the generated programs against the corresponding test cases and retain only trajectories whose generated code passes the tests. We will clarify these filtering procedures in the revised manuscript.

My concerns have been fully addressed, and I would like to keep my score at a positive 3.

Thank you very much for your positive assessment, and we sincerely appreciate your time and thoughtful review.

Reviewer_uF8iOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TAD, a temporal-aware trajectory self-distillation framework for accelerating diffusion language models. During training, a frozen teacher is conditioned on the reference response and generates a trajectory by revealing one token at each step. The masked positions in an intermediate state are partitioned according to their time to reveal: near tokens receive hard cross-entropy supervision, whereas distant tokens receive forward-KL supervision from the privileged teacher.

Experiments on LLaDA-8B-Instruct and Dream-7B-Instruct cover GSM8K, MATH, MBPP, and HumanEval. The paper reports accuracy, tokens per forward pass, and Accuracy Under Parallelism, together with objective ablations, matched-data baseline comparisons, five-run evaluation statistics, and H200 throughput measurements on GSM8K.

Critical strengths

1.The temporal supervision objective is simple and empirically well supported. Table 3 shows that combining hard supervision for near tokens with soft distribution matching for distant tokens provides higher accuracy and AUP than near-only, all-hard, or all-soft supervision on the evaluated MATH and HumanEval settings. Figure 5 further separates the effects of privileged trajectory construction and temporal-aware supervision.

2.The evaluation is reasonably broad and includes several useful controls. Results cover two backbones and four reasoning and code benchmarks, and Appendix D reports five-run means and standard deviations. The matched-data experiments in Table 8 retrain dParallel and d3LLM using the same examples and optimization budget, reducing an important training-data confound.

Critical weaknesses

1.The methodological novelty appears moderate. Privileged-context distillation, trajectory distillation, and hard/soft knowledge-distillation objectives are established ideas. TAD combines them through a time-to-reveal partition that is intuitive and empirically effective, but the paper does not yet sharply establish what fundamentally new principle is introduced beyond adapting these ingredients to the reveal order of diffusion language models. This limits my assessment of the contribution’s significance despite the positive results.

2.The claimed superiority of the accuracy–parallelism frontier is not fully established under matched conditions. In Table 8, TAD-S is nearly tied with matched d3LLM on Dream (59.6 accuracy and 3.81 TPF versus 59.5 and 3.76), while on LLaDA it trades higher accuracy for lower TPF (49.9 and 5.74 versus 48.6 and 6.21). The matched models are not compared using complete accuracy–TPF curves, matched AUP, H200 throughput, or run-level uncertainty. Moreover, the headline AUP values are computed from two operating points and can therefore depend substantially on the selected accelerated configuration. The reported results support a useful trade-off, but not yet a uniformly superior matched frontier.

Questions for authors
  1. Under exactly matched training data and optimization budgets, can the authors report complete accuracy–TPF curves, AUP, H200 throughput, and run-level uncertainty for TAD, d3LLM, and dParallel?

  2. What do the authors consider the principal methodological novelty beyond combining privileged-context distillation with established trajectory-distillation and hard/soft distillation objectives? In particular, how does TAD differ conceptually from the closest privileged-context and trajectory-distillation methods rather than only in its temporal partition rule?

  3. How different are the answer-conditioned teacher states used during training from states visited by the student during inference? Does the accuracy–parallelism trade-off change when training includes student-generated states?

  4. What exact-match or semantic deduplication was performed between the training sources and GSM8K, MATH/MATH500, HumanEval, and MBPP?

4 comments

We appreciate the reviewer’s thoughtful questions and we address each concern below.

W1 & Q2: Methodological novelty and distinction from existing distillation methods.

We agree that context self-distillation and the combination of hard and soft targets have been explored in autoregressive language models, and we acknowledge that these prior works have provided valuable inspiration for our research. However, transferring these techniques to diffusion language models is far from straightforward, as dLLMs exhibit fundamentally different generation dynamics from autoregressive models: multiple masked positions are predicted in parallel, and different tokens have different temporal reveal horizons.

The key contribution of TAD is the identification and exploitation of a dLLM-specific supervision principle: the appropriate supervision signals should depend on each token's temporal distance to reveal. This distinguishes TAD from closely related approaches in several ways. Privileged-context distillation primarily improves the quality of teacher supervision by providing additional information to the teacher, but does not address how supervision should vary across masked positions at different temporal horizons. Trajectory-distillation methods such as d3LLM exploit teacher-generated trajectories to guide student training and parallel decoding, whereas TAD further uses the token-level time-to-reveal information within these trajectories to determine the appropriate supervision target for each masked position. Similarly, our contribution is not simply mixing hard and soft losses, but establishing a principled, temporally conditioned assignment of these objectives. Through extensive experiments and ablations, we show that privileged-information-guided trajectory construction and temporal-aware supervision provide complementary benefits. We believe this work provides useful insights into how to design distillation objectives tailored to the unique generation dynamics of dLLMs, and we sincerely thank the reviewer for recognizing the value of this direction.

W2 & Q1: Matched accuracy–parallelism frontier and additional evaluation.

To address this concern, we further evaluate multiple operating points and provide Accuracy–TPF curves across four benchmarks on both LLaDA and Dream backbones. We have added the corresponding experimental results, including the matched Accuracy–TPF curves, AUP comparisons, and H200 throughput measurements, to the revised PDF (Figures 9 and 10, and Tables 8 and 10). The resulting mean AUP across four benchmarks is:

MethodLLaDA–8BDream–7B
dParallel–Matched168.3135.3
d3LLM–Matched247.9192.8
TAD–Q222.3162.9
TAD–S256.6193.9

Notably, TAD–S achieves the highest mean AUP on both backbones, outperforming the strongest matched baseline, d3LLM, by 3.51% on LLaDA and 0.57% on Dream.

We also measure actual throughput on NVIDIA H200 using GSM8K–CoT:

MethodLLaDA (TPS)Dream (TPS)
dParallel–Matched227.1197.7
d3LLM–Matched409.8309.2
TAD–Q336.4205.7
TAD–S376.9288.4

Although d3LLM achieves the highest raw throughput, TAD–S retains 92.0% and 93.3% of its throughput on LLaDA and Dream, respectively, while achieving higher GSM8K AUP on LLaDA (647.8 vs. 629.8) and comparable GSM8K AUP on Dream (354.9 vs. 356.9).

AuthorsOpenReview ↗

Rebuttal-2

Q3: Distribution gap between teacher training states and student inference states.

We conduct an analysis on 100 sampled trajectories to compare the answer-conditioned teacher training states with student inference states. Specifically, Base Student refers to the original LLaDA before TAD training, while TAD Student refers to the TAD-Q. Both students generate from the same unprivileged prompts. We compare their generated states against the answer-conditioned teacher trajectories. We use two metrics to quantify token-level similarity:

● Token Agreement: The fraction of identical token IDs among positions that have been revealed in intermediate states.

● Final Token Agreement: The fraction of position-wise identical token IDs in the final generated answers, excluding EOS and padding tails. The results are as follows:

MetricTeacher vs. Base StudentTeacher vs. TAD Student
Token Agreement ↑4.6%36.1%
Final Token Agreement ↑3.9%20.2%

TAD improves intermediate-state token agreement by 31.5% and final-answer token agreement by 16.3%. These results demonstrate that TAD reduces the content-level gap between teacher training states and student inference states.

Regarding whether incorporating student-generated states during training changes the accuracy–parallelism trade-off, our existing ablation in Figure 5 investigates this question. Specifically, in the w/o-GT setting, the training trajectories are generated by the student model using question-only prompts. We compare this setting against training with privileged-information-guided trajectories. The results demonstrate that incorporating privileged information into trajectory construction can improve accuracy-parallelism trade-off.

Q4: Training–test data contamination and deduplication.

Ensuring zero train–test overlap is a standard prerequisite in our experimental pipeline, and we performed strict exact-match deduplication between all training sources and evaluation benchmarks prior to training.

Thank you for the detailed response. It has addressed most of my concerns, and I will therefore increase my rating.

Thank you very much for your positive assessment and for increasing the rating, and we sincerely appreciate your time and thoughtful review.

Reviewer_iSrtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The main target of this paper is to keep or improve the accuracy of dLLMs while improving the latency through self-distillation. Key contributions include: (1) feeding teacher model with ground-truth responses for better trajectory constrution; (2) spliting the output tokens into near and distant future and supervising them with different strict levels (more forgiveness for distant tokens to avoid local minimum). Experiments demonstrate the effectiveness of both contributions and the proposed TAD achieves the state-of-the-art speed-accuracy trade-off compared with other acceleration-oriented methods.

Critical strengths
  1. Performance is really good compared with competitive counterparts such as d3LLM and dParallel.
  2. Motivation is clear and easy to follow. Experiments proof the claims well.
Critical weaknesses
  1. Context self-distillation is well-established in autoregressive models. Verifying it also works in dLLMs seems less exciting.
  2. Same applies to mixing hard and soft targets.
  3. In general, I have no problem with re-applying well-studied techniques in a different domain as long as the final combination achieves great performance and the community can still benefit from it knowing what has been tried and what is working. Happy to discuss with other reviewers on this.
Questions for authors

Please refer to the weakness section.

1 comment

We sincerely thank the reviewer for the positive assessment of our work and greatly appreciate the reviewer’s acknowledgment of both the empirical effectiveness and the motivation of TAD.

We agree that context self-distillation and the combination of hard and soft targets have been explored in autoregressive language models, and we acknowledge that these prior works have provided valuable inspiration for our research. However, transferring these ideas to diffusion language models is not straightforward. Unlike autoregressive models, dLLMs generate tokens through iterative denoising, where multiple masked positions are predicted in parallel and exhibit different temporal dynamics. Consequently, directly applying existing distillation objectives does not necessarily lead to effective supervision for parallel decoding. We have therefore devoted substantial effort to investigating how to design appropriate training objectives for dLLMs, including how to assign different supervision signals to tokens at different reveal horizons. Through extensive experiments and ablations, we identify a complementary combination of privileged-information-guided trajectories and temporal-aware hard/soft supervision that achieves a strong accuracy–parallelism trade-off. Notably, this temporal-aware design is also appreciated by Reviewer K5Ec, who highlighted that applying "different training targets depending on when each token will be decoded" offers "a simple way to balance speed and accuracy."

We believe this work provides useful insights into how to design distillation objectives tailored to the generation dynamics of dLLMs, and we sincerely thank the reviewer for recognizing the value of this direction.

Reviewer_whUxOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TAD, a temporal-aware trajectory self-distillation framework for improving the accuracy-parallelism trade-off in diffusion language models. A frozen teacher uses reference answers as privileged information to generate decoding trajectories. The student is trained with hard cross-entropy on tokens revealed within a look-ahead window and soft KL divergence to the teacher's distributions on more distant tokens. Experiments demonstrate improved average accuracy-parallelism trade-offs.

Critical strengths

The method is well motivated and the paper is well written. The LLaDA results, together with the loss and window ablations, provide convincing evidence of an improved accuracy-parallelism trade-off.

Critical weaknesses

The trajectory sampling settings and the preprocessing needed to exclude reference answers from student inputs are not sufficiently documented, limiting reproducibility (see Questions 1 and 2).

Questions for authors
  1. Trajectory sampling. Section 3.2 specifies greedy token-by-token generation via argmax, whereas prepare/llada_gen_math_traj.py defaults to temperature=0.7 and introduces randomness through add_gumbel_noise(), potentially changing both token choices and the unmasking order. What motivates the use of stochastic sampling? Which sampling settings were used in the reported experiments? Have you compared the results of greedy and stochastic sampling?

  2. Reference answers in student inputs. The paper states that the student does not receive the reference answer. However, the generation script above saves a prompt containing the question, reference answer, and "After understanding..." instruction, which TrajectoryDataset.__getitem__() in LLaDA/train_llada.py directly concatenates with the current trajectory state. Without additional preprocessing, the student therefore receives "question + reference answer + teacher instruction + current generation state," contrary to the stated setup. Is there an undocumented preprocessing step that removes the reference answer and teacher-specific instruction? If not, how does their inclusion during training affect evaluation without reference answers and the paper's conclusions?

1 comment

We sincerely thank the reviewer for the positive and encouraging assessment of our work. We are particularly grateful for the recognition that our method TAD is "well motivated" and the manuscript is "well written." We also greatly appreciate the reviewer highlighting that our LLaDA experiments, along with the loss and window ablation studies, "provide convincing evidence of an improved accuracy-parallelism trade-off."

We also thank the reviewer for identifying two implementation details that were not sufficiently documented. We clarify them below.

Q1: Trajectory Sampling.

We clarify that all trajectories used in our reported experiments were generated with temperature = 0.0. Although the released script has a default temperature of 0.7, this default was overridden in all experiments reported in the paper. This choice is consistent with common practice in dLLM evaluation, particularly for mathematical reasoning and code generation. Since dLLMs predict multiple masked positions in parallel, stochastic token sampling may introduce additional sampling noise across positions. Using greedy token selection provides more deterministic trajectory construction. We did not conduct a separate stochastic-sampling comparison, as stochastic trajectory sampling is not a component of our proposed method and is beyond the scope of this work. We will clarify the actual temperature setting and distinguish token sampling from position selection in the revised manuscript and released code.

Q2: Reference Answers in Student Inputs.

We clarify that the raw trajectory records undergo an additional preprocessing step before being loaded into TrajectoryDataset. This step removes the reference-answer content and teacher-specific instruction from the prompt used for student training. Consequently, the actual student input consists only of the original question and the corresponding intermediate diffusion state. The student is never directly exposed to the reference answer or teacher-specific instruction as part of its input. We will clarify this distinction and the teacher–student data construction pipeline in the revised manuscript to improve reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.