acceptodds
Under review as a conference paper at ICLR 2027

TraCo-KD: Compiling Teacher Training Trajectories into Logit Knowledge Distillation

Abstract

Logit-based knowledge distillation (KD) has emerged as a lightweight alternative to feature-based distillation. Recent logit-based methods mainly improve the alignment of teacher and student outputs, with supervision drawn from a teacher at convergence. However, teacher classification accuracy peaks at the end of training, while relations among non-target classes continue to evolve, making it difficult for a teacher at any single training stage to provide both. Motivated by this observation, we propose Trajectory Compilation Knowledge Distillation (TraCo-KD), which compiles the teacher's full training trajectory offline into a frozen operator acting on converged logits. Specifically, TraCo-KD decomposes logits into a target margin, a non-target scale, and non-target relations, aggregates per-sample trajectory targets for each component, and fits a low-rank operator mapping converged logits to these targets. During distillation, the corrected teacher outputs are combined with the original outputs through per-sample mixing and used directly in the existing loss, requiring only a single teacher forward pass and no additional inference cost. Extensive experiments show that TraCo-KD achieves the best performance among logit-based methods and improves feature-based distillation as a plug-in.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge Distillation, Training Trajectory, Dark Knowledge, Inter-class Relations, Efficient Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hzewOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Logit-based knowledge distillation usually matches a student to a fully converged teacher. This paper observes that teacher accuracy peaks at convergence, while non-target class relations evolve differently across training stages. TraCo-KD aggregates per-sample trajectory targets offline (target margin, non-target scale, and non-target relations), fits a low-rank operator to map converged logits to those targets, and mixes corrected and original teacher logits during distillation. It requires only one converged-teacher forward pass and introduces no additional student parameters or inference cost. On CIFAR-100 and ImageNet, It achieves competitive performance against existing KD methods and improves feature-based methods as a plug-in.

Critical strengths
  1. Clear motivation. Fig. 1 and the intermediate-checkpoint study (Sec. 4.5, Table 5) provide an interesting observation that teacher accuracy and non-target relational structure evolve differently. It motivated the use of information from multiple training stages rather than a single checkpoint.

  2. Clean plug-in design. Offline compilation → frozen operator → logit correction (Fig. 2) provides a practical way to incorporate trajectory information into existing KD objectives. The approach does not modify the student architecture or increase its inference cost, and its compatibility with feature-based KD is demonstrated in Table 6.

Critical weaknesses
  1. Unclear experimental provenance and fairness (Sec. 4.1; Tables 1–3). Many baseline results appear identical to those reported in previous KD papers, but the authors do not clarify which methods were re-evaluated and which results were taken from prior work. There are also potential model-version inconsistencies in the MobileNet-V1/V2 settings, particularly for OFD, whose publicly available implementation has reported compatibility issues with MobileNetV2. These concerns raise questions about the reliability and fairness of the comparisons. The authors should clearly identify the sources of all baseline results and verify that the models and evaluation conditions are consistent.

  2. Incomplete comparison with recent and trajectory-based KD methods (Sec. 2; Tables 1–5). Recent logit-based methods, such as SKD (CVPR 2026), are missing from the comparison. Also, although the authors evaluate individual intermediate checkpoints, they do not compare TraCo-KD with existing multi-checkpoint KD methods or checkpoint ensembles. Without stronger and more recent baselines, it remains unclear whether TraCo-KD represents a meaningful advance over the current state of the art.

  3. Unclear contribution of trajectory compilation over the base loss (Sec. 3.1; Tables 1 and 4). The proposed method uses a relatively strong base distillation objective combining predictive distribution, inter-sample, and inter-class alignment. This base loss already achieves 77.22% accuracy, outperforming LSKD (76.62%) without trajectory compilation. The full TraCo-KD improves the base loss by only 0.88 percentage points. Thus, the reported superiority may partly stem from the underlying loss rather than trajectory compilation itself.

  4. Missing efficiency evaluation (Sec. 4). Efficiency is a major advantage of logit-based KD, but the paper provides no quantitative comparison of training time, GPU memory, or computational cost. TraCo-KD additionally requires teacher trajectory logging and offline operator fitting, which may introduce substantial overhead. The authors should report both distillation-time efficiency and offline preparation costs to justify the practical efficiency of the method.

  5. Limited analysis of operator robustness (Secs. 3–4). The operator is fitted using the trajectory of a retrained teacher but applied to separately released teacher weights. The impact of this mismatch is not evaluated. Sensitivity to operator rank, the number of recorded epochs, and augmented or unseen inputs is also insufficiently investigated. These analyses are important for understanding how reliably the learned operator captures and transfers trajectory information.

  6. Limited cross-task generalization (Sec. 4). All experiments focus on image classification using CIFAR-100 and ImageNet. Although multiple teacher–student architectures are evaluated, the applicability of TraCo-KD to other tasks remains unclear. Object detection is a relevant extension commonly considered in KD research, and evaluation on a benchmark such as COCO would help establish whether the proposed trajectory correction generalizes beyond classification.

Questions for authors

Q1. Were the baseline results in Tables 1–3 reproduced under the same experimental conditions or copied from previous papers? How were the MobileNet-V1/V2 settings and the reported OFD compatibility issues handled?

Q2. How much improvement does the trajectory operator provide when applied to standard KD losses such as KD, DKD, and LSKD? Can the authors compare against more recent methods such as SKD (CVPR 2026)?

Q3. If only a few intermediate checkpoints are available, how much of the full-trajectory gain remains? How does this compare with a simple multi-checkpoint ensemble?

Q4. What are the offline storage, computational, and memory costs of trajectory logging and operator fitting, particularly on ImageNet? How do these costs compare with a standard distillation run?

Q5. How sensitive is the operator to the mismatch between retrained and released teacher weights, its rank, and the number of recorded epochs? Does it maintain its effectiveness on augmented inputs or different tasks such as object detection?

Reviewer_BdmTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A key insight of this manuscript is that the relational structure among non-target classes is distributed across the teacher’s training trajectory, and no single checkpoint captures both predictive accuracy and this structure well. The authors therefore aggregate three components across the trajectory (i.e., a target-class margin, a non-target scale, and non-target relations), and combine them with the teacher’s converged logits to construct the final supervision targets.

Critical strengths

The observation that non-target class relations and predictive accuracy evolve inconsistently during teacher's training is useful. And compressing training trajectory information into a reusable operator is also a well-motivated approach to improving teacher supervision.

Critical weaknesses

Although the online procedure requires only one teacher forward pass, it does not sufficiently establish an end-to-end efficiency advantage. The method also requires retraining the teacher to obtain its training trajectory, recording that trajectory, and fitting two additional modules. These costs also raise concerns about scalability, particularly for large language models with large output vocabularies, where multiple next-token predictions may be valid.

Questions for authors
  1. In Sec 3.3, the authors introduce a Tukey-fence filter on per-sample jumps between consecutive epochs to remove optimization transients, such as those associated with learning-rate decays. How does this filter distinguish transient fluctuations from meaningful changes following a learning-rate decay? Please report when and how many records each filter removes, and provide separate ablations disabling each of the two filters.

  2. Sec 3.4 does not sufficiently justify the introduction of UV⊤UV^\top. Why is this low-rank correction necessary? How much improvement comes from relation, margin, and scale correction individually? And evaluate whether the low-rank correction remains effective across diverse datasets and teacher-student architectures. Ablations isolating these components would help clarify their respective contributions.

Reviewer_GwHyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TraCo-KD aggregates a teacher's training trajectory into per-example targets for the target-class margin, non-target scale, and standardized non-target relations. It fits a compact operator that maps converged logits and labels to these targets, together with a predictor that mixes corrected and original logits. Both modules are frozen during student training. Experiments cover 12 teacher–student pairs on CIFAR-100 and two on ImageNet, plus combinations with feature distillation. The method generally improves accuracy while requiring one teacher forward pass during distillation and adding no student inference cost.

Critical strengths
  1. The compilation interface is practically appealing. The operator is small, can be shared across students, and handles augmented inputs without replaying multiple teachers. For CIFAR-100, the reported operator and mixing predictor have only 3,600 and 353 parameters. This offers a useful way to amortize trajectory information across student architectures, provided the offline cost is acceptable.

  2. The results cover several architectures and include one useful controlled ablation. TraCo-KD leads the listed logit methods on 11 of 12 CIFAR-100 pairs and both ImageNet pairs. Table 4 holds the base loss fixed and improves accuracy from 77.22 to 78.10. This is evidence of a benefit beyond the chosen base objective, although repeatability is not reported and the ablation covers only one pair.

Critical weaknesses
  1. The incremental contribution is not supported by a sufficiently convincing practical benefit. As §2 acknowledges, early-stopped teachers, intermediate checkpoints, curricula, and trajectory ensembles already address the limitations of endpoint-only supervision. The new contribution is therefore chiefly the compilation mechanism. Most gains are modest: calculated against the strongest listed logit baseline in each CIFAR-100 column, the mean improvement is about 0.53 percentage points and the median is 0.38; only two pairs exceed one point, and one loses 0.58. ImageNet gains over the strongest listed methods are 0.36 and 0.27 points. The paper does not specify repeat counts or report across-seed variability, so the reliability of these small differences is unknown. Meanwhile, the protocol requires an additional teacher training run and trajectory recording, without runtime/storage measurements. Without matched comparisons to simpler trajectory-based alternatives, the paper does not establish that the added procedure offers sufficient value.

  2. The proposed explanation and advantage over checkpoint selection are not established by the controls. The operator changes margin, scale, and relations together, without component ablations. Figures 1 and 3 demonstrate proximity to the chosen historical aggregate, rather than independently validating its usefulness. The matched-loss improvement is 0.88 points on one pair; the larger 1.48-point headline comparison also changes the base loss. Table 6 lacks feature-method-plus-uncorrected-logit controls. In Table 5, intermediate teachers come from a retrained trajectory, whereas the endpoint baseline uses released weights. Without that trajectory's own endpoint, training-run differences remain a confound. Five checkpoints on one pair cannot establish an inherent limitation of checkpoint selection. The operator's transfer between teacher runs is also unquantified, and the complete base-loss formulas and weights are missing. Taken together, these gaps leave both the source of the gains and the claimed conceptual advance unresolved.

Questions for authors
  1. Reliability and practical value: How many runs underlie each reported accuracy, and what are the means and standard deviations across seeds, particularly on ImageNet? What are the total offline cost and the distillation overhead, and how does the accuracy–cost comparison change when simpler trajectory aggregation methods receive the same base loss?

  2. Protocol clarification: What student accuracy results from using the final checkpoint of the retrained trajectory in Table 5? How do results compare when fitting and distillation use the same teacher run versus the released teacher?

  3. Attribution evidence: Under the same base loss, what happens when correcting only relations versus only margin/scale? For Table 6, does the gain remain relative to adding the same uncorrected logit loss?

  4. Reproducibility clarification: Please specify the three base-loss terms and weights, fitting learning rates/batch sizes, trajectory-recording views, Tukey jump statistic and threshold, and oracle temperature. What evidence supports the operator's accuracy on augmented views and its transfer to released weights?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.