- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Logit-based knowledge distillation usually matches a student to a fully converged teacher. This paper observes that teacher accuracy peaks at convergence, while non-target class relations evolve differently across training stages. TraCo-KD aggregates per-sample trajectory targets offline (target margin, non-target scale, and non-target relations), fits a low-rank operator to map converged logits to those targets, and mixes corrected and original teacher logits during distillation. It requires only one converged-teacher forward pass and introduces no additional student parameters or inference cost. On CIFAR-100 and ImageNet, It achieves competitive performance against existing KD methods and improves feature-based methods as a plug-in.
Critical strengths
-
Clear motivation. Fig. 1 and the intermediate-checkpoint study (Sec. 4.5, Table 5) provide an interesting observation that teacher accuracy and non-target relational structure evolve differently. It motivated the use of information from multiple training stages rather than a single checkpoint.
-
Clean plug-in design. Offline compilation → frozen operator → logit correction (Fig. 2) provides a practical way to incorporate trajectory information into existing KD objectives. The approach does not modify the student architecture or increase its inference cost, and its compatibility with feature-based KD is demonstrated in Table 6.
Critical weaknesses
-
Unclear experimental provenance and fairness (Sec. 4.1; Tables 1–3). Many baseline results appear identical to those reported in previous KD papers, but the authors do not clarify which methods were re-evaluated and which results were taken from prior work. There are also potential model-version inconsistencies in the MobileNet-V1/V2 settings, particularly for OFD, whose publicly available implementation has reported compatibility issues with MobileNetV2. These concerns raise questions about the reliability and fairness of the comparisons. The authors should clearly identify the sources of all baseline results and verify that the models and evaluation conditions are consistent.
-
Incomplete comparison with recent and trajectory-based KD methods (Sec. 2; Tables 1–5). Recent logit-based methods, such as SKD (CVPR 2026), are missing from the comparison. Also, although the authors evaluate individual intermediate checkpoints, they do not compare TraCo-KD with existing multi-checkpoint KD methods or checkpoint ensembles. Without stronger and more recent baselines, it remains unclear whether TraCo-KD represents a meaningful advance over the current state of the art.
-
Unclear contribution of trajectory compilation over the base loss (Sec. 3.1; Tables 1 and 4). The proposed method uses a relatively strong base distillation objective combining predictive distribution, inter-sample, and inter-class alignment. This base loss already achieves 77.22% accuracy, outperforming LSKD (76.62%) without trajectory compilation. The full TraCo-KD improves the base loss by only 0.88 percentage points. Thus, the reported superiority may partly stem from the underlying loss rather than trajectory compilation itself.
-
Missing efficiency evaluation (Sec. 4). Efficiency is a major advantage of logit-based KD, but the paper provides no quantitative comparison of training time, GPU memory, or computational cost. TraCo-KD additionally requires teacher trajectory logging and offline operator fitting, which may introduce substantial overhead. The authors should report both distillation-time efficiency and offline preparation costs to justify the practical efficiency of the method.
-
Limited analysis of operator robustness (Secs. 3–4). The operator is fitted using the trajectory of a retrained teacher but applied to separately released teacher weights. The impact of this mismatch is not evaluated. Sensitivity to operator rank, the number of recorded epochs, and augmented or unseen inputs is also insufficiently investigated. These analyses are important for understanding how reliably the learned operator captures and transfers trajectory information.
-
Limited cross-task generalization (Sec. 4). All experiments focus on image classification using CIFAR-100 and ImageNet. Although multiple teacher–student architectures are evaluated, the applicability of TraCo-KD to other tasks remains unclear. Object detection is a relevant extension commonly considered in KD research, and evaluation on a benchmark such as COCO would help establish whether the proposed trajectory correction generalizes beyond classification.
Questions for authors
Q1. Were the baseline results in Tables 1–3 reproduced under the same experimental conditions or copied from previous papers? How were the MobileNet-V1/V2 settings and the reported OFD compatibility issues handled?
Q2. How much improvement does the trajectory operator provide when applied to standard KD losses such as KD, DKD, and LSKD? Can the authors compare against more recent methods such as SKD (CVPR 2026)?
Q3. If only a few intermediate checkpoints are available, how much of the full-trajectory gain remains? How does this compare with a simple multi-checkpoint ensemble?
Q4. What are the offline storage, computational, and memory costs of trajectory logging and operator fitting, particularly on ImageNet? How do these costs compare with a standard distillation run?
Q5. How sensitive is the operator to the mismatch between retrained and released teacher weights, its rank, and the number of recorded epochs? Does it maintain its effectiveness on augmented inputs or different tasks such as object detection?