acceptodds
Under review as a conference paper at ICLR 2027

From Ranking Skeletons to Teacher Predictions: Knowledge Distillation along Continuous Trajectories

Abstract

Standard logit-based knowledge distillation trains the student to match the teacher's predictions, transferring class rankings and response magnitudes through a single supervision target. This formulation offers limited flexibility in how these two forms of knowledge are distilled. To address this limitation, we propose Ranking-guided Trajectory Knowledge Distillation (RTKD), which extends teacher supervision to a continuous family of targets. We first construct a ranking skeleton that retains the teacher's class ordering while discarding its original response gaps, and then connect the normalized skeleton and teacher prediction via a spherical trajectory. Along this path, class ordering is preserved, while relative response magnitudes gradually change from the skeleton to the teacher prediction. Distillation is performed by integrating the student's discrepancies from targets along the trajectory, with a teacher-dependent prior assigning greater weight to targets closer to the teacher endpoint. An adaptive Huber penalty is applied to geodesic discrepancies to limit the influence of large errors. We evaluate RTKD on two image classification benchmarks, CIFAR-100 and ImageNet-1K, and conduct ablation studies to assess the contributions of its components. On ImageNet-1K, RTKD improves Top-1 accuracy from 70.49% with standard KD to 73.15% when distilling ResNet50 into MobileNet-V1. The code will be released soon.

Primary area:
learning theory
Keywords:
Knowledge Distillation, Logits Trajectory, Spherical Geometry

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_WrNgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Basically, this paper studies how to construct better teacher supervision. The main insight is that the class ordering in the original teacher logits can also serve as a supervision signal. Based on this idea, the authors propose RTKD, which first constructs an ordinal reference vector from the teacher logits, called the ranking skeleton, and connects it to the normalized teacher logits through an order-preserving spherical trajectory. The student is then trained to minimize a geodesic loss over this trajectory, using a teacher-dependent prior and an adaptive Huber geodesic loss. Experiments cover CIFAR-100, ImageNet, and heterogeneous teacher–student architectures.

Critical strengths

The idea of separating class ordering from the relative magnitudes of the logits is well motivated. Using spherical interpolation to preserve class ordering along the trajectory is also intuitive and straightforward.

Critical weaknesses

The main concern about the paper’s soundness is that, the ablation results in Table 5 show only modest gains from trajectory integration. Specifically, full RTKD achieves 77.48%, compared with 77.00% for the teacher endpoint and 77.14% for the prior mean point. These results do not convincingly establish the advantage of the full RTKD objective over directly using the normalized teacher logits or other simple alternatives.

Questions for authors

I'm wondering whether normalized linear interpolation achieves performance similar to spherical interpolation? Specifically, one could linearly interpolate between the normalized teacher logits and the ranking skeleton, then normalize the resulting vector to unit norm. Both approaches preserve unit norm and class ordering. Moreover, under nondegenerate conditions, they trace the same spherical arc with different parameterizations.

This comparison would therefore help clarify whether the gains come from the spherical trajectory itself or from the particular parameterization and weighting used by RTKD.

3 comments
AuthorsOpenReview ↗

Response to Reviewer WrNg

Thank you for the helpful comments. We added repeated-run comparisons with simpler targets and directly tested NLERP to clarify both the benefit of trajectory integration and the role of its parameterization.

Response to Comment 1: Gains over the teacher endpoint and prior mean point.

We agree that the original single-run ablation did not show how consistent the gains were. We now compare Full RTKD with the Teacher endpoint and Prior mean over five seeds (40–44). The results are CIFAR-100 Top-1 accuracy, reported as mean ± sample SD.

Teacher → StudentTeacher endpointPrior meanFull RTKD
ResNet32×4 → ResNet8×476.586 ± 0.10376.976 ± 0.27677.392 ± 0.336
ResNet32×4 → ShuffleNetV176.398 ± 0.15176.722 ± 0.18576.890 ± 0.161
ResNet32×4 → ShuffleNetV276.896 ± 0.09677.146 ± 0.17377.650 ± 0.198
WRN-40-2 → WRN-16-275.986 ± 0.12175.852 ± 0.13476.184 ± 0.252
VGG13 → VGG874.732 ± 0.14574.574 ± 0.12474.912 ± 0.261

Full has the highest mean accuracy in all five pairs. On ResNet8×4, it improves over the Teacher endpoint by 0.806 percentage points and over Prior mean by 0.416 points. On ShuffleNetV2, the gains are 0.754 and 0.504 points, respectively. Full outperforms both alternatives in every seed on these two pairs.

These repeated-run results strengthen the evidence that integration provides a useful benefit beyond the tested single-target alternatives, with the size of the gain varying across teacher–student pairs.

The Prior-mean variant evaluates one point, γ(Ep[τ])\gamma(\mathbb{E}_p[\tau]), whereas Full averages the loss along the trajectory. Because the loss is nonlinear, these are generally different objectives. The experiments above test whether that difference helps the student in practice.

Response to Question: Does normalized linear interpolation perform similarly to SLERP?

We agree with the reviewer: NLERP and SLERP follow the same spherical arc under nondegenerate conditions. Their difference is how the interpolation parameter moves along that arc. We therefore tested both ordinary NLERP and NLERP with a Jacobian correction that preserves the SLERP integration measure.

The following results use seeds 42–44, with ResNet32×4 as the teacher.

InterpolationResNet8×4ShuffleNetV1
SLERP77.453 ± 0.43576.923 ± 0.167
NLERP77.013 ± 0.30776.810 ± 0.173
NLERP with Jacobian correction77.210 ± 0.14576.900 ± 0.241

The results are close, particularly on SV1, where corrected NLERP differs from SLERP by only 0.023 percentage points.

For the normalized linear path η(α)=((1−α)q+αut)/∥(1−α)q+αut∥\eta(\alpha)=((1-\alpha)q+\alpha u_t)/\|(1-\alpha)q+\alpha u_t\|, the change to the SLERP parameter requires the Jacobian

{\theta\|(1-\alpha)q+\alpha u_t\|^2}.$$ Including this factor in both the weighted loss and the prior normalization gives the same continuous objective. We verified this directly using fixed-logit FP64 diagnostics: | Student | NLERP gradient relative error | Jacobian-corrected NLERP gradient relative error | |---|---:|---:| | ResNet8×4 | 6.00e-2 | 1.09e-7 | | ShuffleNetV1 | 6.72e-3 | 6.02e-7 | Errors are measured relative to SLERP and reported as fractions. The sharp reduction after correction confirms that the integration measure explains the objective-level difference. We will clarify that our contribution is the weighted supervision over a ranking-preserving target family. SLERP provides a convenient constant-angular-speed parameterization, while correctly reweighted NLERP offers another implementation of the same continuous objective.

Dear Authors,

Thank you for conducting the additional experiments and addressing my concerns.

However, I remain unconvinced that the results demonstrate a sufficiently substantial benefit of Full RTKD over the other baselines. The improvement over the Teacher endpoint remains marginal, reaching at most 0.806% Top-1 accuracy across the five evaluated settings, but with higher variance.

After considering the authors’ response and the other reviewers’ comments, I have decided to maintain my original score.

Best, Reviewer WrNg

AuthorsOpenReview ↗

Response to Reviewer WrNg

Thank you for taking the time to review our additional experiments and explanations. We appreciate your candid assessment and understand your remaining concern about the magnitude and variability of the gains. Your feedback has helped us clarify the empirical comparisons and the scope of our claims. Thank you again for your careful evaluation and constructive suggestions.

Reviewer_4AvxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes RTKD, a logit-based distiller that extends teacher supervision from a single prediction to a continuous family of targets. A ranking skeleton is constructed by replacing teacher logits with rank-based scores, preserving class ordering while removing the original response gaps. After centering and normalization, the skeleton and teacher logits are connected by a spherical trajectory that preserves class ordering. The distillation objective integrates the angular discrepancies between the student and targets along this trajectory, using a teacher-dependent prior to emphasize targets closer to the teacher prediction and an adaptive Huber penalty to limit the influence of large discrepancies. Extensive experimental results are provided to demonstrate the effectiveness of RTKD.

Critical strengths
  1. The presentation is clear.
  2. The method is novel and reasonable in general.
  3. The visual analysis in Figure 4 is interesting.
Critical weaknesses
  1. Equation 1 is one way to construct an order-preserving target (ranking skeleton), but there are many alternatives. Justifications of using the form in equation 1 is needed.
  2. The general idea of using order-preserving targets as supervisions makes intuitive sense. However, the constructed trajectory in equation 3 is only a subset of all the order-preserving targets. Following the general idea, it could be better to consider the entire set of order-preserving targets (or more trajectories within the region) for integrating the loss.
  3. In equation 4, chord distance is used to measure target proximity. Considering that the angular separation is used in equations 3 and 5 to measure proximity on the unit sphere, it seems more intuitive to be consistent and use angular separation in equation 4 as well.
Questions for authors
  1. How is the integral in equation 7 computed in the actual implementation? How much overhead does it involve?
  2. Why do the authors choose to use Huber penalty over geodesic discrepancy as the integrand within the distillation loss? A more familiar and simple alternative is the cosine distance between student and a point on the trajectory. Would it be better to use it instead?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4Avx

Thank you for the feedback and the useful suggestions. We tested these alternatives to better understand which parts of the design matter in practice.

Response to Weakness 1: We chose the linear skeleton because it is simple and adds no extra parameter: consecutive ranks receive equally spaced scores, followed by centering and normalization. It keeps the full class order without copying the teacher's original response gaps. Other order-preserving choices are also possible.

We compare linear, logarithmic, power, and Top1 skeletons while keeping the rest of the trajectory objective the same. The power variant uses exponent two. The Top1 construction deliberately discards non-top ordering and therefore tests a different question from the strictly rank-preserving alternatives. We report this ablation about skeleton and path on ResNet32×4→ResNet8×4 on CIFAR-100 over seeds 42–44 as below.

ConstructionTop-1 accuracy
Linear full-ranking skeleton77.453 ± 0.435
Logarithmic full-ranking skeleton76.843 ± 0.116
Power full-ranking skeleton77.217 ± 0.244
Top1 skeleton75.687 ± 0.313
Three paths: linear, logarithmic, power77.247 ± 0.225

Linear has the highest observed mean, while the power alternative remains close. The larger gap to Top1, 1.767, with positive differences in all three seeds, provides stronger evidence for retaining full-ranking structure than for the universal optimality of the linear mapping. The linear form is therefore a simple choice that works well in our tests.

Response to Weakness 2: We agree that a single arc covers only part of the order-preserving region. We chose it because it gives a manageable family of targets connected to the teacher, with a clear parameterization and weighting rule. Integrating over the entire region would also require deciding how to weight its targets. Keeping the same class order alone does not tell us which response-gap patterns deserve more weight.

We tested three trajectories using linear, logarithmic, and power skeletons while retaining a total budget of 128 quadrature nodes. The paths receive 43, 43, and 42 nodes; each path is normalized separately and their losses are averaged. This construction reaches 77.247 ± 0.225%, compared with 77.453 ± 0.435% for the single linear trajectory. The paired differences favor the single path in all three seeds, although the mean gap is modest at 0.207.

Adding these paths did not improve performance under the same node budget. This supports the single trajectory as a simple, effective choice for the tested setting. Other paths, weighting rules, or larger target regions may still be worth exploring.

Response to Weakness 3: Both distances express proximity on the same sphere, and chord distance c=2sin(d/2)c=2 sin(d/2) is monotonic in angular distance d over the relevant range. They agree on which targets are closer to the teacher, but assign different weights after exponentiation. The prior decides how much weight each target receives, while the discrepancy measures how well the student matches it. These roles can use different distance functions, but we agree that angular distance is a natural alternative to test. Therefore, we compare different prior and penalty alternatives over seeds 42–44, and the Top-1 accuracy is reported as below. Each row changes only the stated choice relative to the Full reference in this comparison.

VariantResNet8×4ShuffleNetV1
Full: chord prior and adaptive geodesic Huber77.453 ± 0.43576.923 ± 0.167
Angular-distance prior77.023 ± 0.30976.773 ± 0.136
Cosine penalty, 2(1−cos d)76.580 ± 0.33277.030 ± 0.442
Squared geodesic penalty76.807 ± 0.28476.800 ± 0.255

The chord-prior reference exceeds the angular-prior mean by 0.430 on ResNet8×4 and 0.150 on SV1. The means favor the original choice, although three seeds leave considerable uncertainty.

AuthorsOpenReview ↗

Response to Reviewer 4Avx (Part2)

Response to Question 1: Numerical implementation and overhead of the integral.

Thank you for asking about the implementation. We use Gauss–Legendre quadrature with 128 nodes on [0,1][0,1]. The quadrature weights are included in both the weighted loss and the normalization of the teacher-dependent prior:

H_\delta(d_s(\tau_i))} {\sum_{i=1}^{Q}a_i e^{-\sqrt{\kappa}c_t(\tau_i)}}.$$ Here, $\tau_i$ and $a_i$ are the quadrature nodes and weights. Teacher and student logits are computed once per batch and reused across all nodes, so integration adds logit-space calculations without additional network forward passes. We checked numerical convergence against a 1024-node reference. At 128 nodes, the mean relative gradient errors are $7.30\times10^{-8}$ for ResNet8×4 and $3.62\times10^{-7}$ for ShuffleNetV1 in FP64 checkpoint diagnostics, supporting the accuracy of the approximation. We also measured training time and peak allocated GPU memory. The following results use seed 42 and average training time over epochs 2–9, excluding validation. | Method | R8×4 time (s/epoch) | SV1 time (s/epoch) | R8×4 peak memory (MiB) | SV1 peak memory (MiB) | |---|---:|---:|---:|---:| | Standard KD | 29.291 | 30.241 | 286.86 | 608.09 | | Teacher endpoint | 24.329 | 37.221 | 286.86 | 608.09 | | Full RTKD, Q=128 | 24.055 | 36.712 | 286.86 | 608.09 | Full RTKD and the Teacher endpoint have similar observed training times in these runs. Relative to standard KD, Full takes 21.40% longer on SV1, while the ResNet8×4 run records a shorter time. Since the profiling runs show runtime variability, we report these as observed wall-clock measurements rather than a precise estimate of integration overhead. Peak allocated memory is unchanged in the measured window. At inference, only the student is used, with no trajectory computation. **Response to Question 2: Why Huber rather than cosine discrepancy.** Thank you for suggesting this comparison. We chose Huber because it combines quadratic behavior for small angular errors with a capped slope for larger errors. Its derivative with respect to angular discrepancy is $2\min(d,\delta)$, allowing precise alignment near the target while limiting the influence of large discrepancies. We directly compared adaptive geodesic Huber with cosine and squared geodesic losses on two teacher–student pairs. The cosine variant uses $2(1-\cos d)$, which has the same leading-order behavior as $d^2$ near zero. Results below are Top-1 accuracy over seeds 42–44, with ResNet32×4 as the teacher. | Penalty | ResNet8×4 | ShuffleNetV1 | |---|---:|---:| | Adaptive geodesic Huber | 77.453 ± 0.435 | 76.923 ± 0.167 | | Cosine, 2(1−cos d) | 76.580 ± 0.332 | 77.030 ± 0.442 | | Squared geodesic | 76.807 ± 0.284 | 76.800 ± 0.255 | Huber improves over cosine by 0.873 percentage points on ResNet8×4. On ShuffleNetV1, the two are close, with cosine having a 0.107-point higher mean. Huber also has a higher mean than squared geodesic loss on both pairs. These results support Huber as an effective default while showing that cosine is a competitive alternative, particularly on SV1. The comparisons use the same reference distillation weight within each pair. We will include this ablation and clarify that the benefit of the penalty choice.

We thank the authors for their thorough reply. However, one question remains. As implied in my original Weakness 2, is it possible to intergate over a surface of order-preserving targets instead of just an arc? If that can be achieved, the paper will be much more interesting.

AuthorsOpenReview ↗

Response to Reviewer 4Avx (Step2)

Thank you for clarifying this suggestion. Our previous experiment considered separate arcs. We agree that integrating over an order-preserving surface is an interesting extension. Following the suggestion, we have implemented and conducted an initial evaluation of this direction.

Response: We construct a two-dimensional spherical patch from the normalized teacher prediction and the linear and power ranking skeletons. Integration includes the surface-area Jacobian in both the weighted loss and prior normalization, using 16×8=12816\times8=128 quadrature nodes. On CIFAR-100 with ResNet32×4 → ResNet8×4, the results over seeds 42–44 are reported as below.

Target constructionTop-1 accuracy
Logarithmic-skeleton arc76.84 ± 0.12
Power-skeleton arc77.22 ± 0.24
Three trajectories77.25 ± 0.23
Original linear-skeleton arc77.45 ± 0.44
Order-preserving surface77.27 ± 0.09

The surface retains the original linear-arc Huber threshold, whereas the three-trajectory variant uses separate path-dependent thresholds.

These initial results demonstrate that continuous surface supervision is feasible and achieves accuracy close to the trajectory variants. The surface also shows the lowest observed seed-to-seed variation in this comparison, with a standard deviation of 0.09 percentage points and accuracies ranging from 77.18% to 77.35% across the three runs. This provides an encouraging indication of consistent performance alongside the broader target coverage.

We therefore view surface-based supervision as a promising direction for future work, particularly through further study of surface geometry, target weighting, and adaptive thresholds. We will include this preliminary experiment and discuss these directions in the revised manuscript. Thank you for encouraging us to explore this extension.

Reviewer_9UQBOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Ranking-guided Trajectory Knowledge Distillation (RTKD), which extends logit-based distillation from matching a single teacher prediction to learning from a continuous family of targets. It constructs a ranking skeleton that preserves the teacher's class ordering while replacing the original response gaps with rank-based scores. A spherical trajectory connects this skeleton to the normalized teacher prediction, and the student is trained by integrating its geodesic discrepancy along the trajectory with a teacher-dependent prior and an adaptive Huber penalty. The method is evaluated on CIFAR-100 and ImageNet-1K across multiple teacher-student architecture pairs. The results are competitive in several settings. For example, on ImageNet-1K with ResNet50 as teacher and MobileNet-V1 as student, RTKD obtains 73.15% Top-1 accuracy, compared with 70.49% for standard KD. The ablations provide useful evidence for the proposed components. However, the stability of the ranking construction, the rationale for the adaptive Huber threshold, and the numerical implementation of the continuous integral require further clarification.

Critical strengths

see Questions For Authors

Critical weaknesses

see Questions For Authors

Questions for authors
  1. Main Comments & Constructive Feedback 1.1 Sensitivity of the Ranking Skeleton to Near-Tied Logits • The Setup: The ranking skeleton is defined as s_c = C + 1 − r_c, where r_c is the descending rank of class c. Exactly tied teacher logits are handled by assigning average ranks. • The Concern: The concern is not that a frozen teacher's logits change spontaneously during student optimization. For an identical fixed input under deterministic evaluation, the teacher output remains fixed. Rather, the ranking operation is discontinuous around ties. For example, [5.001, 5.000, 1.000] and [5.001, 5.002, 1.000] differ only slightly, but the first two classes exchange ranks, producing a different ranking skeleton. Handling exact ties does not address near-tied scores. Since the skeleton is an endpoint of the subsequent spherical trajectory, a rank swap may also change intermediate trajectory targets and the distillation loss. The magnitude of this downstream effect is not established. • Suggestion: Please analyze the frequency of near-tied class logits and the stability of the ranking skeleton under small, controlled perturbations. If stochastic input augmentation is used, it would also be useful to check whether near-tied classes change order across augmented views. Possible measures include the proportion of class pairs with small logit gaps, rank consistency (e.g., Kendall's rank correlation), and changes in the normalized skeleton or trajectory targets. Please clarify that this is a sensitivity issue of the ranking construction, not a claim that a frozen teacher changes its output for the same fixed input. 1.2 Theoretical Justification of the Adaptive Huber Threshold • The Setup: The sample-dependent threshold is defined as δ(x) = max(θ(x), δ_min), where θ is the angle between the normalized ranking skeleton q and normalized teacher prediction u_t. The Huber penalty, however, is applied to d_s(τ), the angular discrepancy between the student prediction u_s and a trajectory target γ(τ). • The Concern: The threshold is determined by the angular distance between two teacher-derived endpoints, whereas the penalized quantity d_s(τ) measures the distance from the student prediction to a point on the trajectory. These are different geometric quantities. The fact that distances between points on the teacher-defined trajectory are bounded by its angular span θ does not imply that the student-to-trajectory discrepancy is also bounded by θ, because the student prediction need not lie on that trajectory. Thus, the manuscript has not fully justified why θ is an appropriate scale for setting the Huber transition point. This does not necessarily mean that the threshold is ineffective; its rationale needs to be better established. • Suggestion: Please provide a clearer geometric or optimization-based explanation for using the teacher-endpoint angle to set the Huber threshold. It would be helpful to report the distributions of θ and d_s(τ) during training, and the proportion of student-to-target discrepancies in the quadratic and linear regions of the Huber loss. The fixed-threshold ablation is useful, but additional evidence across representative teacher-student pairs would help establish whether the adaptive rule is consistently beneficial. 1.3 Numerical Approximation of the Continuous Trajectory Integral • The Setup: The distillation objective is defined as a continuous integral over τ ∈ [0, 1]. The teacher-dependent prior also contains a normalization integral over the same interval. • The Concern: A practical implementation must approximate these integrals numerically, but the manuscript does not clearly specify the quadrature method, the number and locations of integration points, or their weights. It is also unclear how the normalization integral in the teacher-dependent prior is approximated. Different discretization schemes may produce different estimates of the loss. In addition, the spherical interpolation formula contains sin(θ) in the denominator. Even if the exactly degenerate case θ = 0 is defined, the implementation for very small nonzero θ should be described to establish numerical stability. • Suggestion: Please specify the numerical integration rule (e.g., a Riemann-sum, trapezoidal, or Gaussian quadrature approximation), the number and placement of nodes, their weights, and endpoint handling. Please explain how the prior's normalization integral is computed and how very small θ is handled. An ablation over the number of integration nodes would clarify whether performance depends on a particular approximation. Please also report training-time and memory overhead. Since student logits can generally be computed once and reused for the trajectory nodes, distinguish extra logit-space calculations from any additional network forward passes.
  2. Suggestions on Experiments 2.1 Robustness to Teacher Quality and Ranking Reliability Both the ranking skeleton and the trajectory are derived from the teacher's predicted class ordering. The teacher-dependent prior favors targets closer to the teacher prediction, but it does not explicitly estimate whether that prediction is correct. If the teacher produces an incorrect or unstable ranking, the trajectory may preserve and transfer that ordering to the student. Please evaluate RTKD with teachers of different accuracy levels or controlled perturbations to teacher logits, to identify the conditions under which trajectory-based supervision is beneficial. 2.2 Statistical Reliability of the Reported Improvements Some improvements over strong logit-based baselines are relatively small, while run-to-run variation is not clearly reported for all key comparisons. Please provide multiple independent runs, preferably with matched seeds and training budgets, and report means with standard deviations or confidence intervals. Claims of superiority should be qualified when observed differences are comparable to run-to-run variation. 2.3 Clarification of Computational Overhead Although inference uses the student network alone, trajectory calculations and numerical integration may increase training time and memory use. Please report per-epoch or total training time and peak GPU memory, comparing with standard KD under consistent hardware, data pipelines, and training schedules. Distinguish training-time overhead from inference-time cost. 2.4 Hyperparameter Sensitivity The method depends on λ, κ, and δ_min. Please clarify the hyperparameter selection protocol and whether the same settings are used across experiments. A broader sensitivity analysis on representative teacher-student pairs would strengthen the evidence that the method does not rely on narrowly tuned settings.
  3. Minor Corrections Small-angle implementation: Please state how spherical interpolation is evaluated when θ is very small but nonzero, including any numerical approximation or threshold used. Visualization interpretation: The t-SNE and polar plots are mainly qualitative. Please avoid drawing stronger conclusions about class separation or predictive structure than these visualizations alone can support. Reproducibility: Please consolidate the settings for λ, κ, δ_min, numerical integration, optimizer, and random seeds in the implementation details or supplementary material. Overall Assessment: RTKD presents a coherent approach to organizing logit-level supervision through a ranking skeleton and a continuous trajectory. The empirical results are encouraging in several settings. However, the stability of the ranking construction, the rationale for the adaptive Huber threshold, and the practical numerical implementation of the integral should be clarified before the robustness, theoretical grounding, and reproducibility of the method can be fully assessed.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9UQB

Thank you for these detailed suggestions. We have added measurements for ranking sensitivity, the Huber threshold, numerical integration, and training cost.

Response to Comment 1.1: Sensitivity to near-tied logits and augmented views. Thank you for this suggestion. We added controlled-noise and augmentation diagnostics on 256 fixed examples. At a standardized logit-gap threshold of 0.01, near ties account for 0.7451% of all class pairs and 30.3149% of adjacent ranked pairs.

For controlled perturbations, Gaussian noise is scaled by each sample’s teacher-logit standard deviation. Results are averaged over three noise repetitions.

Noise σKendall τ-bTop-5 overlapR8×4SV1
0.00010.999918100.00%0.04%0.09%
0.0010.99917899.84%0.28%0.62%
0.010.99164198.80%1.72%3.69%

At σ=0.01, Full’s average loss changes are only 0.36% and 0.4321%, and gradient cosine similarities remain 0.999823 and 0.999196 for ResNet8×4 and SV1, respectively. These results show that the observed rank swaps have small average effects on the objective and largely preserve gradient direction.

We also compared two augmented views while keeping student logits fixed. Augmentation produces larger ranking changes: Kendall τ-b is 0.8172, Top-5 overlap is 78.67%, and 28/256 examples change their top prediction.

StudentMethodRelative loss changeRelative gradient changeGradient cosine similarity
ResNet8×4Full22.60%53.10%0.8494
ResNet8×4Teacher endpoint33.53%51.66%0.8453
ShuffleNetV1Full25.21%65.33%0.7789
ShuffleNetV1Teacher endpoint25.73%59.54%0.8127

RTKD preserves the teacher’s ordering within each view, while different views can produce different rankings. These diagnostics directly quantify both the small downstream effects of near-tie perturbations and the stronger supervision changes caused by augmentation.

Response to Comment 1.2: Geometric rationale and empirical behavior of the adaptive Huber threshold. We agree with the reviewer: the teacher-endpoint angle θ is not an upper bound on the student-to-trajectory discrepancy. The student need not lie on the trajectory. Our reason for using θ is that it measures how much the targets vary along the trajectory for that sample. The positive lower bound keeps the threshold from becoming too small when the trajectory is short.

For the penalty used here,$ ∂Hδ(d)/∂d = 2 min(d,δ)$. Choosing δ=max(θ,δmin)δ=max(θ,δ_min) makes errors below this reference scale quadratic and limits the slope with respect to angular discrepancy above it. This choice is meant to control how the loss responds to large errors. It does not imply that student errors are bounded by θ or that all logit gradients share a universal bound.

We show the huber diagnostics at the evaluated checkpoints as below. Region occupancy is averaged using the normalized integration weights, not counted as a fraction of images.

DiagnosticResNet8×4ShuffleNetV1
Mean trajectory angle θ1.10370.54459
Mean adaptive threshold δ1.10370.54459
Prior-weighted mean student–target distance0.46010.44615
Quadratic-region weight93.09%68.59%
Linear-region weight6.91%31.41%

Both branches are used, and their contributions differ substantially between the evaluated configurations. In the SV1 diagnostic, 79 examples have all sampled nodes in the quadratic region, 69 have all nodes in the linear region, and 108 span both. This confirms that the transition is used in practice.

We also compare the adaptive rule against three fixed thresholds on both pairs over seeds 42–44 as below.

Huber thresholdResNet8×4ShuffleNetV1
Fixed 0.176.357 ± 0.21876.177 ± 0.060
Fixed 0.2576.933 ± 0.25576.477 ± 0.136
Fixed 0.576.980 ± 0.22676.707 ± 0.191
Adaptive77.453 ± 0.43576.923 ± 0.167

The adaptive rule has the highest mean in both pairs, exceeding the best tested fixed threshold by 0.473 and 0.217. This supports the adaptive rule as a useful default.

AuthorsOpenReview ↗

Response to Reviewer 9UQB (Part2)

Response to Comment 1.3: Numerical integration and small-angle stability.

We use Q-point Gauss–Legendre quadrature on [0,1][0,1], with Q=128Q=128 for the default Full objective. Given the standard nodes ξi\xi_i and weights wiw_i on [−1,1][-1,1], we set τi=(ξi+1)/2\tau_i=(\xi_i+1)/2 and ai=wi/2a_i=w_i/2. The nodes lie strictly inside the interval, so the Full objective does not require explicit endpoint evaluation. Endpoint ablations evaluate their designated endpoints directly.

The same quadrature rule is used for the weighted loss and the prior-normalization denominator:

\sum_{i=1}^{Q} a_i \exp\left[-\sqrt{\kappa}\,c_t(\tau_i)\right] H_{\delta}\left(d_s(\tau_i)\right) }{ \sum_{j=1}^{Q} a_j \exp\left[-\sqrt{\kappa}\,c_t(\tau_j)\right] }.$$ For numerical stability, the normalized weights are computed using a softmax over $\log(a_i)-\sqrt{\kappa}\,c_t(\tau_i)$, which includes the quadrature weights. For small angles, we construct an orthogonal basis from the teacher endpoint and the skeleton, and evaluate the targets as $$\gamma(\tau)= \cos(\phi)\,u_t+\sin(\phi)\,v, \qquad \phi=(1-\tau)\theta.$$ The angle is computed using atan2 from the orthogonal residual and the endpoint inner product. When the residual norm is at most $10^{-6}$, the path is collapsed to the teacher endpoint. This avoids directly dividing by $\sin(\theta)$. For the squared geodesic discrepancy, let $$x=\frac{\|u_s-\gamma(\tau)\|_2^2}{2}.$$ We use a small-distance series when $x<10^{-3}$ and a linear continuation near the antipodal limit when $x>2-10^{-4}$. The table below reports the mean relative gradient error against the $Q=1024$ reference, measured in FP64 at the evaluated checkpoints. Errors are fractions, not percentages. | Q | ResNet8×4 | ShuffleNetV1 | |---:|---:|---:| | 4 | 2.38e-3 | 3.33e-4 | | 8 | 2.06e-5 | 8.29e-5 | | 16 | 4.89e-6 | 2.32e-5 | | 32 | 1.25e-6 | 5.11e-6 | | 64 | 3.49e-7 | 1.46e-6 | | 128 | 7.30e-8 | 3.62e-7 | | 256 | 2.33e-8 | 8.45e-8 | At $Q=128$, the corresponding mean absolute loss errors are $8.93\times10^{-11}$ and $1.37\times10^{-10}$. Comparing the $Q=1024$ and $Q=2048$ references gives mean relative gradient differences of $1.33\times10^{-9}$ and $6.02\times10^{-9}$. These results show that $Q=128$ is sufficiently accurate on the tested checkpoints. We also report the full-training node-count ablation on ResNet32×4→ResNet8×4 over seeds 42–44. | Nodes Q | Top-1 accuracy | |---:|---:| | 16 | 77.283 ± 0.105 | | 32 | 77.043 ± 0.233 | | 64 | 77.267 ± 0.172 | | 128 | 77.453 ± 0.435 | The mean accuracies span 0.410 percentage points and do not vary monotonically with $Q$. Reducing the node count therefore does not cause a large accuracy drop in this setting. **Response to Comment 2.1 Teacher quality and ranking reliability.** We agree that being close to the teacher does not tell us whether the teacher is correct. If its ranking is wrong, the trajectory can carry that error into the supervision. We therefore evaluated controlled teacher-logit perturbations during training, using per-example noise standard deviation σ times the teacher-logit standard deviation. Top-1 accuracy of ResNet32×4→ResNet8×4 on CIFAR-100 over seeds 42–44 is reported as below. Full retains a positive accuracy margin at each tested noise level. | Noise σ | Full RTKD | Teacher endpoint | Full−Teacher | |---:|---:|---:|---:| | 0 | 77.453 ± 0.435 | 76.660 ± 0.026 | +0.793 | | 0.05 | 77.127 ± 0.099 | 76.413 ± 0.114 | +0.713 | | 0.10 | 77.087 ± 0.040 | 76.510 ± 0.279 | +0.577 | | 0.20 | 77.090 ± 0.075 | 76.437 ± 0.211 | +0.653 |
AuthorsOpenReview ↗

Response to Reviewer 9UQB (Part3)

Response to Comment 2.2: Training-time and memory overhead. We agree that keeping inference unchanged does not mean training has no extra cost. We reports averages training time over epochs 2–9 with seed as 42 for all methods. Timing covers the training loop, including data loading, and excludes validation. The comparison is a short-run profiling result, not total time for a converged 240-epoch training run. Percentage changes are computed relative to the corresponding KD entry as below.

MethodResNet8×4 s/epochChange vs KDSV1 s/epochChange vs KD
Standard KD29.291—30.241—
Teacher endpoint24.329−16.94%37.221+23.08%
Full Q1626.057−11.04%34.999+15.73%
Full Q3225.285−13.68%36.995+22.33%
Full Q6424.031−17.96%37.580+24.27%
Full Q12824.055−17.88%36.712+21.40%

SV1 Full Q128 has a measured 21.40% increase relative to KD. Conversely, the ResNet8×4 records show lower elapsed times for RTKD and no monotonic relationship with Q. These timings do not show that RTKD is inherently faster or that integration has no cost. Implementation differences and runtime fluctuations are still mixed into the measurements. Therefore, we also report the peak allocated GPU memory in the matched epochs 2–9 window as below.

StudentStandard KDTeacher endpointFull Q16/Q32/Q64/Q128
ResNet8×4286.86 MiB286.86 MiB286.86 MiB
ShuffleNetV1608.09 MiB608.09 MiB608.09 MiB

No increase in the recorded peak allocated memory was observed in this window. At inference, only the student network is used, so RTKD adds no inference module or trajectory computation. During training, the additional work is in the logit-space objective; teacher and student predictions are reused across nodes.

Response to Comment 2.3: Hyperparameter sensitivity and selection. We agree that hyperparameter sensitivity needs to be shown explicitly. We varied one parameter at a time on two representative pairs, with the results below.

PairParameterTested valuesTop-1 accuracies in the same order
ResNet32×4 → ResNet8×4κ10, 20, 30, 40, 5077.02, 76.74, 77.90, 77.22, 76.46
ResNet32×4 → ResNet8×4λ30, 40, 50, 60, 7077.59, 77.41, 77.90, 77.32, 77.45
ResNet32×4 → ShuffleNetV1κ10, 20, 30, 40, 5076.50, 77.42, 77.02, 76.68, 76.94
ResNet32×4 → ShuffleNetV1λ20, 30, 40, 50, 6076.54, 76.73, 77.02, 76.73, 76.73
ResNet32×4 → ShuffleNetV1δ_min0.1, 0.2, 0.3, 0.4, 0.577.02, 77.02, 77.00, 76.85, 76.68

The κ sweeps span 1.44 and 0.92, with different best tested values for the two pairs (30 and 20). The λ sweeps span 0.58 and 0.48. The SV1 δ_min sweep spans 0.34, with little change between 0.1 and 0.3. Several nearby settings work well, the results show that the method is not too sensitive to these parameters.

Response to Question 1: Small-angle implementation.

The orthogonal-basis evaluation, residual threshold of 1e-6, and small-distance series described under Comment 1.3 specify the handling of very small nonzero angles. We include these details alongside the quadrature rule so that reviewer can reproduce how the objective is computed.

Response to Question 2: Interpretation of t-SNE and polar plots.

We agree that these projections are qualitative. We will describe them as illustrations of the learned output structure and avoid treating visual separation, radial bands, or projected directions as quantitative evidence of improved class separation, calibration, or causal mechanisms. Our claims about effectiveness will be based on accuracy comparisons and controlled ablations.

Response to Question 3: Consolidated reproducibility information.

Thank you for this suggestion. We will add a consolidated implementation-details table to the revised manuscript, listing the settings for each dataset and teacher–student pair. This will include the loss weight, prior parameter, Huber threshold floor, optimizer, learning-rate schedule, batch size, training epochs, and random seeds. We will also specify the quadrature rule and node count, small-angle handling, and checkpoint-selection criterion. And the code will be open-sourced as soon as possible.

Reviewer_KBhnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper combines normalized teacher logits and a ranking skeleton through a spherical trajectory to form a supervision signal for knowledge distillation of the student model. The method provides flexibility to combine ranking and magnitude information. When distilling ResNet50 into MobileNet-V1, the method (RTKD) improves the Top-1 accuracy to 73.15%, which is the best result among the compared logit-based methods.

Critical strengths
  1. The proposal methods is solid and can be implemented easily.
  2. This paper performance comprehensive experiments to show the superio of the proposed method.
  3. Ablation study is performed across different factors of the proposed method
  4. The proposed method uses full information of the spherical trajectory between the normalized teacher logits and the ranking skeleton.
Critical weaknesses
  1. The major weakness is the concern raised by the ablation study. The normalized teacher endpoint already reaches 77.00%, while the full RTKD reaches 77.48%, so the improvement brought by the ranking skeleton and the trajectory is very minor (0.48 pp, without reported variance). However, the spherical trajectory between the normalized teacher logits and the ranking skeleton is the core novelty of the entire paper. The significance of the spherical trajectory and the ranking skeleton can be further elaborated.

  2. Using the ranking skeleton alone (71.01%) performs even worse than standard KD (73.33% from Table 1) and the student trained from scratch (72.50%), and the reason behind this can be further explained. The motivation for the spherical trajectory between the ranking skeleton and the normalized teacher logits can also be better justified. Since the normalized teacher logits already preserve the class ranking and only remove the overall logit scale, it is unclear what additional information the ranking skeleton provides.

Questions for authors

In Table 2 and 3, what is the performance of using only the normalized teacher endpoint (i.e., the Teacher-endpoint variant in Table 5)? This would show whether the ranking skeleton and the trajectory help in the heterogeneous settings.

2 comments
AuthorsOpenReview ↗

Response to Reviewer KBhn

Thank you for the helpful comments. We agree that the contribution of the trajectory needs clearer evidence. Below, we provide repeated-run comparisons, explain the role of the ranking skeleton, and report the requested results for heterogeneous pairs. All experiments are tested on RTX 4090 GPUs.

Response to Comment 1: We agree that a single result does not show how consistent the gain is. We now compare Full RTKD, the teacher endpoint, and prior mean over five seeds as below.

Table R1. CIFAR-100 Top-1 accuracy over seeds 40–44, reported as mean ± std.

Teacher → StudentFull RTKDT‘s endpointPrior mean
ResNet32×4 → ResNet8×477.392 ± 0.33676.586 ± 0.10376.976 ± 0.276
ResNet32×4 → ShuffleNetV176.890 ± 0.16176.398 ± 0.15176.722 ± 0.185
ResNet32×4 → ShuffleNetV277.650 ± 0.19876.896 ± 0.09677.146 ± 0.173
WRN-40-2 → WRN-16-276.184 ± 0.25275.986 ± 0.12175.852 ± 0.134
VGG13 → VGG874.912 ± 0.26174.732 ± 0.14574.574 ± 0.124

For the main ResNet32×4→ResNet8×4 ablation, Full improves over the Teacher endpoint by 0.806 and over Prior mean by 0.416, with positive differences in every listed seed. The ShuffleNetV2 results provide a second setting with consistent improvements over both alternatives. These results give stronger support for using the full target distribution instead of only the teacher endpoint or the tested intermediate point.

Response to Comment 2: The skeleton is not intended to replace the teacher prediction on its own. It removes the teacher's response gaps and replaces them with rank-based spacing, including among low-ranked classes. This loses information about how strongly the teacher prefers one class over another, which helps explain the poor skeleton-only result. The new three-seed experiment shows the same large gap between skeleton-only supervision and the full trajectory.

Table R2. ResNet32×4→ResNet8×4 target ablation over seeds 42–44.

Supervision targetTop-1 accuracy
Skeleton endpoint71.870 ± 0.383
Two-endpoint mixture76.557 ± 0.234
Teacher endpoint76.660 ± 0.026
Prior mean point77.063 ± 0.205
Full trajectory77.453 ± 0.435

Full improves over the two-endpoint mixture by 0.897 on average, with positive differences in all three seeds. Together with the five-seed comparisons, (the classical KD and from-scratch values, 73.33% and 72.50%), this supports using the skeleton as a reference within a trajectory that stays close to the teacher.

Response to Comment 3: The reviewer is right that the ranking is already present in the teacher logits. The skeleton changes how that information is presented to the student. The normalized teacher keeps its original response-gap pattern, while the skeleton uses rank-based spacing with the same class order. The trajectory connects these two patterns, allowing the gaps to change while keeping the ordering consistent.

Trajectory skeletonSeed 42Seed 43Seed 44Mean ± std
Full-ranking skeleton77.9077.4377.0377.453 ± 0.435
Top-1 skeleton75.9575.3475.7775.687 ± 0.313

The key question is whether this way of presenting the teacher's information helps the student learn. The comparisons with the Teacher endpoint and Prior mean test this directly. The Full versus Top1-skeleton result, 77.453% versus 75.687% across three seeds on ResNet8×4, further supports retaining the complete ordering rather than only the top-class identity. It does not imply that every low-ranked class is equally informative.

AuthorsOpenReview ↗

Response to Reviewer KBhn (Part 2.)

Response to Question: Teacher-endpoint performance in the heterogeneous settings of Tables 2 and 3

Thank you for this suggestion. For CIFAR-100, the comparisons on ResNet32×4→ShuffleNetV1 and ResNet32×4→ShuffleNetV2 are included in Tables R1–R2 above. Full RTKD outperforms the Teacher endpoint in all five seeds for both pairs.

For ImageNet-1K, we report the individual results for seeds 42 and 914 below. We also include Prior mean as another simple alternative. Each entry gives Top-1 / Top-5 accuracy (%).

Teacher → StudentSeedTeacher endpointPrior meanFull RTKD
ResNet50 → MobileNetV14273.140 / 91.47873.214 / 91.58273.316 / 91.610
ResNet50 → MobileNetV191473.296 / 91.51673.292 / 91.55273.422 / 91.540
ResNet34 → ResNet184271.736 / 90.58071.786 / 90.60671.940 / 90.780
ResNet34 → ResNet1891471.904 / 90.70872.024 / 90.76672.190 / 90.880

Full improves Top-1 over the Teacher endpoint by 0.151 for MobileNetV1 and 0.245 for ResNet18, and over Prior mean by 0.116 and 0.160. Each Top-1 paired difference is positive in both seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.