- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Basically, this paper studies how to construct better teacher supervision. The main insight is that the class ordering in the original teacher logits can also serve as a supervision signal. Based on this idea, the authors propose RTKD, which first constructs an ordinal reference vector from the teacher logits, called the ranking skeleton, and connects it to the normalized teacher logits through an order-preserving spherical trajectory. The student is then trained to minimize a geodesic loss over this trajectory, using a teacher-dependent prior and an adaptive Huber geodesic loss. Experiments cover CIFAR-100, ImageNet, and heterogeneous teacher–student architectures.
Critical strengths
The idea of separating class ordering from the relative magnitudes of the logits is well motivated. Using spherical interpolation to preserve class ordering along the trajectory is also intuitive and straightforward.
Critical weaknesses
The main concern about the paper’s soundness is that, the ablation results in Table 5 show only modest gains from trajectory integration. Specifically, full RTKD achieves 77.48%, compared with 77.00% for the teacher endpoint and 77.14% for the prior mean point. These results do not convincingly establish the advantage of the full RTKD objective over directly using the normalized teacher logits or other simple alternatives.
Questions for authors
I'm wondering whether normalized linear interpolation achieves performance similar to spherical interpolation? Specifically, one could linearly interpolate between the normalized teacher logits and the ranking skeleton, then normalize the resulting vector to unit norm. Both approaches preserve unit norm and class ordering. Moreover, under nondegenerate conditions, they trace the same spherical arc with different parameterizations.
This comparison would therefore help clarify whether the gains come from the spherical trajectory itself or from the particular parameterization and weighting used by RTKD.
3 comments
Dear Authors,
Thank you for conducting the additional experiments and addressing my concerns.
However, I remain unconvinced that the results demonstrate a sufficiently substantial benefit of Full RTKD over the other baselines. The improvement over the Teacher endpoint remains marginal, reaching at most 0.806% Top-1 accuracy across the five evaluated settings, but with higher variance.
After considering the authors’ response and the other reviewers’ comments, I have decided to maintain my original score.
Best, Reviewer WrNg
Response to Reviewer WrNg
Thank you for taking the time to review our additional experiments and explanations. We appreciate your candid assessment and understand your remaining concern about the magnitude and variability of the gains. Your feedback has helped us clarify the empirical comparisons and the scope of our claims. Thank you again for your careful evaluation and constructive suggestions.
Response to Reviewer WrNg
Thank you for the helpful comments. We added repeated-run comparisons with simpler targets and directly tested NLERP to clarify both the benefit of trajectory integration and the role of its parameterization.
Response to Comment 1: Gains over the teacher endpoint and prior mean point.
We agree that the original single-run ablation did not show how consistent the gains were. We now compare Full RTKD with the Teacher endpoint and Prior mean over five seeds (40–44). The results are CIFAR-100 Top-1 accuracy, reported as mean ± sample SD.
Full has the highest mean accuracy in all five pairs. On ResNet8×4, it improves over the Teacher endpoint by 0.806 percentage points and over Prior mean by 0.416 points. On ShuffleNetV2, the gains are 0.754 and 0.504 points, respectively. Full outperforms both alternatives in every seed on these two pairs.
These repeated-run results strengthen the evidence that integration provides a useful benefit beyond the tested single-target alternatives, with the size of the gain varying across teacher–student pairs.
The Prior-mean variant evaluates one point, γ(Ep[τ]), whereas Full averages the loss along the trajectory. Because the loss is nonlinear, these are generally different objectives. The experiments above test whether that difference helps the student in practice.
Response to Question: Does normalized linear interpolation perform similarly to SLERP?
We agree with the reviewer: NLERP and SLERP follow the same spherical arc under nondegenerate conditions. Their difference is how the interpolation parameter moves along that arc. We therefore tested both ordinary NLERP and NLERP with a Jacobian correction that preserves the SLERP integration measure.
The following results use seeds 42–44, with ResNet32×4 as the teacher.
The results are close, particularly on SV1, where corrected NLERP differs from SLERP by only 0.023 percentage points.
For the normalized linear path η(α)=((1−α)q+αut)/∥(1−α)q+αut∥, the change to the SLERP parameter requires the Jacobian
{\theta\|(1-\alpha)q+\alpha u_t\|^2}.$$ Including this factor in both the weighted loss and the prior normalization gives the same continuous objective. We verified this directly using fixed-logit FP64 diagnostics: | Student | NLERP gradient relative error | Jacobian-corrected NLERP gradient relative error | |---|---:|---:| | ResNet8×4 | 6.00e-2 | 1.09e-7 | | ShuffleNetV1 | 6.72e-3 | 6.02e-7 | Errors are measured relative to SLERP and reported as fractions. The sharp reduction after correction confirms that the integration measure explains the objective-level difference. We will clarify that our contribution is the weighted supervision over a ranking-preserving target family. SLERP provides a convenient constant-angular-speed parameterization, while correctly reweighted NLERP offers another implementation of the same continuous objective.