- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cross-dimensional embedding distillation by distinguishing representation reduction from coordinate resolution. The authors propose ORBIT-KD, which combines PCA-based reduction of teacher embeddings with corpus-level orthogonal Procrustes alignment to construct student-conditioned targets. The student is trained with a cosine endpoint loss and an auxiliary H0 persistence loss. In experiments with several teacher–student pairs and corpora, ORBIT-KD obtains the highest nine-task average among the compared methods. Controlled analyses show how coordinate alignment affects the distillation results.
Critical strengths
1. Careful analysis of coordinate resolution.
- The paper provides well-controlled experiments that isolate the effect of coordinate alignment while keeping the teacher’s relational geometry unchanged.
- Comparisons with random rotations, different references, and different fitting scopes give convincing evidence that geometrically equivalent targets can lead to different distillation outcomes. I think this observation is useful for practitioners who use reduced teacher targets.
2. Simple method with consistent average improvements.
- ORBIT-KD uses closed-form orthogonal alignment. It does not require a trainable projector or online teacher inference during student training.
- The method obtains the highest nine-task average in all six main settings. It also shows positive results with larger teachers and a task-independent corpus.
Critical weaknesses
1. Limited novelty and missing discussion of closely related work.
- The theoretical results mostly follow from known properties of orthogonal transformations and classical Procrustes analysis. Therefore, the use of terms such as “Theorem” and “exact initialization-headroom identity” might give the impression that the theoretical contribution is larger than it actually is.
- EdgePoint2 (Yao et al., 2025), although in the domain of local feature descriptors, appears to already combine a Gram-preserving low-rank reduction of teacher representations with orthogonal Procrustes alignment to the student for cross-dimensional distillation. This work does not seem to be discussed in the paper. Its per-minibatch alignment is similar to the “per-minibatch fits” variant in Appendix C.2. Therefore, a clearer explanation of what is new (e.g., corpus-level fitting, fixed targets with periodic refresh, the H0 loss) would be needed.
2. The empirical evidence is not yet sufficient to support the practical claims.
- Necessity of full-rank resolution: Figure 13(d) shows that rank-1 alignment already accounts for about 72% of the improvement over the unreoriented target. The authors mention the role of the dominant mean direction, but comparisons with simpler alignment methods (e.g., centering, or aligning only the mean direction) seem necessary to show the additional value of full-rank Procrustes alignment.
- Evaluation scope: Retrieval benchmarks are not included, although retrieval is presented as an important motivation. In addition, the main distillation corpora are drawn from the training splits of the evaluation tasks.
- Size of improvements: Some improvements over TALAS are small (e.g., +0.07 and +0.27 points on the 100K corpus), and only three training seeds are reported without significance tests. The 20-epoch experiments show that the gain persists, but it also becomes smaller over training, and there is no comparison with TALAS under longer or compute-matched training.
- Efficiency: The reported step-time speedups exclude teacher-cache construction and coordinate refresh. The refresh time is reported separately (under 0.7 minutes per run), but this is similar to the whole optimization time in configuration (c) (0.64 minutes). Also, since TALAS uses adaptive SAM, it is difficult to attribute the speed difference to the proposed approach itself.
Questions for authors
Clarification questions
- Relationship to EdgePoint2: Could the authors clarify the methodological differences between ORBIT-KD and the orthogonal Procrustes distillation in EdgePoint2? In particular, which contributions go beyond corpus-level fitting and periodically updated fixed targets?
- Role of the dominant direction: Figure 13(d) indicates that rank-1 alignment explains most of the improvement. How does ORBIT-KD compare with a simpler method that aligns only the mean or dominant direction? How sensitive is the improvement to the anisotropy of the pretrained student embeddings?
- Learned projector baselines: In Appendix C.3, the trainable linear-map baselines perform much lower than even the unreoriented PCA target. Could the authors explain how these baselines were optimized? Also, how would a trainable orthogonal projector initialized with the Procrustes solution compare with the proposed fixed-target approach?
- Longer training: Do the improvements over TALAS remain under longer training schedules or comparable computational budgets? The current 20-epoch experiments show the persistence of the alignment benefit, but they do not address the advantage over the strongest baseline.
- End-to-end efficiency: Could the authors report complete training times, including teacher-cache construction, PCA, and all coordinate refreshes? It would also be helpful to separate the effect of TALAS’s optimizer (adaptive SAM) from the cost of its distillation objective.
- Hyperparameter selection: The paper states that learning rates are selected on validation data. Could the authors also clarify how the H0 weight, the refresh schedule, and the fit-set size were selected, given that the appendix ablations report test-set results?
Suggestions
- If feasible, I would encourage the authors to include retrieval evaluation and uncertainty estimates or significance tests for the smaller performance differences.
3 comments
Clarifying the Nature of the Contribution
Thank you for the follow-up. We appreciate that the additional experiments address your empirical concerns. We also agree with your characterization that PCA and orthogonal Procrustes themselves are classical; we do not claim either as a new algorithmic primitive.
We would, however, like to clarify one point about where we see the contribution. We do not view the novelty as simply replacing a per-batch Procrustes solve with a corpus-level one. The central finding is that, after dimensionality reduction, the teacher representation specifies an equivalence class of targets with identical relational geometry, yet choosing different members of that class can change the trained student substantially. This exposes coordinate resolution as a separate design decision in cross-dimensional embedding distillation.
This distinction is precisely where our use of Procrustes differs from EdgePoint2 and Bhattarai et al. Their objectives optimize over the rotation inside the loss, intentionally making supervision invariant to the coordinate realization. ORBIT-KD instead uses Procrustes to select one student-conditioned realization and then directly supervises the deployed student embedding in that common frame. Our controls show that this distinction is consequential: identical-geometry rotations produce substantially different students, per-batch resolution is 1.93 points below a shared corpus frame, a closer least-squares target is 4.43 points worse, and most of the benefit already appears from the initial coordinate resolution.
We therefore see the contribution as primarily conceptual and empirical, with a deliberately simple method: identifying a previously implicit degree of freedom, showing that it affects optimization and downstream performance despite preserving teacher geometry, and providing a closed-form way to resolve it. We have revised the paper to make this scope explicit and to avoid presenting classical Procrustes results as theoretical novelty.
We understand that the sufficiency of this type of contribution is ultimately a matter of judgment, but we hope this clarifies that the claimed novelty is the formulation and its demonstrated consequence, rather than PCA or Procrustes themselves.
Thank you for the clarification. I understand that the main contribution lies in identifying coordinate resolution as a separate design choice, and I agree that the analysis is meaningful. However, after considering the response, I still find the overall contribution limited. Therefore, I keep my current rating.
Thank you for the detailed response and the additional experiments. They address most of my concerns about the empirical evidence.
However, my concern about methodological novelty remains. PCA and orthogonal Procrustes are classical tools, and prior work such as EdgePoint2 and Bhattarai et al. (2025) already uses Procrustes alignment for distillation. The main methodological difference is a corpus-level fixed rotation instead of a per-batch one. Although the analysis is meaningful, I am not yet convinced that the overall contribution is sufficient, given the limited methodological novelty.
Therefore, I keep my current rating for now.