acceptodds
Under review as a conference paper at ICLR 2027

Equivalent Representations Are Not Equivalent Distillation Targets

164 views

Abstract

Two distillation targets can preserve exactly the same relational geometry yet train the same student to different outcomes. We distinguish two decisions in cross-dimensional embedding distillation: representation reduction determines what teacher information the student retains at its dimension, whereas coordinate resolution determines how that geometry is presented to the pretrained student. We show that relational objectives are indifferent to orthogonally equivalent reduced targets and derive an exact initialization-headroom identity for the endpoint loss recoverable by selecting an orthogonal realization. Under a unique Procrustes solution, the resulting student-conditioned construction is independent of the reducer's arbitrary orthonormal basis and exactly preserves the retained teacher geometry. These results motivate Orthogonal Resolution of Basis-Indeterminate Targets for Knowledge Distillation (ORBIT-KD), which uses a corpus-level closed-form alignment to provide direct endpoint supervision without a trainable cross-space interface. In a matched fixed-target comparison, a single resolution improves a 22M student's nine-task average by points over the unreoriented target. A non-orthogonal least-squares target fits the pretrained student more closely but performs points worse than fixed Procrustes, showing that proximity alone is insufficient. Across six main model–data settings, ORBIT-KD improves over the strongest baseline by up to average points ( relative), with up to a reduction in measured optimization-step time. Our work aims to motivate further research on coordinate resolution as a distinct design axis for student-aware, geometry-preserving transfer across heterogeneous representation spaces.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
knowledge distillation, embedding distillation, representation learning, representation alignment, dimensionality reduction, coordinate selection, orthogonal Procrustes, sentence embeddings, teacher-student learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FNDAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-dimensional embedding distillation by distinguishing representation reduction from coordinate resolution. The authors propose ORBIT-KD, which combines PCA-based reduction of teacher embeddings with corpus-level orthogonal Procrustes alignment to construct student-conditioned targets. The student is trained with a cosine endpoint loss and an auxiliary H0 persistence loss. In experiments with several teacher–student pairs and corpora, ORBIT-KD obtains the highest nine-task average among the compared methods. Controlled analyses show how coordinate alignment affects the distillation results.

Critical strengths

1. Careful analysis of coordinate resolution.

  • The paper provides well-controlled experiments that isolate the effect of coordinate alignment while keeping the teacher’s relational geometry unchanged.
  • Comparisons with random rotations, different references, and different fitting scopes give convincing evidence that geometrically equivalent targets can lead to different distillation outcomes. I think this observation is useful for practitioners who use reduced teacher targets.

2. Simple method with consistent average improvements.

  • ORBIT-KD uses closed-form orthogonal alignment. It does not require a trainable projector or online teacher inference during student training.
  • The method obtains the highest nine-task average in all six main settings. It also shows positive results with larger teachers and a task-independent corpus.
Critical weaknesses

1. Limited novelty and missing discussion of closely related work.

  • The theoretical results mostly follow from known properties of orthogonal transformations and classical Procrustes analysis. Therefore, the use of terms such as “Theorem” and “exact initialization-headroom identity” might give the impression that the theoretical contribution is larger than it actually is.
  • EdgePoint2 (Yao et al., 2025), although in the domain of local feature descriptors, appears to already combine a Gram-preserving low-rank reduction of teacher representations with orthogonal Procrustes alignment to the student for cross-dimensional distillation. This work does not seem to be discussed in the paper. Its per-minibatch alignment is similar to the “per-minibatch fits” variant in Appendix C.2. Therefore, a clearer explanation of what is new (e.g., corpus-level fitting, fixed targets with periodic refresh, the H0 loss) would be needed.

2. The empirical evidence is not yet sufficient to support the practical claims.

  • Necessity of full-rank resolution: Figure 13(d) shows that rank-1 alignment already accounts for about 72% of the improvement over the unreoriented target. The authors mention the role of the dominant mean direction, but comparisons with simpler alignment methods (e.g., centering, or aligning only the mean direction) seem necessary to show the additional value of full-rank Procrustes alignment.
  • Evaluation scope: Retrieval benchmarks are not included, although retrieval is presented as an important motivation. In addition, the main distillation corpora are drawn from the training splits of the evaluation tasks.
  • Size of improvements: Some improvements over TALAS are small (e.g., +0.07 and +0.27 points on the 100K corpus), and only three training seeds are reported without significance tests. The 20-epoch experiments show that the gain persists, but it also becomes smaller over training, and there is no comparison with TALAS under longer or compute-matched training.
  • Efficiency: The reported step-time speedups exclude teacher-cache construction and coordinate refresh. The refresh time is reported separately (under 0.7 minutes per run), but this is similar to the whole optimization time in configuration (c) (0.64 minutes). Also, since TALAS uses adaptive SAM, it is difficult to attribute the speed difference to the proposed approach itself.
Questions for authors

Clarification questions

  1. Relationship to EdgePoint2: Could the authors clarify the methodological differences between ORBIT-KD and the orthogonal Procrustes distillation in EdgePoint2? In particular, which contributions go beyond corpus-level fitting and periodically updated fixed targets?
  2. Role of the dominant direction: Figure 13(d) indicates that rank-1 alignment explains most of the improvement. How does ORBIT-KD compare with a simpler method that aligns only the mean or dominant direction? How sensitive is the improvement to the anisotropy of the pretrained student embeddings?
  3. Learned projector baselines: In Appendix C.3, the trainable linear-map baselines perform much lower than even the unreoriented PCA target. Could the authors explain how these baselines were optimized? Also, how would a trainable orthogonal projector initialized with the Procrustes solution compare with the proposed fixed-target approach?
  4. Longer training: Do the improvements over TALAS remain under longer training schedules or comparable computational budgets? The current 20-epoch experiments show the persistence of the alignment benefit, but they do not address the advantage over the strongest baseline.
  5. End-to-end efficiency: Could the authors report complete training times, including teacher-cache construction, PCA, and all coordinate refreshes? It would also be helpful to separate the effect of TALAS’s optimizer (adaptive SAM) from the cost of its distillation objective.
  6. Hyperparameter selection: The paper states that learning rates are selected on validation data. Could the authors also clarify how the H0 weight, the refresh schedule, and the fit-set size were selected, given that the appendix ablations report test-set results?

Suggestions

  1. If feasible, I would encourage the authors to include retrieval evaluation and uncertainty estimates or significance tests for the smaller performance differences.
3 comments

Thank you for the detailed response and the additional experiments. They address most of my concerns about the empirical evidence.

However, my concern about methodological novelty remains. PCA and orthogonal Procrustes are classical tools, and prior work such as EdgePoint2 and Bhattarai et al. (2025) already uses Procrustes alignment for distillation. The main methodological difference is a corpus-level fixed rotation instead of a per-batch one. Although the analysis is meaningful, I am not yet convinced that the overall contribution is sufficient, given the limited methodological novelty.

Therefore, I keep my current rating for now.

AuthorsOpenReview ↗

Clarifying the Nature of the Contribution

Thank you for the follow-up. We appreciate that the additional experiments address your empirical concerns. We also agree with your characterization that PCA and orthogonal Procrustes themselves are classical; we do not claim either as a new algorithmic primitive.

We would, however, like to clarify one point about where we see the contribution. We do not view the novelty as simply replacing a per-batch Procrustes solve with a corpus-level one. The central finding is that, after dimensionality reduction, the teacher representation specifies an equivalence class of targets with identical relational geometry, yet choosing different members of that class can change the trained student substantially. This exposes coordinate resolution as a separate design decision in cross-dimensional embedding distillation.

This distinction is precisely where our use of Procrustes differs from EdgePoint2 and Bhattarai et al. Their objectives optimize over the rotation inside the loss, intentionally making supervision invariant to the coordinate realization. ORBIT-KD instead uses Procrustes to select one student-conditioned realization and then directly supervises the deployed student embedding in that common frame. Our controls show that this distinction is consequential: identical-geometry rotations produce substantially different students, per-batch resolution is 1.93 points below a shared corpus frame, a closer least-squares target is 4.43 points worse, and most of the benefit already appears from the initial coordinate resolution.

We therefore see the contribution as primarily conceptual and empirical, with a deliberately simple method: identifying a previously implicit degree of freedom, showing that it affects optimization and downstream performance despite preserving teacher geometry, and providing a closed-form way to resolve it. We have revised the paper to make this scope explicit and to avoid presenting classical Procrustes results as theoretical novelty.

We understand that the sufficiency of this type of contribution is ultimately a matter of judgment, but we hope this clarifies that the claimed novelty is the formulation and its demonstrated consequence, rather than PCA or Procrustes themselves.

Thank you for the clarification. I understand that the main contribution lies in identifying coordinate resolution as a separate design choice, and I agree that the analysis is meaningful. However, after considering the response, I still find the overall contribution limited. Therefore, I keep my current rating.

Reviewer_8HGkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the question of whether student embedding and teacher embedding spaces are truly aligned by relational distillation and distillation through leared projector. The authors conceptually disentangled the bridging adapter in knowledge distillation into two parts; dimensionality reduction and corrdinate matching. This gives meaningful intuition to knowledge matching for transfer. Based on this perspective, ORBIT-KD implements dimensionality reduction as truncated PCA and coordinate alignment as orthogonal Procrustes alignment. It was validated on natural language processing benchmarks, demonstrating improved performance, ablation studies and training efficiency analysis.

Critical strengths
  1. The conceptual disentanglement is clear and well-motivated from a geometric perspective.
  2. The proposed method is mathematically simple and computationally efficient.
  3. The empirical analysis examines the geometric properties and design choices behind ORBIT-KD.
  4. ORBIT-KD consistently improves performance across benchmarks.
Critical weaknesses
  1. Weak justification for motivation. The authors begins from a trivial fact: teacher embedding and student embedding dimensions are different, thererfore, since the embedding itself is directly mapped to final output in natural language processing, coordinate resolution is crucial. However, it remains unclear why this ambiguity poses a significant limitation for existing approaches, particularly relational distillation and learned projection-based methods. The authors should provide stronger evidence for when and why explicit coordinate resolution is necessary beyond the direct endpoint supervision setting.
  2. Limitations of previous methods are not convincing. One of the motivation of ORBIT-KD is that relational distillation cannot determine a specific teacher-side coordinate space, and learnable projector absorbes the difference between two coordinate spaces. It is too intuitive, requires evidence. Relational distillation has designed as knowledge transfer independent to a specific coordinate space. The invariance on coordinate system is more like intended attribute rather than a defect. Similarly, if learned projectors already absorbes the coordinate system difference, I would consider the problem as resolved. Orelse, it should be investigated whether the problem is on the coordinate mismatch, optimization, or any other problem. The paper should provide deeper evidence.
  3. Lack of baseline and ablation study. The components for improving distillation in this paper are prthogonal procrustes, structural supervision based on H0H_0. Ablation study on those components are not enough. If coordinate resolution aims to align student-teacher representation spaces, it has to be compared against general orthogonal alignment methods. For example, orthogonality regularization in PointNet (Qi et al., CVPR 2017) or the authors could investigate on more examples. Meanwhile, despite the paper provides PCA endpoint, resolved endpoint, H0H_0 only, resolved endpoint + H0H_0 combination, PCA endpoint + H0H_0 is still missing. Finally, choice of H0H_0 in structural supervision also needs to be justified. The described loss function compares the edge weights of a minimum spanning tree, but it does not explicitly preserve parewise geometry. It raises the question of whether the chosen loss could be replaced by a general relational distillation loss or Gram matrix matching.
  4. Clarity and presentation. The descriptions in the paper are too complex and verbose, degrading the readibility and quality of the manuscript. Introduction typically includes motivation, core problem definition, an overview of proposed methods, and contributions of the paper. This paper also follows this structure. However, it introduces many concepts (representation reduction, coordinate resolution, orthogonal orbit, relational equivalence) before the motivation is clearly stated. I would recommend reorganizing the presentation to establish the motivation before introducing these concepts.
Questions for authors
  1. Why are orthogonal Procrustes and H0H_0-based structural supervision necessary? Could the authors provide the missing PCA endpoint + H0H_0 ablation to clarify their individual contributions?
  2. What is the advantage of H0H_0-based supervision over general relational distillation or Gram matrix matching, given that the proposed loss does not explicitly preserve the full pairwise geometry?
2 comments
Reviewer_8HGkOpenReview ↗

Response to the authors

I thank the authors for the experiments that address my previous questions. Most of my concerns have been resolved.

I have read the authors' clarification in response to Reviewer FNDA's concerns, but my concern about the novelty of ORBIT-KD remains. I find the theoretical analysis, conceptual formulation, and empirical findings valuable and potentially sufficient for ICLR. However, I remain unconvinced that ORBIT-KD constitutes a sufficiently novel methodological contribution.

If the authors intend to position ORBIT-KD primarily as an implementation that demonstrates these findings, I would find the contribution convincing. However, I believe this would require substantial reframing of the manuscript, potentially making it significantly different from the original submission, as described in the ICLR Author Guidelines.

Therefore, I maintain my rating. Nevertheless, I would like to explicitly acknowledge the strength of the theoretical analysis and the value of the conceptual and empirical contributions, and encourage the authors to consider this reframing for a future submission.

AuthorsOpenReview ↗

Clarifying the Intended Positioning of ORBIT-KD

Thank you for the thoughtful follow-up and for explicitly recognizing the value of the theoretical, conceptual, and empirical contributions.

We agree that ORBIT-KD itself is deliberately simple and that neither PCA nor orthogonal Procrustes is a novel algorithmic primitive. We would like to clarify, however, that positioning ORBIT-KD as the operationalization of the paper's main finding is not intended as a new contribution type introduced during rebuttal.

The original paper is centered on the observation that geometry-equivalent reduced teacher representations need not be equivalent distillation targets, and on identifying coordinate resolution as a design decision distinct from representation reduction. ORBIT-KD is the simplest closed-form implementation that follows from this formulation: after deciding what teacher information to retain, it resolves the remaining coordinate freedom relative to the receiving student and then uses the resulting representation as a direct target.

Our revisions therefore do not attempt to turn the submission into a substantially different paper. Rather, in response to the reviews, we have made the intended scope more explicit and removed wording that could suggest that PCA or Procrustes themselves constitute the methodological novelty. The conceptual formulation, analysis, and the simple method that instantiates it remain the same contributions as in the original submission.

We appreciate your distinction between the value of these contributions and the novelty of the underlying mathematical tools, and hope this clarification better conveys the intended positioning of the paper.

Reviewer_zjNROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Basically, this paper studies the so-called cross-dimensional embedding distillation, that is, the teacher and student models have different representation dimensions. The authors propose ORBIT-KD, which first uses PCA to compress the cached teacher embeddings, then uses an orthogonal alignment method to align them with the pretrained student’s embeddings. After that, it trains the student with an endpoint loss plus a topological loss. The theoretical analysis characterizes the recoverable initial loss and establishes invariance and geometry-preservation properties under appropriate assumptions.

Critical strengths

The strongest contribution is the extensive empirical evaluation. The ablation studies cover multiple aspects, including coordinate choices, alignment references, global versus minibatch fitting, orthogonality, refresh schedules, and longer training runs.

Critical weaknesses

The algorithmic and theoretical novelty seems relatively limited. Specifically, the dimension reduction method, the Procrustes alignment method, and several stated properties appear to follow from related studies.

Questions for authors

The method proposed by Bhattarai et al. adopts a similar idea based on Procrustes distance, and the use of Procrustes in both ORBIT-KD and Bhattarai et al. is mathematically closely related. The authors should explain why this method was not included as a baseline in the empirical evaluation. The authors should also clarify the advantages of ORBIT-KD over the method proposed by Bhattarai et al.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.