Equivalent Representations Are Not Equivalent Distillation Targets

Two distillation targets can preserve exactly the same relational geometry yet train the same student to different outcomes. We distinguish two decisions in cross-dimensional embedding distillation: representation reduction determines what teacher information the student retains at its dimension, whereas coordinate resolution determines how that geometry is presented to the pretrained student. We show that relational objectives are indifferent to orthogonally equivalent reduced targets and derive an exact initialization-headroom identity for the endpoint loss recoverable by selecting an orthogonal realization. Under a unique Procrustes solution, the resulting student-conditioned construction is independent of the reducer's arbitrary orthonormal basis and exactly preserves the retained teacher geometry. These results motivate Orthogonal Resolution of Basis-Indeterminate Targets for Knowledge Distillation (ORBIT-KD), which uses a corpus-level closed-form alignment to provide direct endpoint supervision without a trainable cross-space interface. In a matched fixed-target comparison, a single resolution improves a 22M student's nine-task average by $2.22$ points over the unreoriented target. A non-orthogonal least-squares target fits the pretrained student more closely but performs $4.40$ points worse than fixed Procrustes, showing that proximity alone is insufficient. Across six main model–data settings, ORBIT-KD improves over the strongest baseline by up to $1.66$ average points ($2.27$ relative), with up to a $3.9\times$ reduction in measured optimization-step time. Our work aims to motivate further research on coordinate resolution as a distinct design axis for student-aware, geometry-preserving transfer across heterogeneous representation spaces.