acceptodds
Under review as a conference paper at ICLR 2027

Grassmannian Subspace Alignment for Knowledge Distillation

Abstract

In knowledge distillation for neural networks, a compact student learns from a larger teacher while remaining efficient to deploy. When their hidden widths differ, intermediate representations cannot be compared directly, and most feature-based methods learn a projection and then match features coordinate by coordinate. This ties the student to one particular basis of the teacher's representation, even though any rotation of that basis spans the same subspace. We instead align the subspaces themselves: learned projections map student representations into the teacher's space, where we apply point-wise spherical or layer-wise Grassmann subspace alignment. For speech recognition, distilling Whisper Large-v3 into Whisper Medium with the spherical variant and low-rank adaptation (LoRA) reaches 13.49% word error rate (WER) on a LibriSpeech test-other subset while training only 4.55% of the teacher's parameter count. For language modeling, distilling Apertus-8B into a 0.5B student with layer-wise Grassmann alignment shows gains on HellaSwag and PIQA. We further prove that the Grassmann loss has an intuitive interpretation as the expected leakage of student directions outside the teacher subspace.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
knowledge distillation, Grassmann manifolds, Riemannian geometry, representation alignment, automatic speech recognition, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Fx7ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Grassmannian Subspace Alignment, a feature-based knowledge distillation approach for transferring intermediate representations from a larger teacher model to a smaller student with different hidden dimensions. The authors argue that conventional feature matching, which aligns representations coordinate by coordinate, can be unnecessarily restrictive because different feature bases may span the same underlying subspace. To address this, the method first uses a learnable linear projection to map student representations into the teacher's feature space. It then applies geometry-based alignment objectives: spherical angular alignment for speech recognition, which matches feature directions, and Grassmannian projector alignment for language modeling, which matches the subspaces spanned by intermediate representations. These objectives are combined with task supervision and logit knowledge distillation as final loss.

Critical strengths
  1. The paper presents a clear motivation for aligning representation subspaces rather than matching individual feature coordinates.
  2. The authors evaluate geometric representation alignment in both speech recognition and language modeling domains.
Critical weaknesses
  1. Missing comparisons with established feature-distillation methods for hidden-dimension mismatch, such as Projector+MSE and Projector+Cos, CKA-based alignment ( Dasgupta et al., 2025), and Flex-KD (Saadi et al., 2025). These baselines are particularly important because the main contribution is a new representation-alignment objective. Without comparisons under matched training conditions, it is difficult to establish whether Grassmannian alignment provides meaningful advantages over existing feature-distillation techniques when the student and teacher have different hidden sizes.
  2. The speech experiments (Table 2) use spherical alignment rather than Grassmannian alignment and lack a matched CE/KL-only baseline. In language modeling, the baseline and proposed method differ in both the representation loss and CE/KL training schedule (Section 4.3), preventing the contribution of Grassmannian alignment from being isolated. Furthermore, the language-model results (Table 3) show mixed performance.
Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Answers to the Questions one by one

Q1. Comparisons under matched training conditions

We agree that comparisons under matched conditions are needed. In a separate small-scale study, we matched the training data, student initialization, batch order, optimizer, training budget, fixed CE/KL weights, and evaluation protocol across the CE + KL baseline and the alignment arms. We evaluated each alignment method with six seeds at two weights, without method-specific tuning.

Alignment objectiveMean test CE at λ = 0.1 ↓Mean test CE at λ = 1.0 ↓
CE + KL without feature alignment1.8071.807
+ Projector MSE1.8991.930
+ Projector Cosine1.8471.881
+ Spherical Loss1.8521.884
+ Linear CKA1.8821.905
+ VkD-style Orthogonal Projector1.8781.944
+ Grassmann Loss (ours)1.8241.872

Grassmann had the lowest mean test CE among the tested alignment objectives at both weights. At λ = 0.1, it reduced mean test CE by 0.075 and improved test accuracy by 1.38 percentage points relative to Projector MSE. Its mean CE advantage over Projector Cosine was 0.023 at λ = 0.1 and 0.009 at λ = 1.0. CE + KL alone achieved lower mean test CE at both weights. Grassmann beat that baseline in 0/6 seeds at λ = 0.1 and 1/6 at λ = 1.0. The full results are in our reply to Urp3.

The synthetic comparison below illustrates the geometric distinction from linear CKA. The first three transformations preserve the column space in this example. Values are means over 50 draws.

Student features1 − Linear CKAGrassmann Loss
Orthogonal rotation (Aₛ = AₜQ)00
Invertible nonorthogonal remixing (Aₛ = AₜR)0.2920
Positive diagonal column rescaling (Aₛ = AₜD)0.5870
Unrelated subspace0.9760.975

Q2. Scope of the ASR and language-model results

Table 2 evaluates spherical alignment with LoRA in ASR. The experiment lacks a matched CE + KL-only baseline for isolating the effect of spherical alignment. Our code audit also found decoder-position and vocabulary-index mismatches in the original Whisper KL computation, as detailed in our reply to Urp3.

We agree that the original Apertus comparison in Section 4.3 changes both the feature loss and the CE/KL schedule. The mixed results in Table 3 therefore leave the effect of Grassmann alignment unresolved. We will describe them as preliminary. The separate six-seed study above holds the CE/KL weights fixed. The matched Apertus-0.5B comparison is still in progress.

Thank you for the response. I am not sure if the reported CKA baseline reproduces the method of Dasgupta and Cohn (2025) or it uses the standard Linear CKA loss. Also, the additional experiments lack important details, such as the dataset and models used. The authors only mention that they were conducted in a separate small-scale setting. More importantly, CE+KL without feature alignment still achieves better results. This raises concerns about the benefit of the proposed method, and even among the tested feature-alignment methods, the improvements are marginal. The original language-model experiments also remain confounded by differences in training details, and the matched Apertus experiments are still incomplete. For ASR, the authors acknowledge decoder-position and vocabulary-index mismatches in the original KL computation, but no corrected experimental results are reported. These issues raise concerns about the reliability of the reported findings. Finally, no revised version of the manuscript is uploaded. I therefore maintain my original score.

Reviewer_Urp3OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this work, the authors propose a knowledge transfer scheme called Grassmannian Subspace Alignment that enables the transfer of intermediate representations between teacher and student models that have different hidden dimensions. They achieve this by introducing learnable linear projections to project the student model's representations into the teacher model's representation space, then apply geometric alignment objectives. For automatic speech recognition (ASR), they use point-wise spherical angular alignment; for language modeling, they use layer-wise Grassmann subspace alignment. The layer-wise Grassmann subspace alignment objective seeks to minimize the discrepancy between the orthogonal projection matrices formed from the teacher and student model representations. Finally, the authors offer a theoretical explanation of the Grassmann loss function in terms of principal angles and the average amount of "leakage" of the student model's representation directions that fall outside the teacher model's subspace.

The suggested method is evaluated using speech recognition and language modeling. In ASR, the authors distill the Whisper Large-v3 model into the Whisper Medium model via spherical alignment and low-rank adaptation (LoRA). They report a 13.49% word error rate on a 200-utterance subset of LibriSpeech test-other and update about 70.52 million parameters. For language modeling, the authors evaluate Grassmann alignment by distilling the Apertus-8B model into a 0.5B student model on various English and multilingual benchmarks. While some language model results indicate improvement on certain tasks, such as HellaSwag and PIQA, the results do not indicate an overall improvement when compared to the reported baseline.

Critical strengths

A clearly defined mathematical formulation of representation alignment.

The authors create a framework to transfer knowledge from one model to another model that has a different representation dimension. The Grassmann objective proposed by the authors is invariant to any change of the orthonormal basis used to represent the same subspace through the use of orthogonal projectors. Additionally, the authors' use of principal angles and their explanation of expected leakage allow them to provide a clear geometric description of how the loss function operates. As such, this method represents a viable alternative to matching features based on coordinates alone and is the most significant contribution of the paper in terms of methodology.

Exploration of geometric alignment across different model architectures.

The paper studies representation alignment across speech recognition and language modeling through the use of established teacher and student models. In terms of speech recognition, the authors show that it is possible to combine representation-level supervision with low-rank adaptation (LoRA) based parameter-efficient adaptation. In terms of language modeling, the authors directly evaluate Grassmann projector alignment. Additionally, the authors study parameter-update efficiency versus model size at inference time. The authors do not present their results as universally applicable; instead, they present them as mixed experimental outcomes. Overall, this provides a basis for future research.

Critical weaknesses

It has not been shown that Grassmannian alignment improves knowledge distillation or has an independent effect on performance.

There is a major limitation in the form of a lack of convincing experimental evidence that the proposed Grassmann alignment objective actually improves knowledge distillation.

First, the point-wise spherical angular alignment used in the ASR experiments was not based on the Grassmann projector loss. As such, the 13.49% WER reported in this paper does not provide direct validation for the Grassmannian method. Additionally, there are no matched comparisons with CE/KL-only distillation, feature MSE, or any other alternative representation alignment objective. The reduction in the number of updated parameters was due to the use of LoRA and backbone freezing, not because of the geometric objective.

Secondly, the language model experiments directly evaluate Grassmann alignment; however, they do not demonstrate much positive evidence. Table 4 indicates that the average accuracy across all six groups decreases from 39.86% for the baseline to 39.64% using Grassmann alignment. While some of the tasks (HellaSwag and PIQA) showed improved performance, several other tasks experienced decreased performance, and overall the average results were similar to random chance. Additionally, Figure 5 demonstrates that the PIQA improvement was reversed when evaluated using length-normalized accuracy.

The baseline configuration used fixed CE/KL loss weights, while the Grassmann configuration changed both the CE/KL training schedule and the representation alignment objective. Thus, there may be other factors in addition to the geometric objective that have contributed to the performance difference between the two configurations. Until there are experimental studies where the training conditions are held constant and only the alignment objective is changed, the primary methodological contribution will remain insufficiently demonstrated.

The experimental design and implementation limitations restrict how reliable and generalizable the results can be.

I cannot draw definitive conclusions on whether the suggested method will be effective based upon the evaluations that have taken place up to this point.

The reported WER for ASR is based only upon 200 successfully processed LibriSpeech test-other utterances; however, the published reference models are evaluated on the entire test split using a different normalization procedure. The identifiers of the evaluated utterances and their recognition outputs are not retained, making it impossible to reconstruct a matched comparison from the reported records. Because of these differences, we cannot determine the relative recognition performance of the proposed method against the original student model or conventional distillation methods under identical evaluation conditions.

There is a potential technical issue associated with the implementation of ASR as well. In Appendix A, it is stated that the teacher receives decoder_input_ids=labels, whereas the student receives labels=labels. Depending on how the decoder inputs are prepared, there could be different teacher-forcing conditions created by each call. In addition to this, the vocabulary axis of the logits is truncated to the smaller size; however, no documentation has been provided regarding an explicit token mapping. Thus, additional clarification will be required to confirm that the KL objective compares teacher and student predictions that are properly aligned.

The nominal training budget for language modeling is approximately six million tokens; however, the actual consumed token counts, training manifests, and exact checkpoint origins are not provided. Performance benchmarks for these models are near chance level on several tasks. In addition, the reported results do not include repeated training-seed evaluations or paired significance analyses. This makes it difficult to determine if the poor performance observed is due to the alignment objective or because of inadequate training, poor initialization, suboptimal optimization, or evaluation variability.

The current comparative analysis does not provide evidence of an advantage of the proposed method when compared to other projection-based or subspace-based distillation approaches such as VkD and SDMD. Because there is a substantial amount of research on the underlying geometry used in the formulation of the proposed method, it would be advantageous to determine whether the proposed method offers an improvement over these alternatives so as to establish the methodological importance of the proposed formulation.

Questions for authors

Contribution of Grassmann alignment (experimental validation).

Could the authors run some controlled studies of language models with the same training set, initialization, optimization method, training budget, CE/KL schedule, and evaluation protocol, while varying only the alignment objective, including a configuration without Grassmann alignment? It would be beneficial to compare this new method against feature MSE and current methods for preserving structure through distillation, to determine if the proposed geometric loss provides a meaningful advantage over these alternatives.

Effectiveness of the ASR alignment objective (experimental validation).

Would the authors be able to assess the original Whisper Medium student, a CE/KL-only distillation baseline, and the suggested spherical alignment technique using the same complete LibriSpeech test sets under matched training and evaluation conditions? Additionally, as the authors did not utilize Grassmann projector alignment for their ASR implementation, could they explain how the ASR outcomes validate the main Grassmann formulation?

Teacher–student conditioning and vocabulary alignment (technical clarification).

Are the authors able to confirm whether the teacher and student decoder logits are being computed using the same teacher-forcing conditions in the ASR implementation? In particular, what is the procedure for handling decoder shifts, as the teacher uses decoder_input_ids=labels while the student uses labels=labels? Furthermore, how does the system ensure vocabulary alignment when the logits are aligned by truncating the vocabulary dimension?

Reliability of the language-model comparison (experimental validation).

Could the authors present their actual training-token counts, checkpoint origins, and multi-seed training results for both of the language model configurations? Also, it would be useful to know whether the student has reached an adequate performance level so that we can evaluate how much influence the geometric alignment has on the student. Knowing whether the students' near-chance performance is due to issues with the proposed objective or due to other training-related issues will enable us to assess the potential effectiveness of the proposed objective.

2 comments
AuthorsOpenReview ↗

Answers to the Questions one by one

Q1. Controlled comparison of alignment objectives

For the CE + KL baseline and the alignment arms, we matched the training set, student initialisation, batch order, optimiser, training budget, fixed CE/KL weights, and evaluation protocol. The alignment objective was the experimental variable.

Arm (λ=0.1\lambda=0.1)Test CE ↓Test Accuracy (%) ↑ΔCE vs. CE + KL
CE only (no teacher)2.118 ± 0.06443.66 ± 1.21+0.311
CE + KL (no alignment)1.807 ± 0.05147.09 ± 0.75—
+ Projector MSE1.899 ± 0.03945.47 ± 0.82+0.091
+ Projector Cosine1.847 ± 0.04346.41 ± 0.72+0.039
+ Linear CKA1.882 ± 0.04445.76 ± 0.67+0.075
+ VkD-style orthogonal projector1.878 ± 0.05145.83 ± 1.02+0.071
+ Grassmann LGL_G (ours)1.824 ± 0.04546.85 ± 0.74+0.017
Teacher (reference)1.40950.57—

At λ=0.1\lambda=0.1, Grassmann alignment achieved the lowest mean test CE among the tested alignment objectives. CE + KL alone achieved a lower mean test CE (1.807 vs. 1.824). At λ=1.0\lambda=1.0, the differences between Grassmann and projector cosine or linear CKA were not statistically significant in our paired comparisons.

The matched Apertus-0.5B comparison with three seeds and fixed 0.1 CE+0.9 KL0.1\,\mathrm{CE}+0.9\,\mathrm{KL} weights in both arms is still in progress.

Q2 and Q3. Whisper logit alignment

  1. Rerun status. As of 13 November, the corrected ASR rerun was in progress. Rerun materials.
  2. Consequence. The reported run had mismatched decoder positions and vocabulary indices in the KL computation. The logged KL rose from 4.32 (stage I) to 5.89 (stage III), while cross-entropy and the angular loss fell. The encoder-side spherical loss is computed separately. The mis-specified KL could still affect the jointly trained parameters.
  3. Correction. Both models receive decoder_input_ids = shift_right(labels). We compute KL over corresponding shared text-token entries, using each model's own vocabulary indices. For the same model, the two decoder call paths produced identical logits when given identical decoder inputs (maximum absolute difference: 0.0). We will document the mismatches and the correction in Appendix A.

Q4. Checkpoint provenance and controlled comparison

The manifests for the evaluated Apertus checkpoints cannot be recovered. We will state in the main text that we cannot report the number of training tokens consumed or the checkpoint hashes. The near-chance pilot results leave the effect of the alignment objective confounded with potential under-training.

Table 1 reports a separate controlled comparison with six seeds and paired tests. In this setting, the CE + KL student achieved 47.09% test accuracy and the teacher achieved 50.57%. The Apertus reruns use three seeds and log the number of training tokens consumed, the data manifest, and the checkpoint SHA.

Table 1. Controlled comparison of alignment objectives (6 seeds, mean ± s.d.)

Armλ\lambdaTest CE ↓Test Accuracy (%) ↑ΔCE vs. CE + KLWins vs. CE + KLpp vs. CE + KL
CE only (no teacher)—2.118 ± 0.06443.66 ± 1.21+0.3110/6<0.001
CE + KL (no alignment)—1.807 ± 0.05147.09 ± 0.75———
+ Projector MSE0.11.899 ± 0.03945.47 ± 0.82+0.0910/60.004
+ Projector MSE1.01.930 ± 0.01044.75 ± 0.38+0.1230/60.004
+ Projector Cosine0.11.847 ± 0.04346.41 ± 0.72+0.0390/60.006
+ Projector Cosine1.01.881 ± 0.01246.00 ± 0.12+0.0740/60.016
+ Spherical (Eq. 3)0.11.852 ± 0.04446.08 ± 0.71+0.0441/60.007
+ Spherical (Eq. 3)1.01.884 ± 0.01145.86 ± 0.20+0.0770/60.013
+ Linear CKA0.11.882 ± 0.04445.76 ± 0.67+0.0750/60.003
+ Linear CKA1.01.905 ± 0.02545.43 ± 0.56+0.0980/60.006
+ VkD-style orthogonal projector0.11.878 ± 0.05145.83 ± 1.02+0.0710/60.030
+ VkD-style orthogonal projector1.01.944 ± 0.00544.71 ± 0.33+0.1370/60.002
+ Grassmann LGL_G (ours)0.11.824 ± 0.04546.85 ± 0.74+0.0170/60.022
+ Grassmann LGL_G (ours)1.01.872 ± 0.05146.06 ± 0.72+0.0641/60.023
Teacher (reference)—1.40950.57———

Q2 and Q3. Whisper logit alignment

  1. Consequence of the mismatch. The reported run had mismatched decoder positions and vocabulary indices in the KL computation. Consistently, the logged KL rose from 4.32 (stage I) to 5.89 (stage III) while cross-entropy and the angular loss fell. The spherical loss is computed on encoder states and does not enter the KL term, but the mis-specified KL could still have affected the jointly trained LoRA parameters. We therefore no longer use the submitted 13.49% WER as evidence for the method.

  2. Correction. Both models receive decoder_input_ids = [SOT] + labels[:-1], built from each model's own tokenizer, so that at every position both models predict the same target token. The KL term is computed only over ids 0–50,257, renormalised, at positions whose target is a text token or end-of-text; the language, task and timestamp positions are excluded. With identical decoder inputs, the labels= and decoder_input_ids= call paths of the same model give identical logits (maximum absolute difference 0.0), which confirms that the shift is the only difference between the two paths. Appendix A will document the mismatch and the correction.

  3. Vocabulary mapping and its verification. Whisper Medium has 51,865 tokens and Large-v3 has 51,866: the two tokenizers share the same byte-pair vocabulary, and Large-v3 inserts a single additional language token (<|yue|>) at id 50,358. Hence ids 0–50,257 (all text tokens and end-of-text) denote the same strings in both vocabularies, whereas the task, no-timestamps and timestamp tokens from id 50,358 onward are shifted by one, which is exactly what the previous truncation to 51,865 ids got wrong. The correction uses only the shared range and never maps a special-token id across models. We verified this directly: the two tokenizers return identical token strings for all 50,258 ids in 0–50,257 (50,258/50,258), the first differing id is 50,358, and test sentences in English, German, French and Polish receive identical id sequences from both tokenizers. As a per-utterance check, both tokenizers encode each transcript independently and training asserts that the two text-id sequences are identical; utterances for which they differ are dropped.

  4. Matched comparison (new, reduced budget, colab training - https://drive.google.com/file/d/16EbUh-B4fWGMtl4WnAitJKiIPlplK7QE/view?usp=sharing`). Same data order, seed, LoRA configuration (rank 64), three-stage schedule and loss weights as the submitted run; only the encoder-side term differs. Budget: 3,000 MLS utterances (batch 4, 750 optimiser steps; the submitted run used 20,000 utterances at batch 1), one seed. Evaluation: full LibriSpeech test-clean (2,620 utterances) and test-other (2,939), Whisper English normaliser, greedy decoding, final-step checkpoint (no selection on test data), utterance-level bootstrap 95% confidence intervals; per-utterance hypotheses are stored.

Systemtest-clean WER (%) [95% CI]test-other WER (%) [95% CI]
Whisper Medium, no training2.88 [2.55, 3.29]6.04 [5.49, 6.72]
CE + KL (corrected)3.77 [2.95, 4.85]6.32 [6.01, 6.66]
CE + KL + Spherical (corrected)2.88 [2.66, 3.10]7.08 [6.22, 8.34]
CE + KL + Projector-MSE + InfoNCE (corrected)2.83 [2.62, 3.05]6.83 [6.23, 7.76]

Under matched conditions no alignment term improves on CE + KL in a consistent direction: both alignment arms are lower on test-clean and higher on test-other, with overlapping intervals on test-other, and the two alignment arms do not differ from each other on either split. No trained system improves on the untrained student on the English test sets after this short, non-English training budget. We therefore make no claim of an ASR improvement from the alignment term. The training record shows that the alignment terms are optimised (mean between the first and last 10% of steps: angular loss 1.48 → 1.24; projector MSE 1.15 → 0.73) while the corrected KL stays flat in all three arms (1.31 → 1.42). A full-budget run (20,000 utterances) was started but will not complete before the discussion deadline; we will report it in the revised paper with the same protocol.

Reviewer_XLWHOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Grassmannian Alignment method for knowledge distillation. Specifically, the method introduces two loss functions: a Frobenius-norm loss that measures the discrepancy between the basis matrices of teacher and student features, and an angular distance loss that aligns the teacher and student features. The proposed method is evaluated on large language models (LLMs).

Critical strengths

For knowledge distillation, the paper is aligning representation subspaces and not the individual features, so it might avoid sensitivity to arbitrary rotations.

Experiments show improvements in speech recognition and language modeling, requiring significantly fewer trainable parameters through LoRA.

The related work section is comprehensive.

Critical weaknesses

I am not sure if the paper has enough novelty, and moreover, some comparisons with related methods are missing. Subspace alignment has already been extensively explored in knowledge distillation as referenced below. However, the paper does not provide experimental comparisons with these related methods, so it is making it difficult for me to assess the effectiveness and novelty of the proposed approach.

The proposed insights and concepts have been widely explored in machine learning and deep learning area.

The experiments does not exhibit good performance. In some cases, like in Table 4, the performance of the proposed method is even lower than baseline.

I am a bit confused with notations. Many mathematical notations and symbols are introduced but not adequately explained, such as L_geo and symbols involving bu. So it was difficult for me to follow technical details.

References

[1] Miles, Roy, et. al. "VkDV_kD: Improving Knowledge Distillation Using Orthogonal Projections." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.

[2] Zhang, Zexing, et al. "From Teacher Pathways to Invariant Manifolds: Consensus Subspace Distillation for TSFMs." Forty-third International Conference on Machine Learning (ICML), 2026.

Questions for authors

Since projection alignment does not guarantee token-level semantic correspondence, aligning the projection space does not mean aligning the semantic information. Could the authors please clarify this and justify the design of L_{proj}?

Also, please see weaknesses.

2 comments
AuthorsOpenReview ↗

Answers to the Questions one by one

1. Token-level semantics and LprojL_{\mathrm{proj}}. We agree: LprojL_{\mathrm{proj}} alone does not give token-level correspondence, and it is not designed to.

  • Token level. The KL term compares ℓt,bu\ell_{t,bu} and ℓs,bu\ell_{s,bu} for the same example bb and position uu. LprojL_{\mathrm{proj}} is only added to it.
  • What LprojL_{\mathrm{proj}} constrains. Lproj=∥Ps−Pt∥F2=2∑isin⁡2θiL_{\mathrm{proj}}=\lVert P_s-P_t\rVert_F^2=2\sum_i\sin^2\theta_i (θi\theta_i: principal angles), and LG=Lproj/(2k)L_G=L_{\mathrm{proj}}/(2k) is the mean leakage of student directions outside the teacher subspace (Eq. 8). For any x,yx,y, ∣x⊤(Ps−Pt)y∣≤Lproj ∥x∥2∥y∥2\lvert x^\top(P_s-P_t)y\rvert\le\sqrt{L_{\mathrm{proj}}}\,\lVert x\rVert_2\lVert y\rVert_2 (Eq. 9), so for the same input any shared linear readout of the two projections differs by at most Lproj ∥x∥2\sqrt{L_{\mathrm{proj}}}\,\lVert x\rVert_2. This holds for a shared readout only, not for the different teacher and student decoders; hence the KL term.
  • Why basis-free. Coordinate-wise losses also fix the teacher's basis (table in 2). In our controlled study (6 seeds, only the alignment term varies; see reply to Urp3), at λ=0.1\lambda=0.1 the test-CE increase over CE+KL is +0.017+0.017 for LGL_G, vs. +0.039+0.039 (cos), +0.071+0.071 (VkD-style), +0.075+0.075 (CKA), +0.091+0.091 (MSE).
  • Limit. No alignment term, ours included, improves on CE+KL alone (1.8241.824 vs. 1.8071.807). We justify LprojL_{\mathrm{proj}} as the least restrictive tested alignment under width mismatch, not as semantic transfer by itself.

2. Related methods.

  • [1] VkD constrains the projector weights to be orthogonal but still regresses features coordinate-wise. We leave the lift free and make the target basis-free: the loss depends only on span(Us)\mathrm{span}(U_s) and span(Ut)\mathrm{span}(U_t).
  • [2] distils a low-rank consensus subspace with its centre statistics across teacher pathways; we compare per-layer pooled-subspace projectors of one teacher after a learned lift, with a closed-form leakage interpretation (Eq. 8).

Loss when the student spans exactly the teacher subspace in a different basis (mean of 50 draws; 0 = invariant):

Student features AsA_sMSE1−cos⁡1-\cos1−CKA1-\mathrm{CKA}LGL_G (ours)
identical (As=AtA_s=A_t)0000
rotated (As=AtQA_s=A_tQ, QQ orthogonal)2.2071.10200
re-mixed (As=AtRA_s=A_tR, RR invertible)32.8000.9920.2920
rescaled (As=AtDA_s=A_tD, D>0D>0 diagonal)5.91200.5870
unrelated subspace2.0051.0010.9760.975

LGL_G is 0 in all three cases, while CKA gives 0.292 and 0.587 and MSE and cosine are also non-zero. This holds independently of any training setup.

3. Notation is in Table 1 and Sections 3.1–3.3. Some symbols are reused in the appendices for different quantities; we will distinguish them and define each at first use.

  • LgeoL_{\mathrm{geo}} is the task-specific geometric loss: point-wise spherical angular alignment for ASR, and the normalized Grassmann projector loss LGL_G for the LM experiment.
  • In the KL loss, bb indexes a batch example and uu a sequence position; ℓt,bu\ell_{t,bu} and ℓs,bu\ell_{s,bu} are the teacher and student vocabulary-logit vectors at that example and position.
  • Lproj=∥Ps−Pt∥F2L_{\mathrm{proj}}=\lVert P_s-P_t\rVert_F^2 is the unnormalized projector discrepancy, with Ps=UsUs⊤P_s=U_sU_s^\top and Pt=UtUt⊤P_t=U_tU_t^\top. The LM implementation uses LG=Lproj/(2k)L_G=L_{\mathrm{proj}}/(2k), averaged over examples and selected layer pairs.

4. Novelty. (i) ASR: parameter efficiency. (ii) LM, relative to prior work:

  • Orthogonal projection and subspace distillation exist. VkD [1] combines orthogonal projection with task-specific normalization; consensus subspace distillation [2] uses scale-invariant, low-rank consensus subspaces with their centre statistics. Ours lifts student representations into the teacher's ambient space with a learnable map, pools the hidden states of selected layer pairs, and compares the orthogonal projectors of the resulting subspaces.
  • We present this as a specific construction with a geometric interpretation, and do not claim to introduce subspace alignment or orthogonal projection.

5. Revise Appendix

  • We agree that matching subspaces does not establish token-level semantic correspondence. The projector loss measures the discrepancy between the teacher and student subspaces after lifting the student features into the teacher’s feature space. This gives the student a basis-invariant target under hidden-width mismatch. We will update Appendix to document the original mismatches and the correction.
Reviewer_XLWHOpenReview ↗

Re Rebuttal

Many thanks to the authors for their rebuttal. I still couldn't find an explanation about why the results are lower than baseline, so I keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.