- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes Grassmannian Subspace Alignment, a feature-based knowledge distillation approach for transferring intermediate representations from a larger teacher model to a smaller student with different hidden dimensions. The authors argue that conventional feature matching, which aligns representations coordinate by coordinate, can be unnecessarily restrictive because different feature bases may span the same underlying subspace. To address this, the method first uses a learnable linear projection to map student representations into the teacher's feature space. It then applies geometry-based alignment objectives: spherical angular alignment for speech recognition, which matches feature directions, and Grassmannian projector alignment for language modeling, which matches the subspaces spanned by intermediate representations. These objectives are combined with task supervision and logit knowledge distillation as final loss.
Critical strengths
- The paper presents a clear motivation for aligning representation subspaces rather than matching individual feature coordinates.
- The authors evaluate geometric representation alignment in both speech recognition and language modeling domains.
Critical weaknesses
- Missing comparisons with established feature-distillation methods for hidden-dimension mismatch, such as Projector+MSE and Projector+Cos, CKA-based alignment ( Dasgupta et al., 2025), and Flex-KD (Saadi et al., 2025). These baselines are particularly important because the main contribution is a new representation-alignment objective. Without comparisons under matched training conditions, it is difficult to establish whether Grassmannian alignment provides meaningful advantages over existing feature-distillation techniques when the student and teacher have different hidden sizes.
- The speech experiments (Table 2) use spherical alignment rather than Grassmannian alignment and lack a matched CE/KL-only baseline. In language modeling, the baseline and proposed method differ in both the representation loss and CE/KL training schedule (Section 4.3), preventing the contribution of Grassmannian alignment from being isolated. Furthermore, the language-model results (Table 3) show mixed performance.
Questions for authors
See weaknesses
2 comments
Thank you for the response. I am not sure if the reported CKA baseline reproduces the method of Dasgupta and Cohn (2025) or it uses the standard Linear CKA loss. Also, the additional experiments lack important details, such as the dataset and models used. The authors only mention that they were conducted in a separate small-scale setting. More importantly, CE+KL without feature alignment still achieves better results. This raises concerns about the benefit of the proposed method, and even among the tested feature-alignment methods, the improvements are marginal. The original language-model experiments also remain confounded by differences in training details, and the matched Apertus experiments are still incomplete. For ASR, the authors acknowledge decoder-position and vocabulary-index mismatches in the original KL computation, but no corrected experimental results are reported. These issues raise concerns about the reliability of the reported findings. Finally, no revised version of the manuscript is uploaded. I therefore maintain my original score.
Answers to the Questions one by one
Q1. Comparisons under matched training conditions
We agree that comparisons under matched conditions are needed. In a separate small-scale study, we matched the training data, student initialization, batch order, optimizer, training budget, fixed CE/KL weights, and evaluation protocol across the CE + KL baseline and the alignment arms. We evaluated each alignment method with six seeds at two weights, without method-specific tuning.
Grassmann had the lowest mean test CE among the tested alignment objectives at both weights. At λ = 0.1, it reduced mean test CE by 0.075 and improved test accuracy by 1.38 percentage points relative to Projector MSE. Its mean CE advantage over Projector Cosine was 0.023 at λ = 0.1 and 0.009 at λ = 1.0. CE + KL alone achieved lower mean test CE at both weights. Grassmann beat that baseline in 0/6 seeds at λ = 0.1 and 1/6 at λ = 1.0. The full results are in our reply to Urp3.
The synthetic comparison below illustrates the geometric distinction from linear CKA. The first three transformations preserve the column space in this example. Values are means over 50 draws.
Q2. Scope of the ASR and language-model results
Table 2 evaluates spherical alignment with LoRA in ASR. The experiment lacks a matched CE + KL-only baseline for isolating the effect of spherical alignment. Our code audit also found decoder-position and vocabulary-index mismatches in the original Whisper KL computation, as detailed in our reply to Urp3.
We agree that the original Apertus comparison in Section 4.3 changes both the feature loss and the CE/KL schedule. The mixed results in Table 3 therefore leave the effect of Grassmann alignment unresolved. We will describe them as preliminary. The separate six-seed study above holds the CE/KL weights fixed. The matched Apertus-0.5B comparison is still in progress.