acceptodds
Withdrawn from ICLR 2027

RoKA: Robust Knowledge Alignment for Cross-Width Hidden-State Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Distilling hidden states from a wide teacher to a narrower student requires comparing representations with unequal widths. Learned projections address this mismatch but add parameters and geometric bias, whereas relational objectives such as centered kernel alignment (CKA) avoid coordinate-wise matching altogether. We introduce RoKA, a framework for cross-width hidden-state distillation that separates two coupled design choices: the geometry of dimensional alignment and the influence of the resulting residuals on optimization. RoKA combines fixed, learned, and spectral alignment operators with a Student-t-inspired radial penalty that attenuates the gradient contribution of large operator-induced residuals. We further show that even fixed coordinates introduced by dimensional expansion can alter gradients on the retained student coordinates. Across BERT, BART, and GPT-2 teacher–student pairs, we evaluate RoKA against normal knowledge distillation (Normal KD) and CKA over five seeds. In the encoder-only complete-method comparison, student-anchored SVD achieves the highest unweighted nine-metric average, 80.13 against CKA’s 78.59. On XSUM, one padding outperforms CKA on all ROUGE metrics at 1.05× versus CKA’s 2.30× Normal-KD training time in a single-GPU probe. At equal hidden coefficients, robust residual optimization outperforms the corresponding quadratic objective on XSUM; on MNLI, the residual objectives perform similarly, although learned ν exceeds fixed ν = 1 by 0.91 points despite approaching that bound. CKA remains strongest on CNN/DailyMail, and decoder-only differences are small, indicating that no single alignment operator dominates across settings. Overall, RoKA positions explicit alignment geometry and robust residual influence as practical, interacting design axes for cross-width hidden-state distillation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Knowledge Distillation, Hidden-State Distillation, Representation Alignment, Cross-Width Distillation, Robust Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_d5ZyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RoKA, a framework for cross-width hidden-state distillation. It combines an alignment operator that maps student representations into the teacher space with a robust residual objective that downweights gradients from large teacher–student discrepancies. In the encoder-only setting, the student-anchored SVD variant achieves the highest unweighted nine-metric average, outperforming the strongest baseline, CKA, by 1.54 percentage points.

Critical strengths
  1. The paper clearly decomposes cross-width hidden-state distillation into representation alignment and robust residual optimization.
  2. In some settings, simple RoKA alignment variants offer a favorable performance–efficiency trade-off. For example, one-padding achieves the best XSUM results with substantially less training overhead than CKA.
Critical weaknesses
  1. Insufficient ablation studies. Table 5 compares MSE, fixed- ν\nu, and learned- ν\nu, but only for one-padding on XSUM and SVD-S on MNLI. Therefore, it remains unclear whether the benefit of the robust loss generalizes across different alignment operators and tasks. The same comparison should be extended to other representative operator–task combinations under identical settings.
  2. The rationale for downweighting large residuals is insufficiently justified. RoKA assigns smaller gradient weights to large teacher–student residuals, thereby making them contribute less to student updates. However, these residuals may represent difficult but valuable knowledge rather than noise. The paper should analyze whether the down-weighted residuals are truly harmful or are precisely the signals that the student needs to learn.
  3. The necessity of learning ν\nu is not sufficiently demonstrated. In Table 5, learned ν\nu provides only a marginal improvement over fixed ν\nu=50, while Table 7 compares it only with fixed ν\nu=1.
  4. The paper does not provide a principled way to select an alignment operator. The best-performing operator varies across architectures and tasks, leaving practitioners to test all candidates.
Questions for authors
  1. Can the authors provide evidence that the robust objective is effective across different alignment operators and tasks, beyond one-padding on XSUM and SVD-S on MNLI?
  2. When applying RoKA to a new task, can the authors provide a principled way to select an alignment operator, or must practitioners test all candidates?
2 comments

Thank you for the concrete suggestions and for noting the clear decomposition into alignment and residual optimization, and the efficiency of one padding. We ran the requested extension with matched settings within each operator configuration, clarifying where the robust loss helps: on XSUM it beats MSE at both tested MSE hidden weights, while on CoLA, MRPC, and RTE its effect is setting-dependent, and learned ν\nu yields mean scores similar to fixed ν=50\nu=50. The revision will present the robust objective as a setting-dependent option and replace the learned-ν\nu adaptivity and trajectory claims with these observations. Our central contribution is the analysis of how alignment maps and residual losses interact, together with empirical guidance on alignment choices. SVD-S outperforms the evaluated zero-padding configurations across all nine task-objective pairs, although pooling and layer reduction also differ.

Q1. Is the robust objective effective across operators and tasks, beyond one-padding on XSUM and SVD-S on MNLI?

To extend the analysis in Table 5, we conducted additional experiments on CoLA, MRPC, and RTE using zero padding and SVD-S, each with its original pooling and layer reduction settings. Within each configuration, only the residual objective changes (five-seed mean ±\pm standard deviation, final-epoch validation; CoLA MCC, MRPC F1, RTE accuracy):

TaskConfig.MSEFixed ν=50\nu=50Learned ν\nu
CoLAZero pad.41.32±1.3441.32\pm1.3440.44±0.7540.44\pm0.7540.75±1.6840.75\pm1.68
CoLASVD-S46.59±0.9946.59\pm0.9946.22±1.1546.22\pm1.1546.19±1.1346.19\pm1.13
MRPCZero pad.86.77±0.4486.77\pm0.4485.91±0.3885.91\pm0.3885.64±0.5685.64\pm0.56
MRPCSVD-S89.61±0.2789.61\pm0.2789.98±0.4689.98\pm0.4690.04±0.4290.04\pm0.42
RTEZero pad.58.13±1.6458.13\pm1.6458.27±1.5758.27\pm1.5758.13±1.5058.13\pm1.50
RTESVD-S60.72±0.8660.72\pm0.8660.36±1.0060.36\pm1.0060.50±1.0960.50\pm1.09

The robust objective does not consistently outperform MSE across operators and tasks; its benefits are setting-dependent. Of 12 robust-minus-MSE comparisons (mean paired differences), three are positive (largest +0.43+0.43), eight negative (lowest −1.13-1.13), and one zero. SVD-S exceeds zero padding in all nine task-objective pairs, although this is a configuration-level comparison because pooling and layer reduction also differ.

For our additional linear projector experiments, learned-ν\nu is below MSE on CoLA (−2.20-2.20) and MRPC (−1.25-1.25) but above it on RTE (+1.15+1.15) with identity initialization; random initialization shows the same pattern (−0.92-0.92, −0.72-0.72, +0.43+0.43). On XSUM, both robust variants exceed MSE (learned ν\nu: 35.66; fixed ν=50\nu=50: 35.57; MSE: 33.91 ROUGE-Lsum) and the rerun MSE baseline with hidden weight 0.3 (35.01) in all five seeds. Gradient clipping was disabled consistently across these runs, and only two MSE weights were tested. On MNLI, average matched/mismatched accuracy differs by less than 0.04 points across MSE, fixed ν=50\nu=50, and learned ν\nu.

Large residuals. We agree that large residuals may carry useful signal. The influence function establishes attenuation, not that these residuals are noise. Since the comparisons also involve changes in gradient scale and optimization, whether individual large residuals help or hurt remains unresolved. We will clarify this limitation in the revision.

Learning ν\nu. The results support your concern. In the table above, learned ν\nu ends at 48.1-49.5, and mean scores differ from fixed ν=50\nu=50 by at most 0.32 points. For the identity-initialized projector on CoLA, fixed ν=50\nu=50 scores 40.46 versus 39.86 for learned ν\nu (95% paired confidence interval for fixed minus learned: −2.34-2.34 to 3.543.54). On MNLI, learned ν\nu approaches the lower bound of 1, yet its average matched/mismatched accuracy is within 0.02 points of fixed ν=50\nu=50. Table 7 compares only with fixed ν=1\nu=1 and does not establish an advantage over fixed ν=50\nu=50. The revision will present learning ν\nu as optional and replace the trajectory interpretation with these observations.

Q2. A principled way to select an alignment operator?

Our results suggest empirical starting points: SVD-S for the evaluated encoder classification tasks, one padding for XSUM when training cost matters, and CKA for CNN/DailyMail. For a new task, a small, prespecified comparison of these candidates on held-out training data may be practical, although its effectiveness and computational cost remain to be evaluated. We will revise the paper to clarify these recommendations and acknowledge that developing a principled selector remains future work.

Thank you for the detailed response and for running the additional experiments. The new ablations address my request for broader comparisons. Here is how they affect the four weaknesses in my original review.

Weakness 1 — Benefit of the robust objective. In the new CoLA, MRPC, and RTE ablations reported in the response, the robust objective improves on MSE in only 3 of 12 comparisons. I appreciate the gains on XSUM, including over the MSE run with a lower hidden-loss weight. Taken together, the results suggest that the benefit depends on the task and alignment configuration.

Weakness 2 — Are the downweighted residuals harmful? The gradient derivation shows how the loss reduces the weight on large residuals. The experiments compare full objectives, which differ in both overall gradient scale and the relative weight assigned to large residuals. They therefore leave my original question open: does downweighting those particular residuals remove harmful supervision or suppress useful signals that are difficult for the student to learn?

Weakness 3 — What does learning ν add? In the new zero-padding and SVD-S ablations, learned ν and fixed ν = 50 differ by at most 0.32 points in mean score. On MNLI, learned ν improves average accuracy by 0.91 points over fixed ν = 1 (Table 7), but by only 0.02 points over fixed ν = 50 (Table 5). These results do not establish a performance advantage from learning ν over fixing it at 50. I agree with the authors’ plan to present learning ν as optional.

Weakness 4 — How should practitioners select an alignment operator for a new task? The proposed comparison on held-out data gives practitioners a shortlist to try. Its practical value depends on two quantities that the response does not report: the total training cost of evaluating the candidates and how the selected operator performs on data separate from the selection set.

I remain unconvinced by the claimed methodological novelty of RoKA. The gradient interaction follows directly from applying a radial penalty to the full residual vector. The XSUM result shows that this penalty can help in one configuration. For the interaction to support a new design framework, I would expect the analysis to explain when downweighting large residuals is beneficial or to guide the choice of an alignment operator on a new task. The response leaves both questions open, and the experiments show little added value from learning ν over fixing it at 50. For these reasons, my overall recommendation remains unchanged.

Reviewer_GMJLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RoKA (Robust Knowledge Alignment), a framework for hidden-state knowledge distillation between teacher and student models with different hidden dimensions. RoKA considers several dimensional alignment operators, including fixed padding, learnable padding, linear projection, and student- or teacher-anchored SVD reconstruction. The resulting teacher–student feature differences are optimized using a Student-t-inspired robust loss, which reduces the gradient contribution of large residuals. The method also learns a parameter controlling the robustness of this loss.

Critical strengths
  1. The paper evaluates several hidden-state alignment methods across BERT, BART, and GPT-2 using five random seeds. It also provides controlled comparisons between MSE and robust losses, showing meaningful improvements on XSUM while acknowledging limitations in other settings.
Critical weaknesses
  1. The paper mainly compares against standard KD and CKA, but does not include other feature-distillation methods designed to handle teacher–student hidden-dimension mismatches, such as Flex-KD (Saadi et al., 2025), Projector+MSE, and Projector+Cos. Including these baselines would help establish the effectiveness of RoKA relative to existing approaches.
  2. Limited scalability and inconsistent improvements. The largest evaluated teacher is GPT-2 XL (approximately 1.5B parameters), with no experiments on more recent large language models. The improvements are also inconsistent across datasets: CKA performs better on CNN/DailyMail, and the decoder-only results show only marginal perplexity differences.
Questions for authors
  1. See weaknesses.
  2. Could the authors clarify what the main methodological novelty is?
4 comments

Thank you for the concrete baseline suggestions and for recognizing our five-seed evaluation and the XSUM gains. We ran all three requested baselines, Projector+MSE, Projector+Cos, and Flex-KD adapted from the authors' released code, on CoLA, MRPC, and RTE with five seeds. Student-anchored SVD (SVD-S) has the highest mean score on every task, and even the strongest baseline, the identity-initialized projector, stays below it (Q1). Our novelty is to treat width alignment and residual optimization as one coupled design problem (Q2). Q1 also covers the limits you raise: GPT-2 XL is our largest teacher, and CKA is best on CNN/DailyMail.

Q1. See weaknesses: comparison with Flex-KD, Projector+MSE, Projector+Cos; scalability and consistency.

We ran these baselines and SVD-S with a BERT-base teacher, a 12-layer width-512 student, and a ten-epoch schedule (five-seed mean ±\pm standard deviation, final-epoch validation). Cos is cosine loss; robust is the paper's Student-t-inspired loss with learned ν\nu:

MethodCoLA MCCMRPC F1RTE acc.
Projector, identity init + MSE42.07±0.7042.07\pm0.7087.36±0.7687.36\pm0.7657.70±1.0457.70\pm1.04
Projector, identity init + Cos28.29±2.4528.29\pm2.4582.51±1.0682.51\pm1.0656.26±1.7056.26\pm1.70
Projector, identity init + robust39.86±1.6139.86\pm1.6186.11±0.4986.11\pm0.4958.85±1.7258.85\pm1.72
Projector, random init + MSE24.52±1.6224.52\pm1.6280.73±1.0980.73\pm1.0955.18±1.1555.18\pm1.15
Projector, random init + Cos24.46±2.1224.46\pm2.1282.53±0.2782.53\pm0.2756.47±1.2256.47\pm1.22
Projector, random init + robust23.61±2.4823.61\pm2.4880.01±1.3280.01\pm1.3255.61±1.7955.61\pm1.79
Flex-KD (adapted)25.82±2.2625.82\pm2.2685.85±0.7985.85\pm0.7954.89±1.2154.89\pm1.21
SVD-S + MSE46.59±0.9946.59\pm0.9989.61±0.2789.61\pm0.2760.72±0.8660.72\pm0.86
SVD-S + robust (learned ν\nu)46.19±1.1346.19\pm1.1390.04±0.4290.04\pm0.4260.50±1.0960.50\pm1.09

SVD-S has the highest mean in each column (with MSE on CoLA and RTE, with robust loss on MRPC). Projectors are bias-free linear maps (512 to 768) shared across layers, initialized as [I;0][I;0] (learning rate 10−510^{-5}) or randomly (2×10−42\times10^{-4}, from the TinyBERT script released with HSM), with the pooling and layer reduction of our padding configurations; they are not a TinyBERT reproduction and were not tuned further. Identity initialization scores higher than random initialization with MSE and robust loss on all three tasks (with cosine, only on CoLA); initialization and learning rate change together. Flex-KD selects 512 teacher coordinates by gradient importance and uses the authors' released correlation loss and weights (0.5 cross-entropy (CE) + 0.5 correlation) with our models and schedule; our length normalization of the importance scores changes at most 4 of the 512 selected coordinates, and the appendix will document all adaptation choices. As Flex-KD uses CE with last-layer correlation and SVD-S uses KD with multi-layer loss, this compares complete methods.

Scalability: GPT-2 XL (about 1.5B parameters) is our largest teacher; extending RoKA to recent large language models is future work, and the revision will state this scope. Consistency: we agree that gains depend on the setting; CKA is best on CNN/DailyMail, and decoder-only differences are small. The revision will report these as results and recommend a configuration per setting.

Q2. What is the main methodological novelty?

RoKA treats alignment geometry and residual weighting as coupled design choices, rather than relying on fixed operator–loss combinations as in the compared baselines. Our analysis reveals two mechanisms. First, under the Student-t-inspired penalty, fixed padding rescales gradients on retained student coordinates by (ν+1)/(ν+∥rshared∥2+∥radded∥2)(\nu+1)/(\nu+\Vert r_{\mathrm{shared}}\Vert^2+\Vert r_{\mathrm{added}}\Vert^2), even though the added coordinates are constant. Second, the SVD anchor determines the supervision path: with teacher factors fixed, the student influences reconstruction through at most kk singular values per layer and batch, whereas student anchoring also propagates supervision through student singular vectors.

A separate five-seed CoLA experiment provides evidence consistent with the second mechanism: student anchoring achieves 46.17 MCC versus 20.82 for teacher anchoring, despite having smaller radial multipliers at all monitored checkpoints. This supports, but does not prove, our supervision-path hypothesis. Together with the results across BERT, BART, and GPT-2 (SVD-S: 80.13 vs. CKA: 78.59 on the unweighted encoder average; one padding leading on XSUM ROUGE at lower measured training time than CKA), these findings support RoKA's methodological contribution. Since pooling and layer reduction also differ across configurations, we interpret the performance comparisons as setting-specific. We will revise the introduction to clarify these contributions.

Thank you for the response and the additional experiment. However, CKA is missing from the new comparison table. Moreover, these additional experiments are limited to three GLUE tasks. A broader evaluation, particularly on summarization or larger datasets, would provide stronger evidence of the method's effectiveness. The new results also raise questions about the contribution of the robust loss itself. SVD-S with MSE performs better than the robust variant on CoLA and RTE, while the robust variant shows only a small improvement on MRPC. Scalability remains a critical concern. The evaluation is limited to BERT, BART, and GPT-2, with no experiments on more recent or larger language models. The authors acknowledge this limitation but do not provide additional experiments. Finally, no revised manuscript has been uploaded to incorporate the additional results and clarifications. Overall, I believe my main concerns are not adequately addressed, and my original score of 2 is still reasonable.

Thank you for the follow-up. We have uploaded the revised manuscript; the main changes are listed at the end.

CKA in the new comparison. CKA was run on CoLA, MRPC, and RTE with the same teacher, student, and 10-epoch schedule (Table 2), but we left it out of the new table, which reports the final epoch. The final epochs of those five CKA runs are:

MethodCoLA MCCMRPC F1RTE acc.
CKA35.81±1.3935.81\pm1.3987.51±0.7187.51\pm0.7159.21±1.8159.21\pm1.81
SVD-S + MSE46.59±0.9946.59\pm0.9989.61±0.2789.61\pm0.2760.72±0.8660.72\pm0.86
SVD-S + robust (learned ν\nu)46.19±1.1346.19\pm1.1390.04±0.4290.04\pm0.4260.50±1.0960.50\pm1.09

SVD-S + MSE is higher by 10.78 MCC, 2.10 F1, and 1.51 accuracy. The CKA runs are an earlier, separate run set, so this compares means, and the RTE gap is small. With validation-selected checkpoints, Table 2 also shows that SVD-S outperforms CKA on all three tasks (46.50/90.04/61.15 vs. 37.12/88.07/60.07). The revised Appendix C.2 adds the CKA row.

Breadth beyond three GLUE tasks. We have not evaluated the added baselines on summarization or larger datasets, because of limited compute. All rebuttal experiments ran on one server with two 32 GB GPUs. The five-seed XSUM weight control in Appendix B took about 61 hours on both GPUs (about 12 hours per seed), and a GPU hardware fault on November 11 interrupted one seed, which had to be restarted. One CoLA, MRPC, or RTE run takes 2 to 25 minutes. Your follow-up arrived less than three days before the discussion ends, and one more five-seed XSUM configuration alone takes about 2.5 days. We therefore used the discussion period for the three-task GLUE comparisons; broader baseline coverage remains a limitation. The broader results below are from the main evaluation against CKA and Normal KD: the eight GLUE tasks evaluated in the paper, including MNLI and QQP; XSUM and CNN/DailyMail with BART; and fixed subsets of WikiText-103 and OpenWebText with GPT-2, with five seeds per configuration. SVD-S has the highest unweighted average across the nine encoder metrics (80.13 against CKA's 78.59). One padding beats CKA on all XSUM ROUGE metrics (ROUGE-Lsum 35.68 against 35.33), with relative training times of 1.05× for one padding and 2.30× for CKA, compared with Normal KD, in the single-GPU XSUM timing probe. CKA is best on CNN/DailyMail.

Contribution of the robust loss. The added results show no consistent benefit from the robust penalty across these three tasks. In the configuration × objective grid (Appendix C.1), robust minus MSE is negative in 8 of 12 paired comparisons, and SVD-S + MSE is higher than SVD-S + robust on CoLA and RTE. On MNLI, MSE at weight 0.4 reaches 83.62 against 83.52 for the learned robust penalty at weight 1.0 (difference −0.10-0.10, 95% CI [−0.31,0.12][-0.31, 0.12]). In the separate XSUM one-padding ablation, the learned robust loss at hidden weight 1.0 reaches 35.66 ROUGE-Lsum, against 35.01 for MSE at weight 0.3 (+0.65, 95% paired CI [0.53,0.77][0.53, 0.77], 5/5 seeds) and 33.91 for MSE at weight 1.0; only two MSE weights were tested. The revision therefore presents the penalty as an option to test per setting. In the same grid, SVD-S exceeds zero padding under all three objectives, by 5.27–5.79 MCC on CoLA and 2.84–4.40 F1 on MRPC (5/5 seeds). Because the configurations also differ in pooling and layer reduction, this does not isolate alignment geometry.

Scalability. We have no experiments with recent or larger LLMs. Our implementation uses FP32 model parameters without mixed-precision autocasting and reads teacher hidden states at every training step. A 7B teacher alone would require about 28 GB for FP32 weights on a 32 GB GPU, before student training and activations. We could not implement and validate lower-precision or sharded training during the discussion period. Matching our decoder-only setup would also require dataset-specific adaptation of the large teacher. The largest teacher is GPT-2 XL (about 1.5B parameters); the revision states this scope in the contributions and limitations.

Revised manuscript. Main changes:

  • Introduction and contributions: width alignment and residual optimization treated as one design problem; setting-specific recommendations (SVD-S for encoder classification, one padding for XSUM when training cost matters, CKA for CNN/DailyMail).
  • Robust penalty and learned ν\nu presented as optional; claims about ν\nu adaptivity replaced by the measured endpoints and the fixed-ν=50\nu=50 comparison.
  • Appendix B: hidden-loss weight controls on XSUM and MNLI.
  • Appendix C: configuration × objective grid, learned projectors, Flex-KD with the adaptation choices, CKA, an optimal-transport baseline, and the SVD student/teacher diagnostic.
  • Related work (Flex-KD, OT, subspace distillation) and limitations (model scale, three-task scope of the added baselines).

Thank you for the additional clarification and experiments. I understand that evaluating the method on larger datasets and models is computationally expensive. However, I believe such experiments are crucial to assess the effectiveness and generalizability of the proposed method. Moreover, the reported results show that MSE outperforms the robust variant in several settings, which raises concerns about the actual benefit of the proposed method. I am also concerned about the novelty. Overall, while I appreciate the authors' efforts to address my concerns, I think the remaining issues are significant. Therefore, I maintain my original score.

Reviewer_m95YOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is evaluating and conducting empirical ablation studies to find if direct co-ordinate matching in KD can outperform relational alignment metrics like CKA. Also, it conducts experiments to check which operators/spectral anchors deliver the best performance. Furthermore, it empirically checks if robust penalty (student-t) helps over standard L2 loss. All three empirical evaluations are combined into a framework called Robust Knowledge Alignment (RoKA) framework.

Critical strengths

The experiments/evaluations conducted are across multiple architecture types, datasets, various padding/co-ordinate matching configurations and seem statistically reliable.

The three questions of enquiry are quite interesting, and it will also need some motivations added.

Critical weaknesses

I couldn't find novel part in the paper. There are a few ablation studies but conclusions are not clear to me.

For the encoder-only models, the SVD-student seems to perform the best. That is counter-intuitive and conflicting with the paper’s own arguments of the gradient dampening due to zero-padding. The SVD-student is zero-padded, and the student-t interaction is heavily damped due to unlearnable padding. Also, the co-ordinate alignment is arbitrary. The SVD-teacher has none of these problems. It gets sparse but uncorrupted gradient signals. It would be great if the authors clarify this.

The results are presented but I couldn't find any explanation as to why the behaviour is the way it is. Some architectures, some datasets CKA is better, and other places a different coordinate matching operator is better, so I am not sure why.

Not sure if I get the conclusion of the paper. Neither of the operators or spectral anchors are hands down the best compared to relational alignment (CKA). The contribution is said to be a practical design framework, but not sure if I understand that. Would be great if you please clarify.

I guess the paper can be written in a clearer way, and the contribution of the paper (RoKA framework) can be established better.

Questions for authors

Would be great if you could explain the motivation behind the RoKA framework?

Why is the SVD-teacher inferior to the SVD-student?

Is there any theoretical reasoning or basic interpretation of the results?

Why is the comparison done only with CKA? Why not other recent relational alignment metrics based on optimal transport or subspace based distillation techniques? It would be great if the authors please clarify this.

Also, please see weaknesses.

2 comments

Thank you for the careful review and for noting the breadth and reliability of our experiments. RoKA's novelty is to treat width alignment and residual optimization as one coupled design problem: the alignment map determines which student quantities receive hidden-state supervision, and a non-quadratic loss also changes how strongly they are updated. We analyze this coupling and evaluate it on BERT, BART, and GPT-2 with five seeds. Our conclusion is a set of setting-specific recommendations. Student-anchored SVD (SVD-S) has the best unweighted nine-metric encoder-only average (80.13 vs. 78.59 for CKA) and leads our new encoder comparisons. In the new CoLA, MRPC, and RTE runs with MSE, fixed ν=50\nu=50, and learned ν\nu, its gap over zero padding exceeds the loss differences within either configuration, although pooling and layer reduction also differ. One padding has the best XSUM ROUGE at lower measured training time than CKA, and CKA is best on CNN/DailyMail. The revised introduction and conclusion will state this explicitly.

Q1. Motivation behind the RoKA framework?

Cross-width hidden-state distillation requires comparing representations of different dimensionalities. Relational objectives such as CKA avoid explicit alignment, whereas alignment-based methods penalize residual differences. However, the alignment operator determines the residual geometry and magnitude, while the loss function controls their influence on optimization. RoKA addresses this interaction by jointly considering alignment geometry and residual weighting.

Q2. Why is the SVD-teacher inferior to the SVD-student?

The zero suffix adds radial attenuation in student mode, but attenuation alone does not determine gradient effectiveness. For a residual rr, the student gradient is proportional to J⊤w(r)rJ^\top w(r)r, where JJ is the derivative of the alignment map. With teacher SVD factors fixed, the student influences reconstruction through at most kk singular values per layer and batch, whereas SVD-S also propagates supervision through student singular vectors. Both modes reconstruct layer-normalized states against raw teacher states, so this mismatch is not unique to teacher mode.

To examine attenuation, we conducted an additional five-seed CoLA experiment with matched initializations, measuring residuals and radial multipliers at epochs 0, 5, and 10 on 32 fixed training examples (non-padding tokens). Across all seeds and checkpoints, teacher mode showed smaller mean squared residuals and larger radial multipliers, yet substantially lower validation MCC (20.82±0.9020.82\pm0.90 vs. 46.17±1.1346.17\pm1.13). Thus, radial attenuation alone cannot explain the performance gap. These findings are consistent with our anchor-bandwidth hypothesis, although they do not establish causality. We acknowledge that SVD-S retains arbitrary coordinate correspondence and limit our conclusions to the evaluated settings.

Q3. Theoretical reasoning or interpretation of the results?

Our two analytical results explain what each operator changes. With fixed padding, the Student-t gradient on shared coordinates is scaled by (ν+1)/(ν+∥rshared∥2+∥radded∥2)(\nu+1)/(\nu+\Vert r_{\mathrm{shared}}\Vert^2+\Vert r_{\mathrm{added}}\Vert^2), so constants the student cannot change still damp its updates; and the SVD anchor sets the supervision path (Q2). Which operator wins in a given setting is an empirical question: tasks, model families, and compression ratios vary together, so we report the rankings without attributing them to one factor. By "practical design framework" we mean the setting-specific starting points listed above, with CKA as one of them.

Q4. Why compare only with CKA, not OT- or subspace-based methods?

We chose CKA because HSM (Dasgupta & Cohn, 2025) uses it for width-agnostic language-model distillation. We added an optimal transport (OT) baseline, entropic Gromov-Wasserstein (GW) distillation, which compares token relationships within each example and so needs no width match. It uses our CKA baseline's training settings and layer mapping; CKA instead compares valid tokens across the batch. Five-seed mean ±\pm standard deviation with the checkpoint rule of Table 2 (CKA and SVD-S rows from Table 2):

MethodCoLA MCCMRPC F1RTE acc.
GW (new)27.89±1.0727.89\pm1.0783.46±0.3783.46\pm0.3756.69±1.5056.69\pm1.50
CKA37.12±2.8037.12\pm2.8088.07±0.3888.07\pm0.3860.07±1.6160.07\pm1.61
SVD-S46.50±1.1546.50\pm1.1590.04±0.4290.04\pm0.4261.15±0.8861.15\pm0.88

GW is below CKA and SVD-S on all three tasks. As a generic, untuned OT formulation, it shows how this baseline compares under our protocol; published OT-KD variants may behave differently. The revision will discuss OT and subspace distillation in related work and list a subspace comparison as future work.

Reviewer_m95YOpenReview ↗

Official Comment by Reviewer m95Y

Thanks very much for the rebuttal. However, the reasoning for SVD teacher being inferior to SVD student remains unclear to me. I also couldn't find any theoretical interpretation provided. Lastly, would you please clarify, if the new comparisons are taking untuned OT formulations or the published OT-KD variants?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.