- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces RoKA, a framework for cross-width hidden-state distillation. It combines an alignment operator that maps student representations into the teacher space with a robust residual objective that downweights gradients from large teacher–student discrepancies. In the encoder-only setting, the student-anchored SVD variant achieves the highest unweighted nine-metric average, outperforming the strongest baseline, CKA, by 1.54 percentage points.
Critical strengths
- The paper clearly decomposes cross-width hidden-state distillation into representation alignment and robust residual optimization.
- In some settings, simple RoKA alignment variants offer a favorable performance–efficiency trade-off. For example, one-padding achieves the best XSUM results with substantially less training overhead than CKA.
Critical weaknesses
- Insufficient ablation studies. Table 5 compares MSE, fixed- , and learned- , but only for one-padding on XSUM and SVD-S on MNLI. Therefore, it remains unclear whether the benefit of the robust loss generalizes across different alignment operators and tasks. The same comparison should be extended to other representative operator–task combinations under identical settings.
- The rationale for downweighting large residuals is insufficiently justified. RoKA assigns smaller gradient weights to large teacher–student residuals, thereby making them contribute less to student updates. However, these residuals may represent difficult but valuable knowledge rather than noise. The paper should analyze whether the down-weighted residuals are truly harmful or are precisely the signals that the student needs to learn.
- The necessity of learning is not sufficiently demonstrated. In Table 5, learned provides only a marginal improvement over fixed =50, while Table 7 compares it only with fixed =1.
- The paper does not provide a principled way to select an alignment operator. The best-performing operator varies across architectures and tasks, leaving practitioners to test all candidates.
Questions for authors
- Can the authors provide evidence that the robust objective is effective across different alignment operators and tasks, beyond one-padding on XSUM and SVD-S on MNLI?
- When applying RoKA to a new task, can the authors provide a principled way to select an alignment operator, or must practitioners test all candidates?
2 comments
Thank you for the detailed response and for running the additional experiments. The new ablations address my request for broader comparisons. Here is how they affect the four weaknesses in my original review.
Weakness 1 — Benefit of the robust objective. In the new CoLA, MRPC, and RTE ablations reported in the response, the robust objective improves on MSE in only 3 of 12 comparisons. I appreciate the gains on XSUM, including over the MSE run with a lower hidden-loss weight. Taken together, the results suggest that the benefit depends on the task and alignment configuration.
Weakness 2 — Are the downweighted residuals harmful? The gradient derivation shows how the loss reduces the weight on large residuals. The experiments compare full objectives, which differ in both overall gradient scale and the relative weight assigned to large residuals. They therefore leave my original question open: does downweighting those particular residuals remove harmful supervision or suppress useful signals that are difficult for the student to learn?
Weakness 3 — What does learning ν add? In the new zero-padding and SVD-S ablations, learned ν and fixed ν = 50 differ by at most 0.32 points in mean score. On MNLI, learned ν improves average accuracy by 0.91 points over fixed ν = 1 (Table 7), but by only 0.02 points over fixed ν = 50 (Table 5). These results do not establish a performance advantage from learning ν over fixing it at 50. I agree with the authors’ plan to present learning ν as optional.
Weakness 4 — How should practitioners select an alignment operator for a new task? The proposed comparison on held-out data gives practitioners a shortlist to try. Its practical value depends on two quantities that the response does not report: the total training cost of evaluating the candidates and how the selected operator performs on data separate from the selection set.
I remain unconvinced by the claimed methodological novelty of RoKA. The gradient interaction follows directly from applying a radial penalty to the full residual vector. The XSUM result shows that this penalty can help in one configuration. For the interaction to support a new design framework, I would expect the analysis to explain when downweighting large residuals is beneficial or to guide the choice of an alignment operator on a new task. The response leaves both questions open, and the experiments show little added value from learning ν over fixing it at 50. For these reasons, my overall recommendation remains unchanged.
Thank you for the concrete suggestions and for noting the clear decomposition into alignment and residual optimization, and the efficiency of one padding. We ran the requested extension with matched settings within each operator configuration, clarifying where the robust loss helps: on XSUM it beats MSE at both tested MSE hidden weights, while on CoLA, MRPC, and RTE its effect is setting-dependent, and learned ν yields mean scores similar to fixed ν=50. The revision will present the robust objective as a setting-dependent option and replace the learned-ν adaptivity and trajectory claims with these observations. Our central contribution is the analysis of how alignment maps and residual losses interact, together with empirical guidance on alignment choices. SVD-S outperforms the evaluated zero-padding configurations across all nine task-objective pairs, although pooling and layer reduction also differ.
Q1. Is the robust objective effective across operators and tasks, beyond one-padding on XSUM and SVD-S on MNLI?
To extend the analysis in Table 5, we conducted additional experiments on CoLA, MRPC, and RTE using zero padding and SVD-S, each with its original pooling and layer reduction settings. Within each configuration, only the residual objective changes (five-seed mean ± standard deviation, final-epoch validation; CoLA MCC, MRPC F1, RTE accuracy):
The robust objective does not consistently outperform MSE across operators and tasks; its benefits are setting-dependent. Of 12 robust-minus-MSE comparisons (mean paired differences), three are positive (largest +0.43), eight negative (lowest −1.13), and one zero. SVD-S exceeds zero padding in all nine task-objective pairs, although this is a configuration-level comparison because pooling and layer reduction also differ.
For our additional linear projector experiments, learned-ν is below MSE on CoLA (−2.20) and MRPC (−1.25) but above it on RTE (+1.15) with identity initialization; random initialization shows the same pattern (−0.92, −0.72, +0.43). On XSUM, both robust variants exceed MSE (learned ν: 35.66; fixed ν=50: 35.57; MSE: 33.91 ROUGE-Lsum) and the rerun MSE baseline with hidden weight 0.3 (35.01) in all five seeds. Gradient clipping was disabled consistently across these runs, and only two MSE weights were tested. On MNLI, average matched/mismatched accuracy differs by less than 0.04 points across MSE, fixed ν=50, and learned ν.
Large residuals. We agree that large residuals may carry useful signal. The influence function establishes attenuation, not that these residuals are noise. Since the comparisons also involve changes in gradient scale and optimization, whether individual large residuals help or hurt remains unresolved. We will clarify this limitation in the revision.
Learning ν. The results support your concern. In the table above, learned ν ends at 48.1-49.5, and mean scores differ from fixed ν=50 by at most 0.32 points. For the identity-initialized projector on CoLA, fixed ν=50 scores 40.46 versus 39.86 for learned ν (95% paired confidence interval for fixed minus learned: −2.34 to 3.54). On MNLI, learned ν approaches the lower bound of 1, yet its average matched/mismatched accuracy is within 0.02 points of fixed ν=50. Table 7 compares only with fixed ν=1 and does not establish an advantage over fixed ν=50. The revision will present learning ν as optional and replace the trajectory interpretation with these observations.
Q2. A principled way to select an alignment operator?
Our results suggest empirical starting points: SVD-S for the evaluated encoder classification tasks, one padding for XSUM when training cost matters, and CKA for CNN/DailyMail. For a new task, a small, prespecified comparison of these candidates on held-out training data may be practical, although its effectiveness and computational cost remain to be evaluated. We will revise the paper to clarify these recommendations and acknowledge that developing a principled selector remains future work.