- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for knowledge distillation (KD) leveraging the Chain-of-Thought (CoT) vectors. The method consists of two stages: in the first stage, CoT vectors or the compact representations of CoT are learned from the teacher model. The vectors are learned by switching a learnable vector on/off at each layer depending on the presence of the ground-truth CoT along with the query and the answer in the prompt. In the second stage, the student is trained to reason with the frozen teacher's CoT vectors. The method is evaluated on math and coding benchmarks to show the performance gain achieved by the proposed KD method.
Critical strengths
-
The method is straightforward to implement and reproduce. The paper is clearly written and easy to follow.
-
The proposed method achieves significant gain as compared with the baseline student (no KD) for various reasoning tasks.
Critical weaknesses
-
Limited technical contribution. The proposed method heavily relies on the original work on CoT vector [1] (cited in the paper). For example, the first stage of the proposed method is essentially identical to the procedure in [1]. Overall, this method simply transfers CoT vector method to KD without additional technical novelty or significant insights. The level of contribution appears limited considering the publication standard of ICLR conference.
-
Limited comparison with existing baselines. The evaluation section mostly reports the performance gain comparing with or without the proposed KD. This is more of a self-comparison (ablation). This work completely lacks comparison with existing KD methods (e.g., widely studied baselines like SeqKD, Generalized KD (GKD), DistiLLM-1,-2, Speculative KD, or recent methods concerning distilling long reasoning traces from teachers like P-ALIGN [2] or ORION [3]), although the work deals with traditional topic and benchmarks. Thus, it is difficult to judge whether the proposed method is a genuine enhancement of existing KD framework.
[1] Li, Li, et al. "Cot vectors: Transferring and probing the reasoning mechanisms of llms." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Liu, Zhenghao, et al. "Long-chain reasoning distillation via adaptive prefix alignment." Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026.
[3] Wu, Zhuoyang, et al. "Enhancing Long-Chain Reasoning Distillation through Error-Aware Self-Reflection." Findings of the Association for Computational Linguistics: ACL 2026. 2026.
Questions for authors
Please see weaknesses.
3 comments
Thanks
Thank you for the responses. They do not fully address my concerns, in particular the one on the technical novelty or how this work extends beyond CoT vectors. Thus I would like to keep my rating. I gave the score of 1 because the rating scale is coarse (only 4-point scale); in finer scales, I would rate this work a bit lower than weak reject, but not downright reject. I simply didn't have choice this time. I hope the authors could understand this aspect and do not take my rating as too much of discouragement.
Thank you for your candid follow-up.
We respect your perspective on technical novelty, and would like to briefly highlight two aspects where our work extends beyond prior CoT vectors and conventional distillation:
Bridging Incompatible Residual Spaces: Prior CoT vector transfer assumes identical residual dimensions between models, whereas transferring across model scales (e.g., 72B 7B or 7B 1.5B) requires aligning distinct representation spaces. Showing that a lightweight linear projection trained solely with final-answer cross-entropy—without teacher logits, hidden-state alignment, or intermediate traces—can preserve task-level reasoning guidance across dimensions provides a simple yet non-trivial empirical insight into cross-scale activation transfer.
Practical Utility in Low-Resource & Base-Model Settings: Unlike rationale-based or reflection-based distillation pipelines (which often require instruction-following capabilities to generate or filter intermediate trajectories), our two-stage design works directly on uninstructed base models with just 200 final-answer examples, while keeping the teacher's internal weights and generations completely unexposed during student adaptation.
Thank you again for your time, constructive critiques, and thoughtful engagement throughout the review process.
Thank you for your constructive evaluation. We address your specific concerns and questions below.
Technical Contribution, Practical Utility & Privacy Protection:
We agree that Stage 1 builds upon prior work (Li et al., 2026). However, our work introduces a highly practical and secure paradigm: achieving cross-scale reasoning representation transfer while keeping both teacher and student backbones completely frozen. By training only a lightweight linear projector on a tiny subset of data using final-answer supervision, our method avoids the high computational costs and complex pipeline requirements of traditional KD.
Crucially, this design offers exceptional privacy protection for the teacher model. Because student-side adaptation requires no teacher logits, no hidden-state alignment, and no intermediate solution traces, the teacher model's proprietary weights, internal representations, and text generations remain entirely unexposed.
Inclusion of Additional KD Baselines (LoRA, SFT, DistiLLM, P-ALIGN, and ORION):
Following your feedback, we have added extensive comparisons with LoRA, SFT, and DistiLLM under a matching 200-example budget on MATH, where our method consistently outperforms them (59.13% vs. 48.98% / 47.94% / 49.20%).
Regarding recent long-chain reasoning distillation methods such as P-ALIGN and ORION, we attempted to implement them under our base-student setting but found they experience severe bottlenecks during the data-construction phase (yielding a survival rate of only ≈1%). Specifically, P-ALIGN requires uninstructed base students to execute meta-tasks like self-assessing prefix sufficiency (e.g., generating [ENOUGH]/[NOT_ENOUGH] markers), while ORION requires step-by-step error reflection. Base students fail to follow these auxiliary formatting instructions, preventing the successful creation of a training set. We have duly cited both works in our related work section.