- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper questions a fundamental assumption in standard chain-of-thought (CoT) distillation: does the student need to imitate the teacher's reasoning to benefit from it?
In standard CoT distillation (STD-COT), the student is trained to reproduce both the teacher's reasoning trace and final answer. The authors argue that teacher reasoning contains two components: useful information and a particular way of verbalizing that information. They hypothesize that forcing the student to imitate the entire reasoning trace may be unnecessary and can hurt generalization.
EKLAV instead treats teacher reasoning as privileged information. The teacher's reasoning is provided during training, but its tokens receive no direct loss; the student is supervised only on the final answer. This allows the student to benefit from the information in the teacher's reasoning without being forced to reproduce the teacher's reasoning artifact.
The teacher reasoning is available only during training and is removed at deployment. At inference, the student receives the original input and generates its own reasoning before producing the final answer.
Importantly, EKLAV uses task-dependent placement rather than always putting the entire teacher CoT into the prompt. For pointwise passage reranking, the teacher reasoning is split approximately in half: part is placed in the prompt as a hint, while the remaining reasoning stays on the assistant side but is not supervised. For listwise table reranking, the entire teacher reasoning is placed in the prompt, and the student is supervised only on the final ranking output.
Critical strengths
- Interesting reframing of reasoning distillation. The paper challenges the common assumption that teacher reasoning must be explicitly imitated by the student. The distinction between reasoning as an imitation target and reasoning as privileged information is conceptually interesting and provides a simple alternative training paradigm. This is consequential because the approach can potentially retain useful information from teacher reasoning while avoiding the cost and potential overfitting associated with reproducing the teacher's particular reasoning style.
Critical weaknesses
-
The mechanism is not sufficiently isolated. EKLAV changes multiple factors simultaneously: the reasoning loss is removed, reasoning is moved into the prompt, lexical filtering is applied, sequence length changes, and the overall context layout differs from STD-COT. The MASK-ONLY ablation is useful, but it does not fully isolate these components. For example, it would be important to evaluate what happens when the same filtered reasoning hint is placed in the prompt while the remaining reasoning is still supervised. Similarly, a systematic study of different prompt/response splits (e.g., 25/75, 50/50, 75/25, and 100/0) would help clarify which design choice is responsible for the gains.
-
There is a substantial training–inference distribution shift. During training, the student predicts the answer conditioned on teacher-generated reasoning, whereas at inference it receives no teacher reasoning and must generate its own reasoning before producing the answer. This is a substantial reasoning-context shift. The fact that EKLAV performs well despite this mismatch is interesting, but the paper provides limited analysis of why the learned behavior transfers successfully from teacher-generated to student-generated reasoning.
-
The evaluation is limited in task diversity. The strongest experiments focus on passage reranking and table reranking, which are useful but relatively specific settings. Moreover, the authors report that exploratory mathematical-reasoning experiments show the opposite pattern. This makes it difficult to conclude that EKLAV is generally superior for reasoning distillation. The evidence more directly supports the narrower claim that EKLAV is effective in the evaluated reranking settings. Broader evaluation across substantially different reasoning tasks would strengthen the paper's generality.
-
The evaluated model scale is relatively small. The experiments use models ranging from 4B to 14B parameters. While this provides a reasonable range for studying the method, it leaves open whether the observed behavior persists at larger scales and in more realistic teacher–student configurations. For example, it would be valuable to see whether the same conclusions hold for substantially larger teachers and students, such as a 70B teacher distilling into a 7B or 14B student. The interaction between model scale, teacher quality, and the usefulness of privileged reasoning is therefore still unclear.
Questions for authors
Please address the above concerns (Weaknesses section).
Thank you again for the decision-relevant questions. Two new results add to our first reply (tables in Response 7).
Q3, value of teacher context over answer-only training. We completed the hint-size sweep. With a shorter filtered hint, EKLAV's lead over NAIVE-SFT on BRIGHT grows: 34.98 at 10% and 35.50 at 25%, against 33.39 for NAIVE-SFT (+1.59 and +2.11 points, compared with +0.85 at the paper's 50%; Tables 3 and 25). The two models share the training data and recipe and differ only in the teacher hint, so the gain comes from adding a short filtered prefix of the teacher's reasoning to answer-only training.
Generalization. We evaluated the Qwen3-8B passage checkpoints, without retraining, on seven held-out BEIR collections. EKLAV improves on the base model on all seven, and with a 25% hint it is ahead of STD-CoT on all seven (+3.62 points; Table 24), with the largest gains on finance QA and entity retrieval.
If these address your concerns, we would be grateful if you would consider revisiting your assessment.