- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the issue of knowledge distillation in large language models, specifically aiming to bridge the gap where—despite achieving high output consistency—point-wise objectives still leave significant discrepancies between teacher and student models within the input neighborhood. The proposed SaKD method integrates two components: a noise-injected Kullback–Leibler (KL) divergence term and a re-weighting mechanism based on saliency divergence. Experimental results demonstrate performance improvements over existing baselines.
Critical strengths
1 The paper is well-written, with a clear description of the method that is easy to follow.
2 It provides theoretical support for the problem identified: that point-wise objectives, while ensuring high output consistency, can still result in substantial teacher-student gaps within the input neighborhood.
Critical weaknesses
1 The experimental evaluation is somewhat limited; the authors use only three instruction-following benchmarks and report performance gains of around 1 point across three student models, without providing results for a broader range of tasks.
2 While the theoretical analysis explains the relationship between noisy KL divergence and local input sensitivity alignment, it does not prove that the student relies on the same evidence as the teacher or avoids superficial shortcuts; the evidence is insufficient to substantiate the claim that answers are derived through teacher-like reasoning.
Questions for authors
1 The main experiments cover only three instruction-following benchmarks. This selection seems limited, omitting benchmarks commonly used in other distillation studies, such as Self-Instruct and VicunaEval. Furthermore, it remains unclear whether the method yields consistent performance improvements on other types of tasks, such as mathematics or coding.
2 The authors claim the method addresses issues regarding "local sensitivity" and enables reasoning "through teacher-like reasoning rather than surface shortcuts." However, there is a lack of experimental evidence to support this—for instance, results regarding performance on inputs that have been paraphrased or perturbed are missing.