- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a defense against logit-based knowledge distillation by applying a learnable transformation to teacher logits. Motivated by conditional mutual information (CMI), the method combines a utility-preserving cross-entropy objective with a gradient-direction objective intended to disrupt student learning.
Critical strengths
- The parameter-efficient transformation keeps the teacher weights frozen and provides a simple mechanism for modifying its outputs.
- Experiments cover multiple student sizes and distillation objectives, with ablations and comparisons against simple logit-processing defenses.
Critical weaknesses
- Outdated and unrepresentative evaluation. Experiments rely on Qwen2.5 and Llama 3 models and the older GSM8K, MMLU, and MATH-500 benchmarks. In particular, the standard Qwen2.5 teacher does not represent contemporary models specifically trained for extended reasoning. With open reasoning-capable models such as Qwen3.5-27B available, the absence of modern reasoning teachers and more challenging reasoning benchmarks is a major limitation. The results do not establish whether the defense remains effective when distilling stronger teachers and richer reasoning traces.
- Incomplete threat model and missing adaptive attacks. An attacker could discard the modified logits and instead distill the teacher’s generated text, which the method explicitly aims to preserve in quality. This straightforward alternative is not evaluated, and the SFT-only baseline does not test distillation from the defended teacher’s generated responses. Furthermore, the practical relevance of assuming access to full-vocabulary logits is insufficiently justified. Consequently, the experiments support a narrower claim about disrupting selected logit-based objectives, rather than broadly preventing knowledge extraction.
- The information-theoretic justification does not establish information removal. Theorem 2 guarantees only that deterministic post-processing cannot increase CMI. The implemented transformation is not constrained to be non-invertible; whenever it is invertible, Moreover, changing student gradient directions does not establish a reduction in CMI. The observed degradation could therefore reflect interference with particular optimization objectives without demonstrating the claimed removal of distillation-relevant information.
Questions for authors
- Does the defense remain effective against sequence-level distillation using responses generated by the defended teacher, and against attackers that retune their distillation procedure for the modified outputs?
- Can the authors evaluate modern reasoning teachers, such as Qwen3.5-27B, on more challenging reasoning benchmarks with appropriate generation budgets, reporting both student degradation and preservation of teacher utility?