acceptodds
Under review as a conference paper at ICLR 2027

Learning Beyond Full Imitation: Task-Preserving Knowledge Distillation

Abstract

Knowledge distillation transfers knowledge by encouraging a student to match a teacher's predicted class probabilities. These probabilities express not only confidence in the correct class, but also relations among incorrect alternatives. Yet closer imitation does not necessarily yield a better student. Our key observation is that a student may already distinguish the correct class more sharply than its teacher, so further imitation can require giving back discrimination it has acquired. Our main result is an exact separation between full imitation and conditional learning. When the correct class's score advantage over each alternative must be preserved, full teacher-to-student KL minimization is blocked exactly when the student assigns no more probability than the teacher to every incorrect class. Crucially, the teacher's relative probabilities among incorrect classes remain fully learnable. We characterize the exact price of this transfer: a minimum increase in correct-class log-odds that compensates for the largest conditional-probability mismatch. Label fitting and conditional matching can therefore be completed even as full teacher KL diverges. This separation motivates task-preserving knowledge distillation (TPKD), which keeps the label gradient intact and minimally corrects the conditional gradient so that its output update preserves the label step's gains against every incorrect alternative. The corrected conditional direction retains more than half of the original first-order conditional descent at the same step size, with a tight bound. For a fixed positive conditional target and sufficiently small constant output steps, label and conditional errors vanish together. Experiments trace this learning from exact head updates to ordinary network training. TPKD reaches 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, improving over standard distillation by 0.47 and 0.35 percentage points across three seeds.

Primary area:
learning theory
Keywords:
knowledge distillation, task preservation, conditional knowledge transfer, constrained optimization, learning dynamics

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_6z1cOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the potential conflict between full teacher imitation and the student's acquired task discrimination in knowledge distillation. The authors theoretically characterize when full imitation becomes blocked and show that conditional knowledge among non-target classes can still be transferred while preserving classification margins. Based on these findings, they propose Task-Preserving Knowledge Distillation (TPKD), which retains the label gradient and projects the conditional distillation gradient onto a margin-preserving cone. Theoretical analysis and experiments on image and text classification tasks demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The paper studies an interesting and well-motivated problem. It highlights the potential conflict between full teacher imitation and the student's acquired task discrimination, providing a meaningful perspective on what knowledge should be transferred during distillation.

  2. The theoretical analysis is rigorous and insightful. The paper establishes a necessary-and-sufficient condition for full-imitation blocking and derives the minimum confidence compensation required for task-preserving conditional knowledge transfer. These results provide a deeper understanding of the relationship between full imitation and conditional learning.

  3. The proposed TPKD framework is simple and theoretically well-motivated. By preserving the label gradient and projecting the conditional distillation gradient onto a margin-preserving cone, the method allows the student to learn additional teacher knowledge without sacrificing the discrimination gains of the label update. The authors further provide theoretical guarantees on retained conditional learning and joint convergence, strengthening the justification for the proposed method.

Critical weaknesses
  1. The experimental evaluation is relatively limited. The paper reports full-training results on only two datasets, with one teacher–student pair in each domain. Since the proposed method is motivated by differences in teacher and student discrimination, experiments with varying teacher quality and student capacity would help further validate its generality and effectiveness.

  2. The advantage over simpler conditional distillation methods remains unclear. Although the paper includes a variant without projection, it lacks comparisons with a properly tuned CE-plus-conditional-KL baseline. Moreover, several existing methods, particularly DKD, perform substantially worse than CE on CIFAR-100. Further analysis of hyperparameter sensitivity and baseline configurations would help establish whether the reported improvements are consistent and reliable.

Questions for authors

Please see the Weaknesses

Reviewer_sFQNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks what a student can still learn from its teacher when full imitation conflicts with preserving correct-class margins. It characterizes exactly when full teacher-to-student KL minimization becomes blocked and shows that the teacher’s conditional distribution over incorrect classes remains learnable, with a precise confidence-compensation cost. Based on this analysis, TPKD preserves the label gradient and projects the conditional gradient onto a margin-preserving cone. The paper establishes retention and convergence guarantees for output updates, alongside conditional results for parameter updates. Experiments examine the proposed mechanism and report 88.05% accuracy on CIFAR-100 and 93.81% on CLINC150, modestly improving over standard KD.

Critical strengths

I appreciate the progression from exact head updates to native optimizer steps and then successive updates. In particular, the compensation-only control helps distinguish learning non-target relationships from merely increasing correct-class confidence. The paired comparisons restore the same model and optimizer states, making the conditional-learning gains easier to interpret. Reporting three-seed results and the closely matched unprojected ablation is also helpful for assessing what the method actually adds.

Critical weaknesses

Table 5 suggests that much of the accuracy benefit survives without projection: removing it costs only 0.22 points on CIFAR-100 and slightly improves CLINC150. Meanwhile, Appendix E.1 uses different temperatures and loss weights across methods, without a sensitivity study; DKD falls well below CE on CIFAR-100. These choices are documented, but I would like to know whether the gains persist with appropriately tuned baselines, especially DKD, which already separates target and non-target supervision. The current results leave the projection’s practical contribution somewhat unclear.

Questions for authors
  1. Under the actual training optimizers, how frequently and by how much does TPKD violate the CE-reference margin ordering? Could you report this for paired updates from identical states?

  2. How sensitive are the KD and DKD results to temperature and distillation weight under your training protocol? Does TPKD retain its advantage after a small, comparable tuning budget?

Reviewer_TmugOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

During knowledge distillation, a student may already be more discriminative than its teacher on some samples. This paper asks what the student can still learn from the teacher in this case. It proposes Task-Preserving Knowledge Distillation (TPKD), which keeps the margin between the correct class and each incorrect class. Instead of fully mimicking the teacher, the student learns the teacher's relative probabilities among the incorrect classes while keeping all these margins. The paper provides a solid theoretical analysis of the proposed method. It also gives a concrete implementation that projects the update direction onto the directions that keep all margins, so the method is easy to implement. Experiments on CIFAR-100 and CLINC150 show that TPKD improves over standard KD by 0.47 and 0.35 percentage points.

Critical strengths
  1. This paper addresses a common and meaningful problem in knowledge distillation: when further imitation of the teacher limits the student's ability, what can the student still learn from the teacher? The experiments also show that this situation is common in practice.
  2. The theoretical results are solid. Theorem 1 gives a necessary and sufficient condition under which full imitation is blocked, and Theorem 2 derives the attainable minimum confidence cost for learning the teacher's conditional distribution. Theorem 3 further provides tight bounds on both norms, showing that the projected direction retains more than half of the original conditional descent.
  3. The projection has a closed-form solution with O(K log K) complexity, and the update is applied through a simple surrogate loss with ordinary backpropagation, which makes the method easy to implement.
  4. The ablation study is solid and honest. It examines different components of the proposed method, including removing the CE gradient, replacing conditional learning with protected full imitation, and removing the projection.
Critical weaknesses
  1. One major concern is the source of the improvement. TPKD avoids degrading the student in two ways. For the correct class, it keeps the CE gradient and only learns the teacher's conditional distribution among the incorrect classes, so a student that is more confident than the teacher is not pulled back. For the incorrect classes, the projection ensures that the margin against each incorrect class does not decrease. However, the ablation study shows that the projection, which corresponds to the second part, contributes little to the improvement. Thus, the improvement mainly comes from learning only the conditional distribution without pulling back the student's correct-class confidence. The relative strengths of TPKD compared with methods using only the first way need to be further explained.
  2. The accuracy of the teacher models is not reported, and it could be listed together with the student performance to provide full information. It would also be helpful to evaluate TPKD under two conditions: (1) the teacher performs much better than the student, and (2) the teacher performs only slightly better than the student, since the blocking condition depends on how confident the teacher is relative to the student.
  3. Whether preserving the per-class margins is useful, and whether these margins are actually kept during full training, can be further elaborated.
  4. The experiments only use one non-standard teacher–student setting for each domain. Results on standard benchmarks (e.g., CIFAR-100 from scratch and ImageNet) with standard baselines would make the comparison more convincing.
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.