acceptodds
Under review as a conference paper at ICLR 2027

Teaching Students Where to Look: Saliency-Guided Knowledge Distillation for LLMs

Abstract

Knowledge distillation (KD) is the standard approach for compressing large language models (LLMs) into compact deployable students by training the student to match the teacher's output distribution. While many improvements have been proposed, such as alternative divergence choices, on-policy generation, and difficulty-adaptive sample reweighting, existing methods deliver the distillation signal pointwise in output space, providing no constraint on how the student behaves in the neighborhood of training inputs. We empirically show that this pointwise objective leaves large teacher–student gaps in input neighborhoods even when output agreement is exact, and propose SaKD (Saliency-Guided Knowledge Distillation), which closes this gap by lifting pointwise output matching to a saliency-aware first-order matching objective. Concretely, SaKD combines a noise-injected Kullback–Leibler (KL) divergence term that, via a noise–Jacobian identity for autoregressive token-level KL, provides an implicit teacher–student Jacobian-matching signal along coupled perturbation directions, with a saliency-divergence reweighting that focuses the loss on samples with larger teacher–student input-saliency disagreement. Extensive experiments show that SaKD consistently outperforms seven strong KD baselines across Qwen3 and LLaMA student scales, with students reaching correct answers through teacher-like reasoning rather than surface shortcuts.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Knowledge distillation, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LhfpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the issue of knowledge distillation in large language models, specifically aiming to bridge the gap where—despite achieving high output consistency—point-wise objectives still leave significant discrepancies between teacher and student models within the input neighborhood. The proposed SaKD method integrates two components: a noise-injected Kullback–Leibler (KL) divergence term and a re-weighting mechanism based on saliency divergence. Experimental results demonstrate performance improvements over existing baselines.

Critical strengths

1 The paper is well-written, with a clear description of the method that is easy to follow.

2 It provides theoretical support for the problem identified: that point-wise objectives, while ensuring high output consistency, can still result in substantial teacher-student gaps within the input neighborhood.

Critical weaknesses

1 The experimental evaluation is somewhat limited; the authors use only three instruction-following benchmarks and report performance gains of around 1 point across three student models, without providing results for a broader range of tasks.

2 While the theoretical analysis explains the relationship between noisy KL divergence and local input sensitivity alignment, it does not prove that the student relies on the same evidence as the teacher or avoids superficial shortcuts; the evidence is insufficient to substantiate the claim that answers are derived through teacher-like reasoning.

Questions for authors

1 The main experiments cover only three instruction-following benchmarks. This selection seems limited, omitting benchmarks commonly used in other distillation studies, such as Self-Instruct and VicunaEval. Furthermore, it remains unclear whether the method yields consistent performance improvements on other types of tasks, such as mathematics or coding.

2 The authors claim the method addresses issues regarding "local sensitivity" and enables reasoning "through teacher-like reasoning rather than surface shortcuts." However, there is a lack of experimental evidence to support this—for instance, results regarding performance on inputs that have been paraphrased or perturbed are missing.

Reviewer_ArSiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes SaKD, a saliency-guided KD method that adds a noise-injected KL term and saliency-divergence sample reweighting, motivated by empirical observation and theoretical analysis. experiments on small-size LLM KD show its performance over traditional KD methods.

Critical strengths
  • The research gap is clear and motivated by empirical observations, and the study framework is logical and easy to follow;
  • the method is sound and theoretically grounded.
Critical weaknesses
  • Although all theorem statements is correct, the theoretical support for the practical algorithm is limited: 1) in Theorem 3, the residual term is signed and can dominate the Fisher score-gap term away from exact output agreement, so the noisy KL is not a purely positive Jacobian-matching penalty as the paper's interpretation suggests; 2) the saliency-guided noise schedule has a blind spot, that is, equal saliency norms can hide opposite gradient directions, and the zero-gap rule then sets the noise scale to zero, disabling the Jacobian probe exactly where a mismatch exists.
  • the empirical evaluation is limited: the method is only tested on small-scale LLM KD with a single and relatively old dataset (Dolly-15K) on a single task (instruction following), so its performance on complex tasks such as math and code is unclear; besides, recent theory-grounded and advanced KD methods, such as ABKD [1], TAID [2] and DistiLLM-2 [3], are overlooked in the baseline selection, and a hyperparameter sensitivity analysis (e.g., λ\lambda, σ0\sigma_0, τw\tau_w, NN) is lack.

[1] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence, ICML 2025 Spotlight.

[2] TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models, ICLR 2025.

[3] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs, ICML2025 Spotlight.

Questions for authors
  • How does SaKD perform against recent advanced KD methods when distilling larger-scale LLMs on complex tasks (e.g., math and code)?
  • Can the authors clarify the theoretical guarantee in the practical regime: 1. how is the implicit Jacobian-matching interpretation justified when Theorem 3's signed residual dominates the score-gap term away from exact output agreement; 2. would a positive noise floor avoid the blind spot where zero saliency gap coexists with a nonzero Jacobian gap?
Reviewer_r8wHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that standard LLM knowledge distillation matches teacher and student outputs only at the training inputs, leaving the student's behavior around those inputs unconstrained. The proposed method, SaKD, adds two components to the standard KD:

  1. Noise-injected KL. One Gaussian draw is mapped into both models' embedding spaces through fixed random projections P_T and P_S.
  2. Per-sample reweighting. Each sample is weighted according to Jensen–Shannon divergence (JSD) between the normalized teacher and student saliency distributions.

Experiments distil Qwen3-8B → Qwen3-1.7B/0.6B and LLaMA-3.1-8B → LLaMA-3.2-1B on three benchmarks to prove the effectiveness of the proposed method.

Critical strengths
  1. The authors propose a principled, tractable route to first-order matching in LLM distillation. Moving from matching outputs to matching local input sensitivity is well motivated.
  2. The proposed method is both analyzed theoretically and demonstrated empirically, which shows a real effect.
Critical weaknesses
  1. The claimed motivation is not validated in the experiments of the proposed method. It also never re-runs the Figure 1 diagnostic (the perturbed-KL tail) on SaKD students.
  2. The evaluation setup can be further improved. a. Previous work include 5 evaluation benchmarks but in this paper, only three of them are evaluated (dropping SelfInst and VicunaEval).

b. Some important baselines are missing. For example, the author only compare with DistiLLM, but does not compare with DistiLLM-2. AKL [1] is cited but not compared. Other families of KD methods like FDD [2] are not compared.

  1. Some design choices are not fully validated

a. The authors claims The saliency vector is nonzero only at prompt positions. But mathematically, the gradient is non-zero. It is unclear why they only consider the prompt position for saliency.

b. According to Algorithm 1, SaKD periodically uses the proposed loss term because it will introduce extra costs. However, it is unknown how the hyperparameters would affect performance.

[1] Wu et.al Rethinking kullback-leibler divergence in knowledge distillation for large language models.

[2] Gong et.al Beyond logits: Aligning feature dynamics for effective knowledge distillation

Questions for authors
  1. How do results and cost change for N (used in step mod N=0)? Is there any tradeoff between efficiency and performance?

  2. Why is saliency vector is nonzero only at prompt positions?

  3. For position-adaptive noise allocation, how are the cap ratio 5 in Eq. 12 and K=128 for Gaussian decided?

  4. According to Weakness 2, could you add more results regarding the evaluation datasets and baselines?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.