acceptodds
Under review as a conference paper at ICLR 2027

EVIDENCE-ADAPTIVE DUAL-SPACE DISTILLATION FOR LANGUAGE MODELS

Abstract

Cross-tokenizer knowledge distillation requires both alignment between hetero- geneous output spaces and an appropriate allocation of supervision across them. We introduce Evidence-Adaptive Knowledge Distillation (EAKD), which uses detached student and teacher predictive cross-entropies to adapt the directional co- efficients of an existing dual-space objective. A learnable convex fusion preserves their common loss level, while a fixed absolute-gap term provides a conservative upper bound against local redistribution of the fusion coefficients. We formulate supervision allocation as a loss-based generalized-Bayesian decision problem: under specified endpoint actions, the unique control distribution and its posterior- mean action recover the implemented complementary directional weights exactly. The controller is recomputed on every mini-batch, preserves a fixed coefficient budget, and requires no additional language model. Our primary evaluation uses two LoRA-based TinyLLaMA-1.1B distillation settings with a fixed distillation temperature and dynamic weight allocation, improving average ROUGE-L over DSKD by 2.04 points for Mistral-7B → TinyLLaMA-1.1B and 2.08 points for LLaMA2-7B → TinyLLaMA-1.1B. The code will be released soon.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Cross-vocabulary knowledge distillation, Large Model, Adaptive distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q9cGOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence-Adaptive Knowledge Distillation (EAKD), an adaptive loss-weighting mechanism built on the existing DSKD framework. It combines student and teacher cross-entropies through a learnable convex fusion and an absolute-gap correction, then uses a sigmoid controller to adjust weights between three directional distillation losses. The paper provides a fusion bound clarification and a generalized-Bayesian interpretation of the weighting rule. Experimental results are reported in two primary TinyLLaMA-based distillation settings across 5 benchmarks.

Critical strengths
  1. EAKD introduces a lightweight adaptive weighting mechanism. It uses batch-level student and teacher cross-entropies to adaptively adjust the weights of distillation losses. It requires only two trainable scalar logits without introducing additional networks.
  2. EAKD consistently improves over DSKD in both primary distillation settings across five evaluation benchmarks.
Critical weaknesses
  1. The experiments do not make a strong case for the proposed adaptive weighting scheme. The paper fixes the total coefficient scale at 15.15, while unit-weight DSKD has a total of 3, as mentioned in Appendix A.1. This difference can also affect the contribution of supervised student loss L_{CE} and the distillation losses. Moreover, Table 4 illustrates that changing w_{\max} affects the final performance. So the role of adaptive weighting has not been satisfactorily established.

  2. To further expand on point 1, while Figure 1 provides some empirical motivation for adaptive weighting by showing variations in student CE and teacher-student CE it `does not establish the main application --- how these statistics relate to the relative effectiveness of different distillation losses. In Section 3.2, Eq. (5) combines student and teacher cross-entropies and their absolute gap into d. Eq. (11) further shows that larger d leads to lower weight on L_A and higher weight on L_C. However, different training states can produce the same d. For example, a state that both student and teacher incur high CE with a small gap potentially receives the same allocation as a state with a large CE gap. It remains unclear why these states should have the same shift in distillation losses.

Questions for authors
  1. Could the authors clarify the explicit coefficient configuration of the DSKD baseline in Table 1 and provide comparisons against scale-matched state weights, (5.05, 5.05, 5.05) configuration suggested in Appendix A.1? This would help to clarify the contribution of the proposed scheme.
  2. Could the authors provide a controlled analysis separating the effects of overall CE level and the CE gap on the learned directional weights? In particular, why should training batches with high CE for both models and batches with large CE gap favor the same allocation, and is this assumption empirically supported? This would help clarify the motivation of the adaptive weighting scheme.
Reviewer_xZH5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Cross-tokenizer knowledge distillation (KD) must align heterogeneous output spaces and allocate supervision across aligned channels. This paper proposes Evidence-Adaptive Knowledge Distillation (EAKD) on top of dual-space KD (DSKD). It uses detached student/teacher predictive cross-entropies as batch-level evidence, fuses them with a learnable convex fusion plus a fixed absolute-gap term, and derives directional loss weights from a generalized-Bayesian decision rule under a fixed coefficient budget. The controller is recomputed each mini-batch without an extra language model. On LoRA distillation of a TinyLLaMA-1.1B student from Mistral-7B and LLaMA2-7B teachers, EAKD improves average ROUGE-L over DSKD by about +2.0 points (Table 1). Gains hold under several student-space divergences (Table 2). Code is promised but not released.

Critical strengths
  1. Clear problem split. Separating “where to align” (dual-space projection) from “how much to weight each channel” is a sensible complement to DSKD (Sec. 1–2).

  2. Lightweight controller. Batch-level CE evidence and closed-form weights avoid an auxiliary policy network or extra LM forward pass (Sec. 3; Fig. 2).

  3. Consistent gains in the reported protocol. Improvements appear on all five eval sets in both primary settings (Table 1) and remain positive across KL-family discrepancies (Table 2); Fig. 3 shows closer teacher–student relational structure.

Critical weaknesses
  1. Incremental relative to prior adaptive KD (Sec. 2.2). Adjusting distillation weights from difficulty or training state is already studied in Dynamic KD, RW-KD, and related work. In practice EAKD mainly reallocates the existing DSKD loss terms; the paper could state more clearly what goes beyond that line.

  2. Theory is over-packaged for the controller (Sec. 3.3; Sec. 4 opening; Appendix A.1). The generalized-Bayesian derivation is heavier than the rule that is actually run. The authors already note that conditional decision optimality is not meant as a prediction of test performance. Tonning down the theory narrative and adding a short main-text ablation of fusion logits vs. absolute-gap vs. dynamic weights would help.

  3. Possible loss-scale confound vs. DSKD (Appendix A, Eqs. 19–20; Table 1). With (w_min, w_max)=(0.1,10), EAKD’s directional coefficients sum to 3w_mid=15.15, whereas unit-weight DSKD sums to 3. The authors themselves say a scale-matched static baseline would use three coefficients of 5.05. If Table 1 compares against unit weights, some of the +2 ROUGE-L may come from a larger KD loss scale rather than adaptive allocation.

  4. Narrow evaluation and missing code (Sec. 4.1; Table 1; Appendix B.6; Abstract). The primary experiments use only two teacher models with the same TinyLLaMA-1.1B student, which is insufficient to establish generalizability across different student architectures. Although the appendix includes a Qwen1.5→GPT2 experiment, it additionally enables dynamic temperature adaptation and uses a different fine-tuning protocol, making it difficult to isolate the generalizability of the proposed weight-only controller. The evaluation also relies mainly on ROUGE-L, and the code is listed as “will be released soon” rather than being available for review.

  5. Insufficient justification for the directional weight allocation (Sec. 3.3; Eqs. 10–11). The proposed controller always decreases the T→S CE weight and increases the S→T KD weight as the fused evidence increases. However, larger evidence can arise from different causes, such as higher student CE, higher teacher CE, a larger teacher–student CE gap, or both models encountering difficult batches. It is unclear why these different conditions should always favor the same allocation direction. Furthermore, the rationale for keeping the T→S KD weight fixed while adjusting the other two loss coefficients is not sufficiently explained.

Questions for authors

Q1. What coefficients does Table 1’s DSKD baseline use? Please report a scale-matched static DSKD (weights 5.05 each) or EAKD with weights normalized to sum 3.

Q2. How sensitive are the gains to the evidence definition (token-level CE vs. batch mean; student-only vs. dual-CE)? A short ablation of fusion logits / absolute-gap / dynamic weights in the main text would also help.

Q3. Relative to Dynamic KD, RW-KD, and fixed-weight DSKD, what should readers take as the concrete advance of EAKD? Does the method still help when dual-space projectors are weak or frozen?

Q4. Why keep temperature fixed in the primary protocol (Sec. 4.1) but enable it for Qwen→GPT2 (Appendix B.6)? Given this difference, how can the generalizability of the weight-only controller across different student architectures be established? Please release code and add at least one non-ROUGE metric.

Q5. What motivates the chosen allocation direction and the fixed T→S KD coefficient? Could the authors compare the proposed policy with reversed or alternative allocation strategies to validate these design choices?

Reviewer_BxF7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose evidence-adaptive knowledge distillation, which tackles the problem of knowledge distillation where the teacher and student have different tokenizer/vocabulary. They build on the existing dual space knowledge distillation, where distillation happens in both teacher's and student's vocab space (after projection). The authors' contribution is a weighting factor on which side to emphasize, and they have a single parameter that nudges toward the side with a smaller loss.

Critical strengths
  1. The authors propose an interesting idea that helps to reweigh the losses in dual space KD.
  2. The authors show consistent improvement over DSKD.
Critical weaknesses
  1. No detailed analyses in the main paper. Although the main contribution of the paper is the weights for the objective, there is no analyses on that (at least in the main paper).
  2. Table 1 shows the student very consistently outperform the teacher, when the vocab is the same, which shows perhaps the experimental setup may need improvements (e.g., better evaluation)
Questions for authors

The method introduces a new hyper-parameter γ while removing some other hyper-parameters. How hard is it to tune γ.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.