- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Evidence-Adaptive Knowledge Distillation (EAKD), an adaptive loss-weighting mechanism built on the existing DSKD framework. It combines student and teacher cross-entropies through a learnable convex fusion and an absolute-gap correction, then uses a sigmoid controller to adjust weights between three directional distillation losses. The paper provides a fusion bound clarification and a generalized-Bayesian interpretation of the weighting rule. Experimental results are reported in two primary TinyLLaMA-based distillation settings across 5 benchmarks.
Critical strengths
- EAKD introduces a lightweight adaptive weighting mechanism. It uses batch-level student and teacher cross-entropies to adaptively adjust the weights of distillation losses. It requires only two trainable scalar logits without introducing additional networks.
- EAKD consistently improves over DSKD in both primary distillation settings across five evaluation benchmarks.
Critical weaknesses
-
The experiments do not make a strong case for the proposed adaptive weighting scheme. The paper fixes the total coefficient scale at 15.15, while unit-weight DSKD has a total of 3, as mentioned in Appendix A.1. This difference can also affect the contribution of supervised student loss L_{CE} and the distillation losses. Moreover, Table 4 illustrates that changing w_{\max} affects the final performance. So the role of adaptive weighting has not been satisfactorily established.
-
To further expand on point 1, while Figure 1 provides some empirical motivation for adaptive weighting by showing variations in student CE and teacher-student CE it `does not establish the main application --- how these statistics relate to the relative effectiveness of different distillation losses. In Section 3.2, Eq. (5) combines student and teacher cross-entropies and their absolute gap into d. Eq. (11) further shows that larger d leads to lower weight on L_A and higher weight on L_C. However, different training states can produce the same d. For example, a state that both student and teacher incur high CE with a small gap potentially receives the same allocation as a state with a large CE gap. It remains unclear why these states should have the same shift in distillation losses.
Questions for authors
- Could the authors clarify the explicit coefficient configuration of the DSKD baseline in Table 1 and provide comparisons against scale-matched state weights, (5.05, 5.05, 5.05) configuration suggested in Appendix A.1? This would help to clarify the contribution of the proposed scheme.
- Could the authors provide a controlled analysis separating the effects of overall CE level and the CE gap on the learned directional weights? In particular, why should training batches with high CE for both models and batches with large CE gap favor the same allocation, and is this assumption empirically supported? This would help clarify the motivation of the adaptive weighting scheme.