- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an adaptive loss fusion strategy for knowledge distillation (KD) which is intrinsically composed of two types of loss functions derived from ground-truth label and teacher-network supervision. The weight for those losses is regarded as a target parameter to be controlled and the authors introduce PID-control mechanism to tune it adaptively based on training errors. According to PID, the prediction errors of the student model are harnessed in the forms of proportional, integral and derivatives, all of which are integrated into a PID-score to be mapped into the loss weight parameter. In the experiments on KD tasks in image classification/detection, the proposed approach renders performance improvement to various KD methods.
Critical strengths
- PID control mechanism is interestingly applied to tuning loss weights in a knowledge distillation framework. As the weight parameter is tuned by hands in most cases, the approach would reduce costly practitioners' effort to improve KD performance.
- Efficacy of the proposed method is validated in a comprehensive manner across broad types of KD approaches.
Critical weaknesses
-
Rationale to regard loss weight tuning as PID controlling is unclear and less articulated in the paper. From engineering perspectives, PID is applied to align an output with a target by mitigating the issues of bias, overshoot and instability during an approach of the output signal toward the target. On the other hand, the KD training of DNN models is less relevant to those issues that PID focuses on. Namely, alignment of student output (logits) to the target (ground-truth label) in (6) is not solely the objective of the KD learning; the KD objective is rather to slightly deviate the student output from the target to attain favorable characteristics (a.k.a dark knowledge) embedded in the teacher model. Or more generally speaking, the ultimate goal of KD learning is to enhance generalization performance of the student network, which is less relevant to the goal of PID control. Therefore, the approach simply based on PID seem to lack theoretical soundness; specifically, it is unclear how the P/I/D terms are connected to the weight parameter for improvement of KD learning.
-
The presented approach resorts to tons of hyper-parameters to be heuristically tuned. Even the key mapping function (12) is implemented by means of loogup table which is not clearly shown at all in this paper. While, in control engineering fields, PID gains (hyper-parameters) are designed in a principled way such as via transfer functions, the presented approach is formulated in a ad-hoc manner.
- In the method, gains are tuned during the training in an ad-hoc way, which further deviates the method from principled PID concept. In addition, the descriptions in lines 328-338 are quite confusing. For example, the statement of "poorer mastery correspond to larger " (line 335) seems to be wrong since "larger indicates poorer mastery" (line 329) and is negatively correlated with in (13).
-
The error function (6) is just a linear approximation (lower-bound) for the cross-entropy loss and they are one-to-one corresponded (on ). However, as shown in Table 11, there is considerable performance gap between those criteria; they should be close by properly tuning the mapping function . In addition, from the KD perspectives, it is natural to regard the teacher logits as target signals in PID framework. Therefore, the error (6) missing the teacher signals is less convincing; one can guess it is employed from practical performance viewpoint in Table 11, being away from PID concept.
Questions for authors
- Please provide clear response to the above-mentioned concerns, especially regarding theoretical connection (rationale) to PID mechanism.
13 comments
Thank you for the careful and constructive review (W1 & Q1-Part 2/2)
W1 (continued): "specifically, it is unclear how the P/I/D terms are connected to the weight parameter for improvement of KD learning."
Response to W1 and Q1 (part 2/2):
3. The weights enter the joint training objective, modulating the relative contributions of task supervision and teacher supervision to the student update. Following Eq. (13) in the revised manuscript, the final training objective is:
,
where denotes the total number of samples, and the two types of loss respectively preserve the internal coefficients, temperature factors, and sub-loss structures of the base method. Therefore, altering changes the relative contributions of the gradients from the two losses to the student parameter updates, causing the supervision intensity to dynamically vary with the sample's learning state. We agree that the ultimate goal of KD is to improve the student's generalization performance, and that the non-target class probabilities in the teacher's soft distribution and their relative relationships constitute important supervision information. Hence, PIDScore does not discard the original distillation objective. The actual update is still determined by the joint loss of and .
4. The above process forms a continuous feedback loop from the student state to the supervision weight allocation, and back to the student state. First, the objective deviation is generated based on the student's prediction. The P, I and D components then form the sample score based on the current error, recent history, and variation trend. This score is mapped to obtain the supervision weight , which subsequently influences the student update via the joint loss. The updated student produces new predictions, continuously refreshing the historical state of the same sample for use in subsequent scoring and weight computation. This forms a closed loop: "Student State→P/I/D Scores→Supervision Weights→Joint Loss and Student Update→New Student State".
5. Branch contribution analysis and ablation experiments demonstrate that the P, I and D branches all participate in generating the supervision weight , and that jointly utilizing the information from all three branches yields performance benefits. Tables 16 and 17 in Appendix G.2 vary the initial gain parameters and the adaptive adjustment step size, respectively, reporting the accuracies across seven teacher-student model pairs under three random seeds, as well as the relative contributions of P, I and D to the fusion weights. On ResNet56→ResNet20, with the default initialization and , the mean contributions of the three components are 59.68%, 21.84%, and 18.48%, respectively, with I and D combined accounting for 40.32%. This demonstrates that the current state, historical performance, and variation trend all make observable contributions to the fusion weights.
Table 5 in the main text further compares P-only, I-only, D-only and the complete PID. Across the six method and architecture settings corresponding to KD, NRKD, CAT-KD, and MLKD, the complete PID achieves the highest accuracy. Lines 509-517 of the revised manuscript have clarified that the single-branch experiments disable the Score-Guided Adaptation (SGA, see Section 4.2 of the main text), fix the gain parameter of the enabled branch, and keep the gain parameters of the remaining branches at 0 throughout the training. For instance, when the I branch is disabled, its gain parameter is set to 0 and maintained at 0 throughout the entire training process. In the ResNet56→ResNet20 KD setting, P-only, I-only, D-only and the complete PID achieve 70.66%, 71.31%, 71.59%, 71.74% and 72.15%, respectively. The complete PID improves by 0.84 percentage points compared to P-only and by 0.41 percentage points compared to the strongest single component, D, providing performance evidence for the joint utilization of the three types of feedback information.
Thank you for the careful and constructive review (W2-Part 1/3)
W2: "The presented approach resorts to ...., the presented approach is formulated in ... correlated with in (13)."
Response to W2 (part 1/3): We thank the reviewer for pointing out the need for further clarification regarding the lookup table construction, its design rationale and the gain parameter adjustment direction. In Section 4.2 of the revised manuscript, we have supplemented the construction of the lookup table and the correspondence between indices and table values. We present the final training objective function in Eq. (13) and specify the score-guided PID gain parameter adjustment rules in Eqs. (14)–(15). Specifically, Eq. (14) has been corrected to reflect the proper correspondence among the learning levels, adjustment multipliers, and proportional gain parameters. Appendix I further details the complete feedback process linking student state observation, supervision weights, and student updates. Below, we address the three issues sequentially: look-up table construction, method design and empirical basis, and the gain parameter adjustment direction.
1. The lookup table is pre-constructed following a unified rule. As described in Section 4.2 of the main text, for each sample score , we floor and normalized it as follows: and . Here, serves as the index for the lookup table , and is the value corresponding to the index . Specifically, represents the teacher supervision weight for sample , i.e., . The tuple is then added as an element to the lookup table . Therefore, the lookup table is defined as:
,
where is a trade-off coefficient, and is a method-specific constant. For example, with for KD and for DKD. All table entries are generated via a single linear mapping and require no tuning.
During training, the sample score is linearly scaled to and then floored to obtain the lookup table index . Subsequently, using as the index, we retrieve the corresponding table value from the lookup table , hereby obtaining the distillation supervision weight coefficient for this sample:
.
Thus, the task supervision weight coefficient is . Since , both coefficients lie in , and their sum is . Consequently, a higher corresponds to a larger distillation supervision coefficient, while a lower corresponds to a larger task supervision coefficient.
2. PIDScore possesses a complete operational logic spanning state observation, temporal scoring, supervision allocation, and subsequent updates; its design is validated through cross-framework experiments, ablations, and parameter analysis. Our design rationale is to dynamically adjust the supervision weight of each sample based on the student model's mastery degree of that sample, thereby adapting to the student's requirements under different learning states. To this end, PIDScore defines a unified state observation based on the student's predicted probability of the ground-truth class. It utilizes the P, I, and D components to describe the current error, recent history, and variation trend, respectively, and subsequently maps the sample score to the weights for task supervision and teacher supervision. The two types of weighted losses jointly update the student, and the updated predictions are continually utilized for subsequent scoring. Following Eqs. (14)–(15) in Section 4.2, the score is also employed to determine the learning level, adjusting the gain parameters of the three branches according to predefined rules. These steps seamlessly interconnect, ensuring that the scoring, loss weighting, and gain parameter adjustment all serve the same supervision allocation objective.
This operational logic also dictates the approach for integration into different distillation frameworks. The sample score exclusively utilizes student predictions, ground-truth labels, and the historical state of the same sample, independent of specific teacher features or relation representations. Upon integration, the original loss coefficients, temperature factors, and sub-loss composition of the base method are preserved, after which the unified scoring and mapping rules modulate the relative contributions of task supervision and distillation supervision. Therefore, the same feedback module can be readily combined with distillation methods based on logits, features, relations, and multiple teachers.
Thank you for the careful and constructive review (W1 & Q1-Part 1/2)
Response to W1 and Q1 (part 1/2): We thank the reviewers for pointing out this issue. In Appendix I, "PID Feedback in Knowledge Distillation" (lines 1256–1268) of the revised manuscript, we have supplemented this connection, explicitly distinguishing the objective deviation used to observe the student's learning state from the joint loss used to actually optimize the student. We also explain how P/I/D affect the student update and form continuous feedback via supervision weights, and how teacher knowledge participates in the optimization through the original distillation objective. Meanwhile, in Section 3 and Eq. (6) of Section 4.1, we distinguish the student's raw logits from predicted probabilities and clarify the definition of the objective deviation. In Section 4.2, we supplement the mapping from sample scores to fusion weights and specify the final training objective function.
Below, we sequentially explain how the three components form the sample score, how the sample score is converted into supervision weights, how the supervision weights influence student learning, then explain the resulting feedback mechanism, and finally introduce the experimental contributions of each component.1. The P, I and D components describe the current error, recent history and error variation, respectively, and their sum forms the sample score. According to Eq. (6) in Section 4.1 of the revised manuscript, the objective deviation is defined as the difference between the ground-truth class label and the student's predicted probability for that class. Therefore, this signal reflects the student's current mastery state of the ground-truth class and serves as the common input for the three feedback branches (P, I, D).
Following Eqs. (8)–(11) in the main text, the P component combines the current error with the proportional gain parameter, reflecting the student's immediate performance on the current sample. The I component aggregates the recent error history of the same sample, supplementing the temporal information of the student's sustained performance on this sample. The D component employs a second-order difference to describe how adjacent error increments change, supplementing the evolutionary information of the student's learning trend on this sample. The three components are individually modulated by their corresponding gain parameters and then summed according to Eq. (7):
wn(t)=wnp(t)+wni(t)+wnd(t).
Therefore, the comprehensive score simultaneously incorporates the current state, historical performance, and variation trend. Even if two samples share the same current confidence for the ground-truth class, they may still receive different scores as long as their recent history or variation trends differ.
2. The comprehensive score is scaled and mapped via a look-up table to obtain the weights for task supervision and teacher supervision. As described in Section 4.2 of the main text, for each sample score wn(t), we floor and normalized it as follows: in(t)=⌊wn(t)⌋ and wn′(t)=⌊wn(t)⌋/Wm. Here, in(t) serves as the index for the lookup table F, and λwn′(t) is the value corresponding to the index in(t). Specifically, λwn′(t) represents the teacher supervision weight αn(t) for sample n, i.e., αn(t)=λwn′(t). The tuple (in(t),λwn′(t)) is then added as an element to the lookup table F. Therefore, the lookup table F is defined as:
F={(in(t),λwn′(t))∣in(t)∈{0,1,…,Wm}},
where α∈[0,λ] is a trade-off coefficient, and λ is a method-specific constant. For example, with λ=1 for KD and λ=2 for DKD.
During training, the sample score wn(t) is linearly scaled to [0,Wm] and then floored to obtain the lookup table index in(t)=⌊wn(t)⌋. Subsequently, using in(t) as the index, we retrieve the corresponding table value λwn′(t) from the lookup table F, hereby obtaining the distillation supervision weight coefficient for this sample:
αn(t)=λwn′(t).
Thus, the task supervision weight coefficient is λ−αn(t). A higher wn(t) indicates a poorer mastery state under this scoring rule, corresponding to a larger relative weight for teacher supervision, and a lower wn(t) corresponds to a larger relative weight for task supervision. Thus, the learning state described by the three components is translated into the actual supervision allocation.