acceptodds
Under review as a conference paper at ICLR 2027

PIDScore: A Student-Centric Adaptive Fusion Method for Knowledge Distillation

Abstract

Knowledge distillation aims to train lightweight student models by balancing soft supervision from a teacher model with hard supervision from ground-truth labels, making it a key challenge to determine an appropriate fusion weight. Existing methods that use fixed fusion weights lack adaptability, while another class of strategies that rely on dynamic adjustments based on teacher signals fail to fully account for the learning state of the student model itself. To overcome these limitations, we propose PIDScore, a student-centric adaptive fusion method inspired by Proportional-Integral-Derivative (PID) controller theory. PIDScore relies solely on the student's own state and the ground-truth labels, dynamically adjusting the fusion weight at the sample level and operating completely independent of the teacher model. Specifically, it employs a PID branch that relies solely on the student model, integrating student’s current prediction error (P), historical performance (I) and learning trend (D) to offer precise, fine-grained optimization guidance. As a teacher-agnostic, plug-and-play module, PIDScore demonstrates strong effectiveness and excellent generalization, consistently improving the performance of various state-of-the-art distillation methods on CIFAR-100, MS-COCO and ImageNet across extensive experiments. The code of this work will be released upon acceptance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Knowledge Distillation, PID, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LizrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an adaptive loss fusion strategy for knowledge distillation (KD) which is intrinsically composed of two types of loss functions derived from ground-truth label and teacher-network supervision. The weight for those losses is regarded as a target parameter to be controlled and the authors introduce PID-control mechanism to tune it adaptively based on training errors. According to PID, the prediction errors of the student model are harnessed in the forms of proportional, integral and derivatives, all of which are integrated into a PID-score to be mapped into the loss weight parameter. In the experiments on KD tasks in image classification/detection, the proposed approach renders performance improvement to various KD methods.

Critical strengths
  • PID control mechanism is interestingly applied to tuning loss weights in a knowledge distillation framework. As the weight parameter is tuned by hands in most cases, the approach would reduce costly practitioners' effort to improve KD performance.
  • Efficacy of the proposed method is validated in a comprehensive manner across broad types of KD approaches.
Critical weaknesses
  • Rationale to regard loss weight tuning as PID controlling is unclear and less articulated in the paper. From engineering perspectives, PID is applied to align an output with a target by mitigating the issues of bias, overshoot and instability during an approach of the output signal toward the target. On the other hand, the KD training of DNN models is less relevant to those issues that PID focuses on. Namely, alignment of student output (logits) to the target (ground-truth label) in (6) is not solely the objective of the KD learning; the KD objective is rather to slightly deviate the student output from the target to attain favorable characteristics (a.k.a dark knowledge) embedded in the teacher model. Or more generally speaking, the ultimate goal of KD learning is to enhance generalization performance of the student network, which is less relevant to the goal of PID control. Therefore, the approach simply based on PID seem to lack theoretical soundness; specifically, it is unclear how the P/I/D terms are connected to the weight parameter α\alpha for improvement of KD learning.

  • The presented approach resorts to tons of hyper-parameters to be heuristically tuned. Even the key mapping function (12) is implemented by means of loogup table which is not clearly shown at all in this paper. While, in control engineering fields, PID gains (hyper-parameters) are designed in a principled way such as via transfer functions, the presented approach is formulated in a ad-hoc manner.

    • In the method, gains Kp/i/dK^{p/i/d} are tuned during the training in an ad-hoc way, which further deviates the method from principled PID concept. In addition, the descriptions in lines 328-338 are quite confusing. For example, the statement of "poorer mastery correspond to larger KpnK_p^n" (line 335) seems to be wrong since "larger LL indicates poorer mastery" (line 329) and KpnK_p^n is negatively correlated with LL in (13).
  • The error function (6) is just a linear approximation (lower-bound) for the cross-entropy loss and they are one-to-one corresponded (on zn(t)(k)>0z_n(t)^{(k)}>0). However, as shown in Table 11, there is considerable performance gap between those criteria; they should be close by properly tuning the mapping function ff. In addition, from the KD perspectives, it is natural to regard the teacher logits as target signals in PID framework. Therefore, the error (6) missing the teacher signals is less convincing; one can guess it is employed from practical performance viewpoint in Table 11, being away from PID concept.

Questions for authors
  • Please provide clear response to the above-mentioned concerns, especially regarding theoretical connection (rationale) to PID mechanism.
13 comments
AuthorsOpenReview ↗

Thank you for the careful and constructive review (W1 & Q1-Part 1/2)

W1: "Rationale to regard loss weight tuning as .... weight parameter α\alpha for improvement of KD learning."

Q1: "Please provide clear response to the above-mentioned concerns, especially regarding theoretical connection (rationale) to PID mechanism."

Response to W1 and Q1 (part 1/2): We thank the reviewers for pointing out this issue. In Appendix I, "PID Feedback in Knowledge Distillation" (lines 1256–1268) of the revised manuscript, we have supplemented this connection, explicitly distinguishing the objective deviation used to observe the student's learning state from the joint loss used to actually optimize the student. We also explain how P/I/D affect the student update and form continuous feedback via supervision weights, and how teacher knowledge participates in the optimization through the original distillation objective. Meanwhile, in Section 3 and Eq. (6) of Section 4.1, we distinguish the student's raw logits from predicted probabilities and clarify the definition of the objective deviation. In Section 4.2, we supplement the mapping from sample scores to fusion weights and specify the final training objective function. Below, we sequentially explain how the three components form the sample score, how the sample score is converted into supervision weights, how the supervision weights influence student learning, then explain the resulting feedback mechanism, and finally introduce the experimental contributions of each component.

1. The P, I and D components describe the current error, recent history and error variation, respectively, and their sum forms the sample score. According to Eq. (6) in Section 4.1 of the revised manuscript, the objective deviation is defined as the difference between the ground-truth class label and the student's predicted probability for that class. Therefore, this signal reflects the student's current mastery state of the ground-truth class and serves as the common input for the three feedback branches (P, I, D).

Following Eqs. (8)–(11) in the main text, the P component combines the current error with the proportional gain parameter, reflecting the student's immediate performance on the current sample. The I component aggregates the recent error history of the same sample, supplementing the temporal information of the student's sustained performance on this sample. The D component employs a second-order difference to describe how adjacent error increments change, supplementing the evolutionary information of the student's learning trend on this sample. The three components are individually modulated by their corresponding gain parameters and then summed according to Eq. (7):

wn(t)=wnp(t)+wni(t)+wnd(t)w_n(t)=w_n^p(t)+w_n^i(t)+w_n^d(t).

Therefore, the comprehensive score simultaneously incorporates the current state, historical performance, and variation trend. Even if two samples share the same current confidence for the ground-truth class, they may still receive different scores as long as their recent history or variation trends differ.

2. The comprehensive score is scaled and mapped via a look-up table to obtain the weights for task supervision and teacher supervision. As described in Section 4.2 of the main text, for each sample score wn(t)w_n(t), we floor and normalized it as follows: in(t)=⌊wn(t)⌋i_n(t)=\lfloor w_n(t)\rfloor and wn′(t)=⌊wn(t)⌋/Wmw_n^{\prime}(t)=\lfloor w_n(t)\rfloor/W_m. Here, in(t)i_n(t) serves as the index for the lookup table FF, and λwn′(t)\lambda w_n^{\prime}(t) is the value corresponding to the index in(t)i_n(t). Specifically, λwn′(t)\lambda w_n^{\prime}(t) represents the teacher supervision weight αn(t)\alpha_n(t) for sample nn, i.e., αn(t)=λwn′(t)\alpha_n(t)=\lambda w_n^{\prime}(t). The tuple (in(t),λwn′(t))(i_n(t), \lambda w_n^{\prime}(t)) is then added as an element to the lookup table FF. Therefore, the lookup table FF is defined as:

F={(in(t),λwn′(t))∣in(t)∈{0,1,…,Wm}}F=\lbrace (i_n(t),\lambda w'_n(t)) \mid i_n(t)\in \lbrace 0,1,\ldots,W_m \rbrace \rbrace,

where α∈[0,λ]\alpha \in [0,\lambda] is a trade-off coefficient, and λ\lambda is a method-specific constant. For example, with λ=1\lambda=1 for KD and λ=2\lambda=2 for DKD.

During training, the sample score wn(t)w_n(t) is linearly scaled to [0,Wm][0,W_m] and then floored to obtain the lookup table index in(t)=⌊wn(t)⌋i_n(t)=\lfloor w_n(t)\rfloor. Subsequently, using in(t)i_n(t) as the index, we retrieve the corresponding table value λwn′(t)\lambda w'_n(t) from the lookup table FF, hereby obtaining the distillation supervision weight coefficient for this sample:

αn(t)=λwn′(t)\alpha_n(t)=\lambda w'_n(t).

Thus, the task supervision weight coefficient is λ−αn(t)\lambda-\alpha_n(t). A higher wn(t)w_n(t) indicates a poorer mastery state under this scoring rule, corresponding to a larger relative weight for teacher supervision, and a lower wn(t)w_n(t) corresponds to a larger relative weight for task supervision. Thus, the learning state described by the three components is translated into the actual supervision allocation.

AuthorsOpenReview ↗

Thank you for the careful and constructive review (W1 & Q1-Part 2/2)

W1 (continued): "specifically, it is unclear how the P/I/D terms are connected to the weight parameter α\alpha for improvement of KD learning."

Response to W1 and Q1 (part 2/2):

3. The weights enter the joint training objective, modulating the relative contributions of task supervision and teacher supervision to the student update. Following Eq. (13) in the revised manuscript, the final training objective is:

L(t)=1N∑n=1N[(λ−αn(t))LCEn(t)+αn(t)LKDn(t)]\mathcal L(t)=\frac{1}{N}\sum_{n=1}^{N}\big[(\lambda-\alpha_n(t))\mathcal L_{\mathrm{CE}}^n(t)+\alpha_n(t)\mathcal L_{\mathrm{KD}}^n(t)\big],

where NN denotes the total number of samples, and the two types of loss respectively preserve the internal coefficients, temperature factors, and sub-loss structures of the base method. Therefore, altering αn(t)\alpha_n(t) changes the relative contributions of the gradients from the two losses to the student parameter updates, causing the supervision intensity to dynamically vary with the sample's learning state. We agree that the ultimate goal of KD is to improve the student's generalization performance, and that the non-target class probabilities in the teacher's soft distribution and their relative relationships constitute important supervision information. Hence, PIDScore does not discard the original distillation objective. The actual update is still determined by the joint loss of LCEL_{CE} and LKDL_{KD}.

4. The above process forms a continuous feedback loop from the student state to the supervision weight allocation, and back to the student state. First, the objective deviation en(t)e_n(t) is generated based on the student's prediction. The P, I and D components then form the sample score wn(t)w_n(t) based on the current error, recent history, and variation trend. This score is mapped to obtain the supervision weight αn(t)\alpha_n(t) , which subsequently influences the student update via the joint loss. The updated student produces new predictions, continuously refreshing the historical state of the same sample for use in subsequent scoring and weight computation. This forms a closed loop: "Student State→P/I/D Scores→Supervision Weights→Joint Loss and Student Update→New Student State".

5. Branch contribution analysis and ablation experiments demonstrate that the P, I and D branches all participate in generating the supervision weight αn(t)\alpha_n(t) , and that jointly utilizing the information from all three branches yields performance benefits. Tables 16 and 17 in Appendix G.2 vary the initial gain parameters and the adaptive adjustment step size, respectively, reporting the accuracies across seven teacher-student model pairs under three random seeds, as well as the relative contributions of P, I and D to the fusion weights. On ResNet56→ResNet20, with the default initialization (Kp,0,Ki,0,Kd,0)=(0.80,0.15,0.05)(K_{p,0},K_{i,0},K_{d,0})=(0.80,0.15,0.05) and δp=0.05\delta_p=0.05, the mean contributions of the three components are 59.68%, 21.84%, and 18.48%, respectively, with I and D combined accounting for 40.32%. This demonstrates that the current state, historical performance, and variation trend all make observable contributions to the fusion weights.

Table 5 in the main text further compares P-only, I-only, D-only and the complete PID. Across the six method and architecture settings corresponding to KD, NRKD, CAT-KD, and MLKD, the complete PID achieves the highest accuracy. Lines 509-517 of the revised manuscript have clarified that the single-branch experiments disable the Score-Guided Adaptation (SGA, see Section 4.2 of the main text), fix the gain parameter of the enabled branch, and keep the gain parameters of the remaining branches at 0 throughout the training. For instance, when the I branch is disabled, its gain parameter KinK_i^n is set to 0 and maintained at 0 throughout the entire training process. In the ResNet56→ResNet20 KD setting, P-only, I-only, D-only and the complete PID achieve 70.66%, 71.31%, 71.59%, 71.74% and 72.15%, respectively. The complete PID improves by 0.84 percentage points compared to P-only and by 0.41 percentage points compared to the strongest single component, D, providing performance evidence for the joint utilization of the three types of feedback information.

AuthorsOpenReview ↗

Thank you for the careful and constructive review (W2-Part 1/3)

W2: "The presented approach resorts to ...., the presented approach is formulated in ... correlated with LL in (13)."

Response to W2 (part 1/3): We thank the reviewer for pointing out the need for further clarification regarding the lookup table construction, its design rationale and the gain parameter adjustment direction. In Section 4.2 of the revised manuscript, we have supplemented the construction of the lookup table and the correspondence between indices and table values. We present the final training objective function in Eq. (13) and specify the score-guided PID gain parameter adjustment rules in Eqs. (14)–(15). Specifically, Eq. (14) has been corrected to reflect the proper correspondence among the learning levels, adjustment multipliers, and proportional gain parameters. Appendix I further details the complete feedback process linking student state observation, supervision weights, and student updates. Below, we address the three issues sequentially: look-up table construction, method design and empirical basis, and the gain parameter adjustment direction.

1. The lookup table is pre-constructed following a unified rule. As described in Section 4.2 of the main text, for each sample score wn(t)w_n(t), we floor and normalized it as follows: in(t)=⌊wn(t)⌋i_n(t)=\lfloor w_n(t)\rfloor and wn′(t)=⌊wn(t)⌋/Wmw_n^{\prime}(t)=\lfloor w_n(t)\rfloor/W_m. Here, in(t)i_n(t) serves as the index for the lookup table FF, and λwn′(t)\lambda w_n^{\prime}(t) is the value corresponding to the index in(t)i_n(t). Specifically, λwn′(t)\lambda w_n^{\prime}(t) represents the teacher supervision weight αn(t)\alpha_n(t) for sample nn, i.e., αn(t)=λwn′(t)\alpha_n(t)=\lambda w_n^{\prime}(t). The tuple (in(t),λwn′(t))(i_n(t), \lambda w_n^{\prime}(t)) is then added as an element to the lookup table FF. Therefore, the lookup table FF is defined as:

F={(in(t),λwn′(t))∣in(t)∈{0,1,…,Wm}}F=\lbrace (i_n(t),\lambda w'_n(t)) \mid i_n(t)\in \lbrace 0,1,\ldots,W_m \rbrace \rbrace,

where α∈[0,λ]\alpha \in [0,\lambda] is a trade-off coefficient, and λ\lambda is a method-specific constant. For example, with λ=1\lambda=1 for KD and λ=2\lambda=2 for DKD. All table entries are generated via a single linear mapping and require no tuning.

During training, the sample score wn(t)w_n(t) is linearly scaled to [0,Wm][0,W_m] and then floored to obtain the lookup table index in(t)=⌊wn(t)⌋i_n(t)=\lfloor w_n(t)\rfloor. Subsequently, using in(t)i_n(t) as the index, we retrieve the corresponding table value λwn′(t)\lambda w'_n(t) from the lookup table FF, hereby obtaining the distillation supervision weight coefficient for this sample:

αn(t)=λwn′(t)\alpha_n(t)=\lambda w'_n(t).

Thus, the task supervision weight coefficient is λ−αn(t)\lambda-\alpha_n(t). Since wn′(t)∈[0,1]w'_n(t) \in [0, 1], both coefficients lie in [0,λ][0,\lambda], and their sum is λ\lambda. Consequently, a higher wn(t)w_n(t) corresponds to a larger distillation supervision coefficient, while a lower wn(t)w_n(t) corresponds to a larger task supervision coefficient.

2. PIDScore possesses a complete operational logic spanning state observation, temporal scoring, supervision allocation, and subsequent updates; its design is validated through cross-framework experiments, ablations, and parameter analysis. Our design rationale is to dynamically adjust the supervision weight of each sample based on the student model's mastery degree of that sample, thereby adapting to the student's requirements under different learning states. To this end, PIDScore defines a unified state observation based on the student's predicted probability of the ground-truth class. It utilizes the P, I, and D components to describe the current error, recent history, and variation trend, respectively, and subsequently maps the sample score to the weights for task supervision and teacher supervision. The two types of weighted losses jointly update the student, and the updated predictions are continually utilized for subsequent scoring. Following Eqs. (14)–(15) in Section 4.2, the score is also employed to determine the learning level, adjusting the gain parameters of the three branches according to predefined rules. These steps seamlessly interconnect, ensuring that the scoring, loss weighting, and gain parameter adjustment all serve the same supervision allocation objective.

This operational logic also dictates the approach for integration into different distillation frameworks. The sample score exclusively utilizes student predictions, ground-truth labels, and the historical state of the same sample, independent of specific teacher features or relation representations. Upon integration, the original loss coefficients, temperature factors, and sub-loss composition of the base method are preserved, after which the unified scoring and mapping rules modulate the relative contributions of task supervision and distillation supervision. Therefore, the same feedback module can be readily combined with distillation methods based on logits, features, relations, and multiple teachers.

Reviewer_31cHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to set the weight between the ground-truth loss and the distillation loss in knowledge distillation. Existing methods use a fixed weight or adjust it with teacher-student differences. The paper proposes PIDScore, which sets a weight for each sample using only the student's own state. Experiments on CIFAR-100, ImageNet and MS-COCO show that it improves more than ten distillation methods, mostly by 0.05 to 1.56 points.

Critical strengths
  1. The method is plug-and-play. The weight depends only on the student's prediction and the ground-truth label, so it can be added to logit-based, feature-based and relation-based methods.

  2. The experiments cover a wide range. They include 14 teacher-student pairs, more than ten baselines, three datasets.

Critical weaknesses
  1. The description of the core design contradicts itself. Eq. (13) gives Kpn=Kp0−m(L)⋅δpK_{p}^n=K_{p0}-m(L)\cdot\delta_{p}, where m(L) grows as mastery gets worse. So a poorly learned sample gets a smaller KpK_{p}, but the text says it gets a larger KpK_{p}. The next sentence says that better mastery gives a larger X=m(L)⋅δpX = m(L)\cdot\delta_{p}, which also conflicts with the definition of m(L). In addition, the final weight comes from a precomputed lookup table f (Eq. 12), but the table is not given. This is the key step that sets the weight, so the method cannot be reproduced from the paper.

  2. The gains are small, and their reliability is unclear. Many gains are only 0.1 to 0.4 points, for example +0.10 for DKD on WRN-40-1 and for NRKD on ResNet20. On CIFAR-100, the variation across random seeds is usually about 0.2 to 0.3 points. It is hard to tell whether the gains are solid.

  3. CIFAR-100 has no official validation set, and the paper does not say which data was used to choose the initial gains, δp\delta_{p} and the window size. Appendix H reports that one setting is best on all seven pairs. If these values were chosen by test accuracy, the results are too optimistic.

Questions for authors
  1. Does a poorly learned sample get a larger or a smaller KpK_{p}? Please also give the content of the lookup table f and explain how it is built.

  2. On which data were the hyperparameters selected? If the test set was used, how do the results change when they are selected on a held-out validation split?

9 comments
AuthorsOpenReview ↗

Thank you for the careful and constructive review (W1 & Q1)

W1: "The description of the core design contradicts itself. Eq. (13) gives Kpn=Kp0−m(L)⋅δpK_p^n = K_{p0} - m(L)\cdot\delta_p, where m(L) grows as mastery gets worse. So a poorly learned sample gets a smaller KpK_p, but the text says it gets a larger KpK_p. The next sentence says that better mastery gives a larger X=m(L)⋅δpX = m(L)\cdot\delta_p, which also conflicts with the definition of m(L). In addition, the final weight comes from a precomputed lookup table f (Eq. 12), but the table is not given. This is the key step that sets the weight, so the method cannot be reproduced from the paper."

Q1: "Does a poorly learned sample get a larger or a smaller KpK_p? Please also give the content of the lookup table f and explain how it is built."

Response to W1 and Q1: We thank the reviewers for pointing out the contradictory description regarding the adjustment direction of the gain parameter and the insufficient definition of the lookup table. In Eq. (14) of Section 4.2 in the revised manuscript, we have corrected the correspondence among the learning levels, adjustment multipliers, and proportional gain parameters, and explicitly listed the adjustment multipliers for each level. Furthermore, we have clarified the construction of the look-up table and the correspondence between indices and table values in Section 4.2. We explain these two issues separately below.

1). The revised levels and adjustment multipliers ensure a consistent direction for gain adjustment. The score is defined along the direction of the prediction error: a higher score and a larger level LL indicate a poorer mastery state under this metric, while a lower score and a smaller LL indicate a better mastery state. Eq. (14) in the revised manuscript specifies:

Kpn=Kp,0−m(L)δpK_p^n=K_{p,0}-m(L)\delta_p,

and explicitly defines:

m(1)=6,m(2)=5,m(3)=4,m(4)=3,m(5)=2m(1)=6,\quad m(2)=5,\quad m(3)=4,\quad m(4)=3,\quad m(5)=2.

Therefore, m(L)m(L) decreases as LL increases. Given the initial gain parameter Kp,0K_{p,0} and a positive adjustment step size δp\delta_p, samples with poorer mastery correspond to a larger LL, a smaller m(L)m(L), and a smaller subtracted value X=m(L)δpX=m(L)\delta_p, thus obtaining a large KpnK_p^n. Samples with better mastery correspond to a larger m(L)m(L) and XX, thus obtaining a smaller KpnK_p^n. This correction ensures consistency among the formula, the adjustment multipliers, and the description in the main text.

2). The lookup table is pre-constructed following a unified rule. As described in Section 4.2 of the main text, for each sample score wn(t)w_n(t), we floor and normalized it as follows: in(t)=⌊wn(t)⌋i_n(t)=\lfloor w_n(t)\rfloor and wn′(t)=⌊wn(t)⌋/Wmw_n^{\prime}(t)=\lfloor w_n(t)\rfloor/W_m. Here, in(t)i_n(t) serves as the index for the lookup table FF, and λwn′(t)\lambda w_n^{\prime}(t) is the value corresponding to the index in(t)i_n(t). Specifically, λwn′(t)\lambda w_n^{\prime}(t) represents the teacher supervision weight αn(t)\alpha_n(t) for sample nn, i.e., αn(t)=λwn′(t)\alpha_n(t)=\lambda w_n^{\prime}(t). The tuple (in(t),λwn′(t))(i_n(t), \lambda w_n^{\prime}(t)) is then added as an element to the lookup table FF. Therefore, the lookup table FF is defined as:

F={(in(t),λwn′(t))∣in(t)∈{0,1,…,Wm}}F=\lbrace (i_n(t),\lambda w'_n(t)) \mid i_n(t)\in \lbrace 0,1,\ldots,W_m \rbrace \rbrace,

where α∈[0,λ]\alpha \in [0,\lambda] is a trade-off coefficient, and λ\lambda is a method-specific constant. For example, with λ=1\lambda=1 for KD and λ=2\lambda=2 for DKD. All table entries are generated via a single linear mapping and require no tuning.

During training, the sample score wn(t)w_n(t) is linearly scaled to [0,Wm][0,W_m] and then floored to obtain the lookup table index in(t)=⌊wn(t)⌋i_n(t)=\lfloor w_n(t)\rfloor. Subsequently, using in(t)i_n(t) as the index, we retrieve the corresponding table value λwn′(t)\lambda w'_n(t) from the lookup table FF, hereby obtaining the distillation supervision weight coefficient for this sample:

αn(t)=λwn′(t)\alpha_n(t)=\lambda w'_n(t).

Thus, the task supervision weight coefficient is λ−αn(t)\lambda-\alpha_n(t). Since wn′(t)∈[0,1]w'_n(t) \in [0, 1], both coefficients lie in [0,λ][0,\lambda], and their sum is λ\lambda. Consequently, a higher score corresponds to a larger distillation supervision coefficient, while a lower score corresponds to a larger task supervision coefficient.

AuthorsOpenReview ↗

Thank you for the careful and constructive review (W2)

W2: "The gains are small, and their reliability is unclear. Many gains are only 0.1 to 0.4 points, for example +0.10 for DKD on WRN-40-1 and for NRKD on ResNet20. On CIFAR-100, the variation across random seeds is usually about 0.2 to 0.3 points. It is hard to tell whether the gains are solid."

Response to W2: We thank the reviewer for raising this issue. We have further tuned the PIDScore parameters specifically for the teacher-student combinations that exhibited smaller gains in the original manuscript and re-ran the experiments. The updated results have been added to Table 1 in the main text and Table 8 in the appendix of the revised manuscript. Below, we explain the reasons for the relatively small original gains, as well as the updated experimental results.

The original experiments shared PID parameters within each method and did not perform individual tuning for each teacher-student model pair. The design goal of PIDScore is to serve as a plug-and-play module to improve the performance of existing distillation methods. To evaluate its effectiveness upon direct integration, in the original experiments, we selected a single set of PIDScore parameters for each distillation method based on validation set performance and uniformly applied it to different teacher-student model pairs under that method.

Different student architectures exhibit variations in their learning progressions, error levels, and variation trends; thus, a single set of scoring parameters may not fully adapt to every combination. Consequently, some combinations under the shared configuration still have room for parameter adaptation, which is one reason for the relatively small performance gains observed in certain cases in the original manuscript. The original results primarily reflected the effectiveness of directly integrating PIDScore under a unified configuration.

Further parameter adaptation improved the settings that originally exhibited smaller gains. On WRN-40-2→WRN-40-1, PIDScore increases the DKD accuracy from 74.81% to 75.62%, a gain of 0.81 percentage points. On ResNet56→ResNet20, it improves the NRKD accuracy from 71.92% to 72.56%, a gain of 0.64 percentage points. These results further demonstrate that these teacher-student model pairs still possess room for parameter optimization under the original shared parameter configuration. Overall, taking the equally weighted average of the accuracy improvements across the seven teacher-student combinations in Table 1, PIDScore yields an average improvement of 1.23 percentage points on KD, and 0.86 and 0.64 percentage points on MLKD and MLKD+LS, respectively. PIDScore adjusts the allocation of weight coefficients between task supervision and distillation supervision. Its gain magnitude varies depending on the base method and model combination. The updated results indicate that, with further parameter adaptation, it can not only improve the baseline KD but also continue to yield performance improvements on stronger distillation methods.

The updated Tables 1 and 8 are presented below. Accuracy is expressed in percentages, and arrows indicate the percentage point improvements relative to the corresponding baselines.

AuthorsOpenReview ↗

Thank you for the careful and constructive review (Updated Table 1)

The following table provides the updated Table 1 results discussed in our response to W2. Accuracy is reported in percent, arrows indicate gains in percentage points over the corresponding baseline.

TypeMethodResNet32×4 → ResNet8×4VGG13 → VGG8WRN-40-2 → WRN-40-1ResNet56 → ResNet20ResNet32×4 → SHN-V2ResNet32×4 → WRN-40-2ResNet50 → MN-V2
—Architecture relationshipSimilar architecturesSimilar architecturesSimilar architecturesSimilar architecturesDifferent architecturesDifferent architecturesDifferent architectures
—Teacher accuracy79.4274.6475.6172.3479.4279.4279.34
—Student accuracy72.5070.3671.9869.0671.8275.6164.60
FeatureFitNet (ICLR’15)73.5071.0272.2469.2173.5477.6963.16
FeatureAT (ICLR’17)73.4471.4372.7770.5572.7377.4358.58
FeatureRKD (CVPR’19)71.9071.4872.2269.6173.2177.8264.43
FeatureCRD (CVPR’20)75.5173.9474.1471.1675.6578.1569.11
FeatureOFD (ICCV’19)74.9573.9574.3370.9876.8279.2569.04
FeatureReviewKD (CVPR’21)75.6374.8475.0971.8977.7878.9669.89
FeatureSimKD (CVPR’22)78.0874.8974.5371.0578.3979.2969.97
FeatureCAT-KD (CVPR’23)76.9174.6574.8271.6278.4178.5971.36
FeatureCAT-KD+PIDScore77.53 (↑0.62)75.41 (↑0.76)75.51 (↑0.69)72.25 (↑0.63)79.02 (↑0.61)79.47 (↑0.88)71.98 (↑0.62)
FeatureTopKD (ICML’24)74.6073.8274.0271.2675.6577.4469.12
FeatureTopKD+PIDScore75.38 (↑0.78)74.58 (↑0.76)74.70 (↑0.68)71.93 (↑0.67)76.26 (↑0.61)78.01 (↑0.57)69.80 (↑0.68)
LogitKD (NeurIPS’14)73.3372.9873.5470.6674.4577.7067.35
LogitKD+PIDScore74.84 (↑1.51)73.68 (↑0.70)74.43 (↑0.89)72.15 (↑1.49)75.97 (↑1.52)78.41 (↑0.71)69.11 (↑1.76)
LogitKD+LS (CVPR’24)76.6274.3674.3771.4375.5677.9269.02
LogitKD+LS+PIDScore77.34 (↑0.72)74.98 (↑0.62)75.00 (↑0.63)72.08 (↑0.65)76.61 (↑1.05)78.77 (↑0.85)70.30 (↑1.28)
LogitDKD (CVPR’22)76.3274.6874.8171.9777.0778.4670.35
LogitDKD+PIDScore77.09 (↑0.77)75.32 (↑0.64)75.62 (↑0.81)72.50 (↑0.53)77.67 (↑0.60)79.09 (↑0.63)71.02 (↑0.67)
LogitCTKD (AAAI’23)73.3973.5273.9371.1975.3777.6668.67
LogitCTKD+PIDScore74.25 (↑0.86)74.40 (↑0.88)74.55 (↑0.62)72.33 (↑1.14)76.41 (↑1.04)78.28 (↑0.62)69.49 (↑0.82)
LogitMLKD (CVPR’23)77.0875.1875.3572.1978.4479.2671.04
LogitMLKD+PIDScore78.30‾\underline{78.30} (↑1.22)75.98 (↑0.80)76.33 (↑0.98)72.79 (↑0.60)79.08‾\underline{79.08} (↑0.64)80.36 (↑1.10)71.73 (↑0.69)
LogitMLKD+LS (CVPR’24)78.2875.2275.5671.5078.7679.6671.19
LogitMLKD+LS+PIDScore78.84 (↑0.56)75.83‾\underline{75.83} (↑0.61)76.22‾\underline{76.22} (↑0.66)72.28 (↑0.78)79.31 (↑0.55)80.34‾\underline{80.34} (↑0.68)71.86‾\underline{71.86} (↑0.67)
LogitSDD (CVPR’24)75.3373.6674.1670.7276.4978.5469.31
LogitSDD+PIDScore76.29 (↑0.96)74.26 (↑0.60)74.81 (↑0.65)71.60 (↑0.88)77.17 (↑0.68)79.37 (↑0.83)70.23 (↑0.92)
LogitTeKAP (ICLR’25)74.5273.7973.8071.3275.4376.8368.75
LogitTeKAP+PIDScore75.42 (↑0.90)74.52 (↑0.73)74.66 (↑0.86)71.97 (↑0.65)76.29 (↑0.86)78.24 (↑1.41)69.51 (↑0.76)
LogitEKD (ICCV’25)77.2174.7173.9569.9777.6577.7670.38
LogitEKD+PIDScore77.87 (↑0.66)75.36 (↑0.65)74.74 (↑0.79)70.58 (↑0.61)78.38 (↑0.73)79.32 (↑1.56)71.07 (↑0.69)
RelationNRKD (NN’25)76.3474.6274.3071.9276.7278.5568.16
RelationNRKD+PIDScore76.98 (↑0.64)75.28 (↑0.66)74.93 (↑0.63)72.56‾\underline{72.56} (↑0.64)77.42 (↑0.70)79.35 (↑0.80)69.10 (↑0.94)
Reviewer_xBgwOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript presents an interesting attempt to incorporate feedback-control concepts into knowledge distillation. However, the limited methodological novelty, inconsistencies in the mathematical formulation, insufficient theoretical justification, and unresolved questions regarding the adaptive weighting mechanism substantially weaken the contribution.

Although extensive experiments are reported, they do not convincingly establish that the proposed PID mechanism is theoretically justified, methodologically necessary, or reliably superior to simpler adaptive weighting alternatives.

These concerns affect the foundation of the proposed method rather than merely its presentation.

Critical strengths

see Questions For Authors

Critical weaknesses

see Questions For Authors

Questions for authors

The manuscript proposes PIDScore, a student-centric adaptive fusion method that introduces a proportional-integral-derivative (PID) mechanism to dynamically balance hard-label supervision and knowledge distillation. Although the authors report performance improvements across several distillation methods and datasets, the proposed framework raises substantial concerns regarding its methodological novelty, mathematical consistency, theoretical justification, and experimental validation. In particular, several fundamental issues remain unresolved, making it difficult to establish the scientific validity and necessity of the proposed approach.

  1. Insufficient Methodological Novelty and Distinction from Existing Adaptive Weighting Strategies

The primary contribution of PIDScore is to dynamically adjust knowledge distillation weights using the student's prediction error and its historical variations. However, the underlying idea of adapting supervision weights according to student confidence, sample difficulty, or training dynamics has already been extensively explored.

Although the manuscript compares PIDScore with several existing adaptive weighting strategies, it does not sufficiently establish what fundamental methodological advance is introduced by the PID formulation.

In particular, the authors should clarify:

i. How PIDScore fundamentally differs from conventional confidence-based and history-based sample reweighting methods. ii. Whether the integration of P, I, and D components represents a genuinely new optimization mechanism or merely a heuristic combination of existing signals. iii. Why a relatively complex PID-based design is necessary when simpler adaptive weighting strategies may achieve similar benefits.

Without convincing evidence of a distinct methodological contribution, the novelty of the proposed framework remains limited.

  1. Fundamental Mathematical Inconsistencies in the Proposed Formulation

The mathematical formulation of PIDScore contains inconsistencies that directly affect the interpretation of its central mechanism.

First, Equation (6) defines the prediction error using the student's output zn(k)z_n^{(k)}, which is introduced as a logit in Section 3 but subsequently treated as a probability in Section 4.1. Consequently, the claimed boundedness of the error signal is not justified by the stated notation.

More importantly, Equation (13) defines the proportional coefficient as Knp=K0p−m(L)δpK_n^p=K_0^p-m(L)\delta_p, where m(L)m(L) increases as sample mastery deteriorates. According to this formulation, poorly mastered samples receive smaller proportional coefficients, directly contradicting the authors' explanation that such samples should receive larger coefficients.

Furthermore, the manuscript does not adequately establish how the combined PID signal is constrained to the assumed nonnegative score range, particularly when the derivative component can take negative values.

These inconsistencies raise fundamental concerns about whether the mathematical formulation accurately represents the implemented algorithm.

  1. Insufficient Theoretical Justification for Introducing PID Control

The proposed framework is primarily motivated by an analogy between student learning dynamics and PID control. However, the manuscript lacks a rigorous theoretical connection between the PID mechanism and the optimization process of knowledge distillation.

In particular, Equation (11) adopts a second-order difference as the derivative component, whereas conventional PID control uses a first-order derivative. The authors argue that this design captures learning trends while reducing random fluctuations, but second-order differences do not inherently suppress noise and may amplify high-frequency disturbances.

Moreover, no formal analysis is provided to demonstrate that the proposed controller improves optimization stability, convergence behavior, or the quality of knowledge transfer.

Although the appendix reports empirical comparisons between first-order and second-order differences, these results do not establish the theoretical validity of the proposed control mechanism.

Consequently, the PID formulation appears largely heuristic rather than theoretically grounded.

  1. Incomplete Definition of the Adaptive Weight Mapping Mechanism

The adaptive mapping from PID outputs to knowledge distillation weights is a central component of PIDScore, yet its mathematical specification remains insufficient.

Equation (12) introduces a precomputed lookup function f(⋅)f(\cdot) without fully specifying how the PID score is converted into the final supervision weight.

Furthermore, Section 3 defines the fusion coefficient within [0,1][0,1], whereas Appendix I introduces additional loss weights within [0,2][0,2]. The relationship between these formulations and the actual training objective is not sufficiently clarified.

The manuscript should explicitly explain:

i. How the lookup function is constructed and applied. ii. How the PID score is normalized and mapped into the final loss weights. iii. How the original loss coefficients are preserved or modified across different distillation frameworks.

Without a precise and consistent formulation of this mapping process, the central optimization mechanism remains difficult to reproduce and evaluate.

  1. Insufficient Specification of Sample-Level Historical State Tracking

PIDScore requires historical prediction errors to compute its integral and derivative components. However, the manuscript does not adequately explain how these sample-level temporal states are maintained during standard mini-batch training.

Since training samples are randomly shuffled and may reappear at different iterations, it is unclear how the historical errors of the same sample are consistently retrieved and updated.

The authors should clarify how historical states are initialized, stored, and synchronized with individual samples, particularly when using shuffled mini-batches or distributed training.

Although Appendix F investigates different historical window sizes, these experiments do not resolve the implementation ambiguity of the sample-level temporal tracking mechanism.

This issue is particularly important because the claimed student-centric historical feedback is the central distinction of PIDScore from conventional adaptive weighting methods.

  1. Insufficient Ablation Evidence for the Necessity of the Complete PID Mechanism

Although the manuscript provides component-wise ablations and hyperparameter sensitivity analyses, the experimental evidence does not adequately isolate the contribution of the complete PID mechanism.

In particular, Table 16 reports nonzero contributions from the I and D branches even when their initial coefficients are set to zero. While this may result from the adaptive coefficient updates, it raises questions about whether the reported configurations genuinely isolate the intended PID components.

Furthermore, the manuscript does not sufficiently compare the full framework against simpler student-only alternatives, such as confidence-based weighting, exponential moving-average error tracking, or basic temporal weighting schedules.

The reported improvements may therefore arise from adaptive sample weighting or historical smoothing rather than from the specific PID architecture.

Without carefully controlled comparisons that isolate these factors, the necessity and effectiveness of the complete PID formulation remain insufficiently demonstrated.

  1. Insufficient Statistical Validation and Overstated Robustness and Generalization Claims

The manuscript reports improvements over numerous distillation baselines, but several gains are relatively small, sometimes approximately 0.1–0.3 percentage points.

Although the appendix reports repeated-run statistics for certain hyperparameter studies, the main performance comparisons do not consistently include standard deviations, confidence intervals, or statistical significance tests. It is therefore difficult to determine whether the smaller improvements are statistically reliable.

Moreover, the manuscript repeatedly emphasizes robustness and broad generalization. However, improvements across different architectures and distillation methods do not necessarily demonstrate robustness to noisy labels, unreliable teachers, distribution shifts, or training perturbations.

In particular, since PIDScore determines supervision weights without explicitly evaluating teacher reliability, its behavior under incorrect or misleading teacher supervision remains unclear.

Consequently, the current experiments do not adequately support the broader robustness and generalization claims.

13 comments
AuthorsOpenReview ↗

Thank you for the careful and constructive review (Q1-Part 1/2)

Q1:

In particular, the authors should clarify:

i. How PIDScore fundamentally differs from conventional confidence-based and history-based sample reweighting methods.

Response to Q1 (part 1/2): We thank the reviewer for raising these questions. In Section 2.2 of the revised manuscript, we have supplemented the distinctions between PIDScore and ADA-KD, RW-KD and WLS-KD regarding weight construction and cross-method adaptability, and have moved the direct comparisons with these methods from the original appendix to Table 4 in the main text. Below, we address the three questions respectively.

i). The distinctions between PIDScore and existing methods are reflected in the weight construction mechanism, as well as the adaptability across distillation methods and model architectures. Regarding weight construction, PIDScore generates per-sample supervision weights based on student predictions, ground-truth labels and the recent history of the same sample. ADA-KD pre-estimates class-level weights based on teacher class reliability on a labeled hold-out set, where samples of the same class share the same weight. WLS-KD constructs sample weights based on the current classification predictions of both the teacher and the student. RW-KD, on the other hand, estimates per-sample loss weights via meta-learning, utilizing loss gradients on the training set and a meta-dataset. PIDScore uses student-side temporal feedback as a unified scoring criterion to dynamically adjust the relative contributions of task supervision and teacher supervision. Therefore, in terms of weight construction, PIDScore is distinct from ADA-KD, RW-KD and WLS-KD.

Mechanistically, since ADA-KD constructs weights based on teacher class reliability, it cannot continuously reflect each student's learning progression on specific samples during training. WLS-KD constructs sample weights based on the current classification predictions of the teacher and student, with the weights depending on the instantaneous confidence relationship between them regarding the ground-truth class. When the teacher-student architecture is changed, differences in confidence and calibration will alter this relationship. When the distillation objective changes, the teacher supervision benefit corresponding to the same prediction relationship may also vary. Furthermore, this rule does not utilize the learning history and trend of the same sample, making it difficult to distinguish samples with similar current confidence but different learning processes. Therefore, directly applying the original weighting rule may not consistently yield the same benefits under different architectures and distillation objectives. The weights in RW-KD rely on the gradients of the training objective and meta-objective of the chosen KD method. When the distillation method changes from KD to DKD, because DKD decouples the objective, the loss structure and gradients upon which these weights act will also change. PIDScore uses ground-truth labels as a common reference, observes the student's learning state through a unified scoring rule, and then applies the weights to the task loss and distillation loss branches, preserving the sub-loss structure within the base method. Its scoring does not depend on specific teacher features or relation matrices, nor does it require additional meta-optimization steps. These design differences provide a mechanistic explanation for the better cross-method and cross-architecture adaptability observed in the aforementioned experiments. Therefore, mechanistically, PIDScore is also distinct from ADA-KD, RW-KD and WLS-KD.

Regarding cross-method and cross-architecture performance, Table 4 in the main text provides direct evidence. After transferring to DKD, ADA-KD and RW-KD underperform the original DKD on all seven model pairs, and WLS-KD degrades on six of them. For example, on ResNet32x4 → ShuffleNetV2, the three methods decrease by 0.32, 1.92, and 0.84 percentage points, respectively, while PIDScore improves by 0.60 percentage points. Even within KD, changing the model combination leads to degradation: on ResNet32x4 → WRN-40-2, ADA-KD and WLS-KD decrease by 0.11 and 0.24 percentage points, respectively, while PIDScore improves by 0.71 percentage points. RW-KD decreases by 1.20 percentage points on VGG13 → VGG8, whereas PIDScore improves by 0.70 percentage points. PIDScore achieves performance improvements across all fourteen settings and attains the highest accuracy. Tables 1 and 2 in the main text further verify its applicability in logit-based, feature-based, relation-based, and multi-teacher distillation. Therefore, ADA-KD, RW-KD, and WLS-KD exhibit poor transferability. In contrast, PIDScore exhibits superior transferability. This is also one of the primary differences and advantages of PIDScore compared to ADA-KD, RW-KD and WLS-KD.

AuthorsOpenReview ↗

Thank you for the careful and constructive review (Q1-Part 2/2)

Q1: In particular, the authors should clarify:

ii. Whether the integration of P, I, and D components represents a genuinely new optimization mechanism or merely a heuristic combination of existing signals. iii. Why a relatively complex PID-based design is necessary when simpler adaptive weighting strategies may achieve similar benefits.

Response to Q1 (part 2/2): ii). The three components, P, I and D, incorporate the student's current state and temporal information into a unified feedback loop for continuous optimization. The P component responds to the current prediction error, the I component aggregates the recent error history of the same sample, and the modified D component characterizes the variation in the error trend via second-order differences. Together, they generate the score, which is then mapped to the weights for task supervision and distillation supervision. Consequently, even if two samples share the same current confidence for the ground-truth class, they can still receive different supervision allocations as long as their recent history or variation trends differ.

During training, the student predictions and their historical states are first used to compute the P/I/D scores and supervision weights. These weights are applied to the task loss and distillation loss branches, and the student is updated via joint backpropagation. The updated student produces new predictions in subsequent observations, which are then used to update the sample state and recalculate the weights. This forms a closed-loop optimization strategy: "Student State→P/I/D Scores→Supervision Weights→Weighted Loss and Student Update→New Student State". Because the feedback signals and scoring rules are independent of specific distillation representations, this process can be integrated with various distillation methods as a plug-and-play module.

iii). The direct comparisons and ablation studies jointly demonstrate that the complete PID design yields practical benefits and does not require complex, independent hyperparameter tuning in application. In Table 4 of the main text, the performance degradation of ADA-KD, RW-KD and WLS-KD after transferring to DKD indicates that adaptive weighting strategies do not consistently yield performance gains. In contrast, PIDScore brings improvements across these settings, reflecting the practical value of a reusable student feedback mechanism.

The ablation study in the main text provides direct experimental evidence for retaining the complete PID structure. Table 5 in the main text compares P-only, I-only, D-only and the complete PID under KD, NRKD, CAT-KD and MLKD, covering two teacher-student architectures and a total of six method-architecture combinations. In the single-component experiments (P-only, I-only, D-only), we disable the Score-Guided PID Adaptation (SGA, see lines 347-371 in the main text), fix the gain parameter of the enabled branch, and keep the gain parameters of the disabled branches strictly at 0 throughout the training process. For example, when only the P branch is enabled, the gain parameters for the I and D branches are kept at 0, i.e., Kin=Kdn=0K_i^n=K_d^n=0. The complete PID achieves the best results across all six settings. For instance, in KD (ResNet56→ResNet20), the accuracies for the baseline, P-only, I-only, D-only and complete PID are 70.66%, 71.31%, 71.59%, 71.74% and 72.15%, respectively. The complete PID improves upon P-only by 0.84 percentage points and still outperforms the strongest single component, D, by 0.41 percentage points. Averaged over the six settings, the complete PID improves by approximately 0.61 percentage points compared to P-only, and by approximately 0.37 percentage points compared to the strongest single component in each respective setting.

These results indicate that the current error, recent history and variation trend play complementary roles, and using any single piece of information alone fails to match the performance of the complete combination. They support the joint use of P, I and D in the evaluated settings of this paper and provide empirical evidence for the necessity of adopting the complete PID structure.

In the practical application of PIDScore, only two parameters need to be tuned: the initial proportional gain parameter KpK_p (denoted as Kp,0K_{p,0} in the paper) and the adjustment step size δp\delta_p. According to Eq. (16) in Appendix G.1, the initial I and D gain parameters are determined by Ki,0=34(1−Kp,0)K_{i,0}=\tfrac{3}{4}(1-K_{p,0}) and Kd,0=14(1−Kp,0)K_{d,0}=\tfrac{1}{4}(1-K_{p,0}). Subsequent updates during training follow the predefined score-guided rules. Therefore, utilizing three feedback branches does not imply that the gains of the three branches need to be independently tuned.

AuthorsOpenReview ↗

Thank you for the careful and constructive review(Q2-Part 1/2)

Q2: "The mathematical formulation of PIDScore contains inconsistencies that directly affect the interpretation of its central mechanism.

First, Equation (6) defines the prediction error using the student's output zn(k)z_n^{(k)}, which is introduced as a logit in Section 3 but subsequently treated as a probability in Section 4.1. Consequently, the claimed boundedness of the error signal is not justified by the stated notation.

More importantly, Equation (13) defines the proportional coefficient as Knp=K0p−m(L)δpK_n^p=K_0^p-m(L)\delta_p, where m(L)m(L) increases as sample mastery deteriorates. According to this formulation, poorly mastered samples receive smaller proportional coefficients, directly contradicting the authors' explanation that such samples should receive larger coefficients."

Response to Q2 (part 1/2): We thank the reviewer for pointing out the inconsistencies in the mathematical expressions. We have made the following revisions in the revised manuscript: In Section 3 and Eq. (6) of Section 4.1, we distinguish between raw logits and predicted probabilities, and explicitly define the scoring error. In Eq. (14) of Section 4.2, we correct the correspondence between the learning levels, adjustment multipliers and proportional gains, and list the adjustment multipliers for each level. In Eqs. (12)-(13) of Section 4.2, we supplement the relationship among the linearly scaled scores, the look-up coefficients, and the final training objective. We explain these issues in turn below.

First, the scoring error is defined based on the predicted probability, with its range being [0,1]. In Section 3 of the revised manuscript, zn(t)z_n(t) denotes the raw logits. In Eq. (6) of Section 4.1 (lines 252-256), pn(t)p_n(t) denotes the predicted probability used for scoring, given by:

pn(t)=softmax⁡(zn(t))p_n(t)=\operatorname{softmax}(z_n(t)),

en(t)=yn(k)−pn(t)(k)=1−pn(t)(k)∈[0,1]e_n(t)=y_n^{(k)}-p_n(t)^{(k)}=1-p_n(t)^{(k)}\in[0,1],

where kk is the ground-truth class of the sample nn, so the corresponding component of the one-hot label is yn(k)=1y_n^{(k)}=1. The student's predicted probability for the ground-truth class lies in [0, 1], so the error is non-negative and does not exceed 1. A larger error indicates lower confidence in the ground-truth class. The scoring uses a softmax with a temperature of 1, which is distinguished from the distillation temperature of the KD loss itself.

Second, the revised levels and adjustment multipliers ensure a consistent direction for gain adjustment. The score is defined along the direction of the prediction error: a higher score and a larger level LL indicate a poorer mastery state under this metric, while a lower score and a smaller LL indicate a better mastery state. Eq. (14) in the revised manuscript specifies:

Kpn=Kp,0−m(L)δpK_p^n=K_{p,0}-m(L)\delta_p,

and explicitly defines:

m(1)=6,m(2)=5,m(3)=4,m(4)=3,m(5)=2m(1)=6,\quad m(2)=5,\quad m(3)=4,\quad m(4)=3,\quad m(5)=2.

Therefore, m(L)m(L) decreases as LL increases. Given the initial gain parameter Kp,0K_{p,0} and a positive adjustment step size δp\delta_p, samples with poorer mastery correspond to a larger LL, a smaller m(L)m(L), and a smaller subtracted value X=m(L)δpX=m(L)\delta_p, thus obtaining a large KpnK_p^n. Samples with better mastery correspond to a larger m(L)m(L) and XX, thus obtaining a smaller KpnK_p^n. This correction ensures consistency among the formula, the adjustment multipliers, and the description in the main text.

Comments on the paper

9 comments
AuthorsOpenReview ↗

To AC, SAC, PC and Reviewers

Dear AC, SAC, PC and Reviewers,

We have revised the manuscript according to the reviewers’ comments. All changes are highlighted in blue in the manuscript, and the revised manuscript has been updated on OpenReview. We have also updated and refined the rebuttal responses, with more detailed explanations where helpful, to facilitate review. We hope our responses address the reviewers’ concerns and would be happy to continue the discussion. We sincerely thank the reviewers for their careful reading and constructive suggestions, and thank the AC, SAC, and PC for their time and effort.

Best regards,

Authors

AuthorsOpenReview ↗

Response to Reviewer (New Q1 & Q6-Part 1/2)

Q1: Methodological Novelty The authors clarify how PIDScore differs from ADA-KD, RW-KD, and WLS-KD. However, combining current prediction error, historical error, and error variation does not yet establish why a PID-based formulation is necessary. The P, I, and D components resemble existing error-based, history-based, and trend-based weighting strategies. Could the authors compare PIDScore with student-confidence weighting, EMA-based weighting, and simple non-PID combinations of current and historical errors under identical settings?

Q6: Necessity of the Complete PID Mechanism The clarification of single-component ablations and SGA is helpful. Nevertheless, the current ablations do not fully establish that the gains arise from the PID architecture rather than simply from using additional historical information and adaptive weights. Comparisons with EMA-based error tracking and simple linear combinations of current and historical errors would provide stronger evidence for the necessity of the complete PID design.

Response to Q1 and Q6 (Part 1/2): We sincerely thank the reviewer for the suggestion to include simpler student-side weighting methods to further analyze the effect of the complete PID design. On CIFAR-100, using vanilla KD as the baseline, we compared student-confidence weighting, EMA-base error weighting, a simple linear combination of current and historical errors, fixed-gain parameter PID, and the full PIDScore on two teacher-student pairs: ResNet32x4→WRN-40-2 and ResNet56→ResNet20. All methods share the same experimental setup.

Specifically, student-confidence weighting utilizes the student's current confidence error regarding the ground-truth class. EMA applies an exponential moving average to the error of the same sample with a decay rate of 0.9. The simple linear combination integrates current and historical errors over a time window of 3. The fixed-gain parameter PID retains all three components (P, I, D) with fixed gain parameters set to (0.8, 0.15, 0.05), while disabling Score-Guided PID Adaptation (SGA). The full PIDScore uses the same initial gain parameters but enables SGA. The table below reports the accuracy (%).

MethodResNet32×4 → WRN-40-2ResNet56 → ResNet20
Vanilla KD77.7070.66
Student-Confidence Weighting77.8271.31
EMA-Base Error Weighting77.8171.42
A Linear Combination of Current and Historical Errors77.8971.49
Fixed-Gain Parameter PID78.1971.77
PIDScore78.4172.15

1. Confidence weighting lacks the ability to distinguish the learning process, whereas PIDScore supplements the current state with temporal information. Relying solely on the current prediction error, confidence weighting cannot distinguish samples with similar current confidence but distinct recent trends (e.g., continuously decreasing or repeatedly fluctuating errors). PIDScore incorporates historical errors and error trend variations into the scoring mechanism, enabling the supervision allocation to simultaneously consider the current state and the recent learning process. On both model pairs, confidence weighting outperforms the baseline KD, while the full PIDScore further improves upon confidence weighting by 0.59 and 0.84 percentage points, respectively. This supports the further utilization of temporal information to evaluate sample states.

2. EMA aggregates temporal information into a single signal, whereas PIDScore characterizes and combines these pieces of information separately. Although EMA introduces historical errors, identical smoothed values may correspond to entirely different current errors and recent variation processes; thus, it is difficult to distinguish these states based solely on this single value. PIDScore utilizes the P, I, and D components to respectively characterize the current error, recent cumulative error, and error trend variations, which are then jointly used to generate the score, thereby providing a more comprehensive basis for supervision allocation. On both model pairs, the full PIDScore improves upon EMA by 0.60 and 0.73 percentage points, respectively, demonstrating that this scoring approach outperforms weighting based solely on smoothed errors in the tested settings.

AuthorsOpenReview ↗

Response to Reviewer (New Q1 & Q6-Part 2/2)

Response to Q1 and Q6 (Part 2/2):

3. While the simple linear combination utilizes current and historical errors, it does not explicitly characterize error trend variations, and the relative weights of the two types of information remain fixed. PIDScore further introduces the D component and employs SGA to adjust the gains of each component based on the sample state, allowing the scoring to account for diverse temporal information and their relative contributions. The linear combination achieves 77.89% and 71.49% on the two model pairs, respectively, and the full PIDScore further improves upon this by 0.52 and 0.66 percentage points. Below, we further isolate the contributions of the three-component scoring and gain adaptation.

4. The comparison with the fixed-gain PID supports retaining the three-component scoring. To investigate the trend variation information not explicitly utilized by the simple linear combination, we introduce a fixed-gain parameter PID with SGA disabled. This setup jointly uses P, I, and D with constant component gain weights, while the supervision weight still updates according to the sample state. It achieves 78.19% and 71.77% on the two model pairs, respectively, which is 0.30 and 0.28 percentage points higher than the linear combination. This indicates that, in the tested settings, the three-component scoring yields additional gains even without gain adaptation.

5. SGA further addresses the limitations of the fixed-gain parameter PID. Although the fixed-gain parameter PID utilizes the three types of information, the relative contributions of each component cannot be dynamically adjusted according to the sample state. SGA updates the component gain weights based on the sample score, allowing the scoring to adjust the gain weights for the three modules: current error, historical performance, and trend variations. Under identical component definitions and initial gains, enabling SGA improves the accuracy of the two model pairs by 0.22 and 0.38 percentage points, respectively, providing further support for the role of gain adaptation.

The design contribution of PIDScore lies in organizing the three types of student states into a per-sample feedback score and adjusting their relative contributions via SGA, thereby dynamically allocating task supervision and teacher supervision. The newly added comparisons demonstrate that the full PIDScore outperforms all the aforementioned strategies on both model pairs, improving upon the baseline KD by 0.71 and 1.49 percentage points, respectively. The comparisons involving the fixed-gain PID and SGA further support that these gains are attributed to both the three-component scoring and its adaptive adjustment. These results provide more direct empirical evidence for adopting the full PIDScore under our evaluation settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.