- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes adaptive knowledge distillation for domain generalization using intermediate student predictions. A forecaster uses auxiliary logits and uncertainty statistics to predict student correctness, assigning larger supervised-loss weights to predicted difficult examples while keeping the distillation coefficient fixed. All auxiliary components are removed at inference. Experiments on three vision datasets and two text ID–OOD task pairs show improvements over vanilla KD for ResNet-10 and the reported transformer settings, but mixed results for ResNet-18.
Critical strengths
The method preserves inference efficiency. It adapts supervision using student predictions and removes all auxiliary components after training, making it practical for compact-model deployment.
The evaluation includes an informative KD baseline. Table 2 reports domain-level results over three seeds. Vanilla KD exceeds ERM and the reproduced KDDG on all six student–dataset averages, and the proposed method further improves all three ResNet-10 averages.
The ablations support the usefulness of readout signals. Adding uncertainty features improves correctness AUROC from 63.7% to 76.7% (Figure 2). Additional readout stages correlate with higher forecaster AUROC and better OOD accuracy (Figure 3(f)), although these results alone do not establish why the resulting weights improve generalization.
Critical weaknesses
The gains are inconsistent across architectures. ResNet-18 underperforms KD on nine of twelve target domains and two of three dataset averages (Table 2). The ViT gain is only 0.26 percentage points, and both ViT and text results lack variability estimates (Figures 3(a)–(b)).
The distinct contribution is insufficiently isolated. Missing controls include a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights. AMAL and DEDIER are also not evaluated, leaving the advantage over simpler and closely related approaches unclear.
The computational trade-off remains unresolved. Figure 3(c) reports approximately twice the FLOPs and 1.5 times the wall time for 10,000 student optimizer steps. Without a KD comparison under a similar computational budget, the practical value of the modest and mixed gains is uncertain.
Questions for authors
Clarification: Are the instance weights detached using stop-gradient during student updates? Please clarify the gradient paths in Figure 1 and Algorithm 1.
Clarification: For text experiments, which data are used for checkpoint selection and hyperparameter tuning? Are these separate from the gold validation splits used to train the forecaster?
Additional evidence: Can you compare against a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights? Comparisons with AMAL and DEDIER would also clarify the contribution.
Additional evidence: Can you report paired seed-level differences with uncertainty estimates and compare against KD under a similar training budget? Please also analyze why the ResNet-10 gains largely disappear for ResNet-18.
5 comments
Response to Reviewer mB2s (2/3)
W2: The distinct contribution is insufficiently isolated. Missing controls include a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights. AMAL and DEDIER are also not evaluated, leaving the advantage over simpler and closely related approaches unclear.
Q3: Can you compare against a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights? Comparisons with AMAL and DEDIER would also clarify the contribution.
We present domain-wise OOD performance (averaged over 3 seeds) of DeDiER and AMAL against the existing results from the paper on DomainNet, PACS and OfficeHome for ResNet-10 below. Similar to our main results, we sweep method-specific hyperparameters and LR on DomainNet126 and transfer unchanged to PACS and OfficeHome.
DomainNet126
| Method | Clipart | Painting | Real | Sketch | Average |
|---|---|---|---|---|---|
| ERM | 51.27 ± 0.93 | 42.30 ± 0.47 | 48.25 ± 0.27 | 45.80 ± 1.22 | 46.90 ± 0.47 |
| KDDG | 54.90 ± 1.24 | 46.63 ± 0.49 | 52.22 ± 0.34 | 48.89 ± 0.82 | 50.66 ± 0.71 |
| KD | 56.00 ± 0.21 | 47.70 ± 0.37 | 53.67 ± 0.27 | 50.65 ± 0.45 | 52.01 ± 0.16 |
| AMAL | 52.34 ± 1.56 | 46.34 ± 0.86 | 53.63 ± 0.14 | 49.72 ± 1.34 | 50.51 ± 0.21 |
| DeDiER | 43.63 ± 0.48 | 44.19 ± 0.41 | 41.92 ± 0.40 | 45.73 ± 0.82 | 43.87 ± 0.29 |
| Ours | 56.93 ± 0.61 | 48.53 ± 1.02 | 53.73 ± 0.57 | 52.73 ± 1.45 | 52.98 ± 0.54 |
PACS
| Method | Art | Cartoon | Photo | Sketch | Average |
|---|---|---|---|---|---|
| ERM | 60.18 ± 1.46 | 70.40 ± 2.21 | 79.92 ± 2.25 | 67.93 ± 0.96 | 69.61 ± 0.34 |
| KDDG | 62.17 ± 0.58 | 66.77 ± 0.53 | 80.84 ± 0.13 | 66.47 ± 3.24 | 69.06 ± 0.82 |
| KD | 67.20 ± 2.06 | 67.73 ± 2.82 | 82.81 ± 0.31 | 69.33 ± 1.25 | 71.77 ± 0.41 |
| AMAL | 67.40 ± 1.15 | 67.13 ± 2.40 | 82.58 ± 0.35 | 68.44 ± 0.51 | 71.39 ± 0.71 |
| DeDiER | 58.74 ± 1.49 | 64.59 ± 1.25 | 75.95 ± 1.55 | 61.59 ± 3.68 | 65.21 ± 1.19 |
| Ours | 70.31 ± 2.08 | 69.35 ± 2.31 | 82.41 ± 1.53 | 68.76 ± 0.74 | 72.71 ± 0.79 |
OfficeHome
| Method | Art | Clipart | Product | Real World | Average |
|---|---|---|---|---|---|
| ERM | 30.11 ± 1.06 | 31.37 ± 1.76 | 49.68 ± 0.63 | 48.63 ± 0.65 | 39.95 ± 0.55 |
| KDDG | 31.86 ± 1.13 | 33.62 ± 1.16 | 53.25 ± 0.89 | 52.75 ± 1.49 | 42.87 ± 1.16 |
| KD | 37.95 ± 1.57 | 37.24 ± 1.31 | 58.48 ± 0.59 | 58.69 ± 0.80 | 48.09 ± 0.66 |
| AMAL | 37.08 ± 0.62 | 36.74 ± 0.67 | 59.11 ± 0.58 | 59.14 ± 0.91 | 48.02 ± 0.38 |
| DeDiER | 28.73 ± 0.57 | 31.60 ± 1.59 | 50.50 ± 0.85 | 55.09 ± 0.99 | 41.48 ± 0.28 |
| Ours | 39.65 ± 0.73 | 38.78 ± 0.81 | 58.62 ± 0.99 | 57.90 ± 0.59 | 48.74 ± 0.63 |
Here, AMAL serves as a representative meta-learning-based adaptive instance-specific weighting algorithm for mixing KD and CE loss at an instance level. In this approach, the instance-wise mixing weights are learned using a meta-objective on a held-out in-domain validation set. On DomainNet, ours achieves the best average (52.98%), outperforming KD by 0.97 points and AMAL by 2.47 points. On PACS we achieve the best average (72.71%), outperforming KD by 0.94 points and AMAL by 1.32 points. On OfficeHome, our approach again achieves the best average (48.74%), but the gain over KD is smaller (+0.65 points). Notably, AMAL achieves the best accuracy on Product (59.11%) and Real World (59.14%), outperforming both KD and ours on these domains.
On simpler weighting alternatives: We further clarify that KDDG and DeDiER provide complementary hand-crafted weighting baselines. KDDG uses student’s final-layer true-label confidence to suppress highly confident examples in both CE and KD loss, while DeDiER uses early readouts to upweight the KD loss for examples the early readouts confidently misclassify, without a learned forecaster. In our experiments, DeDiER underperforms KD on all three datasets, with particularly large drops on DomainNet126 and OfficeHome. DeDiER focuses on robustness to spurious feature–label correlations by identifying confidently wrong early readouts, and increasing teacher supervision on those examples. This simple weighting based on early layer confidence does not transfer to our DG setting. These comparisons therefore cover simpler rule-based hard-example weighting based on both final and intermediate student prediction confidence. AMAL additionally provides a comparison with meta-learned instance-wise loss mixing. Our method achieves the highest average OOD accuracy among these alternatives on all three ResNet-10 benchmarks as presented above. We will include these baselines for the larger student ResNet-18 in the final draft of the paper.
Response to Reviewer mB2s (3/3)
(W2, Q3 continued)
Contribution of Readouts, Forecaster: We provide additional ablations on DomainNet126 with a ResNet-10 student to isolate the contributions of the forecaster and intermediate readouts to the generalization ability of the student. Here, we vary the inputs available to the forecaster while retaining the adaptive weighting framework. We observe that the forecaster achieves its best performance when using all intermediate-layer readouts, compared to using only final-layer representations or both. Our approach also outperforms a matched constant CE weight baseline that eliminates both readouts and the forecaster in the distillation setup.
| Forecaster Input | Accuracy (mean ± SD) |
|---|---|
| Constant CE weight | 52.06 ± 0.53% |
| Intermediate layers (A) | 52.98 ± 0.54% |
| Final layer (B) | 52.11 ± 0.61% |
| All layers (A + B) | 52.40 ± 0.37% |
The motivation to use intermediate readouts is that the final-layer prediction alone does not reveal how an instance was resolved within the student’s layers. Some examples are confidently correct at shallow layers, while others have low true-class confidence at intermediate layers and become confidently correct only near the final head. Using only the final layer predictions can therefore lose this signal: whether the example was resolved early or required deeper processing. Intermediate readouts expose the changes in prediction confidence across layers, giving the forecaster additional evidence about sample difficulty beyond the final-layer alone. Consistent with this observation, intermediate-only inputs achieve the highest mean OOD accuracy, while adding final-layer inputs provides no further improvement. We will include this analysis in the revised manuscript through a figure showing the fraction of examples first becoming stably correct at each readout and their mean true-class confidence across layers.
Our additional results on DeDiER provide a complementary comparison with hand-designed weighting based on intermediate readouts. On DomainNet126, it achieves 43.87%, compared with 52.01% for vanilla KD. This suggests that intermediate readouts alone do not guarantee improved generalization when their information is incorporated through a manually specified weighting rule.
W3.The computational trade-off remains unresolved. Figure 3(c) reports approximately twice the FLOPs and 1.5 times the wall time for 10,000 student optimizer steps. Without a KD comparison under a similar computational budget, the practical value of the modest and mixed gains is uncertain.
| Method | Student updates | OOD Accuracy |
|---|---|---|
| KD | 10K | 55.72% |
| Ours | 10K | 56.42% |
| KD (FLOPs-matched) | 19K | 58.23% |
| KD (wall-time matched) | 13K | 56.25% |
However, this delta improvement appears to be dataset- and model-dependent. With a ResNet-18 student, on OfficeHome (Seed 0, Art), our method achieves 54.84%, compared with 54.17% for vanilla KD at 10K updates, while FLOPs-matched KD achieves 55.08%, a narrower improvement at 2X student optimizer steps. We will make this additional observation in the Limitations section more explicit in the revised draft.
Q1: Are the instance weights detached using stop-gradient during student updates? Please clarify the gradient paths in Figure 1 and Algorithm 1.
Yes. During student updates, instance weights are detached, so the student loss does not update the forecaster. The forecaster is trained separately using detached readout outputs, so its loss does not update the student or readouts. Similarly, the auxiliary readouts get detached intermediate representations from the student and their weight updates do not alter the student backbone. We will explicitly mark these stop-gradient paths in Figure 1 and Algorithm 1 in the revision.
Q2: For text experiments, which data are used for checkpoint selection and hyperparameter tuning? Are these separate from the gold validation splits used to train the forecaster?
Hyperparameters are selected using the MRPC test split, which is separate from the MRPC validation split used to train the forecaster, and MRPC train set used to train the student backbone. QQP is the held-out OOD test set. Hyperparameters once chosen are retained for MNLI.
Response to Reviewer mB2s (1/3)
We thank the reviewer for the thoughtful and constructive assessment of our work. We appreciate their recognition of the method’s practicality and their positive assessment of the KD comparisons and readout ablations. We also value the reviewer’s careful identification of the limitations, which we address below.
W1: The gains are inconsistent across architectures. ResNet-18 underperforms KD on nine of twelve target domains and two of three dataset averages (Table 2). The ViT gain is only 0.26 percentage points, and both ViT and text results lack variability estimates (Figures 3(a)–(b)).
Q4: Can you report paired seed-level differences with uncertainty estimates and compare against KD under a similar training budget? Please also analyze why the ResNet-10 gains largely disappear for ResNet-18.
a)We will add the variance estimates to the subfigures 3a and 3b. The table below reports the mean ± standard deviation across three seeds for the ViT and text experiments.
b)The stronger RN18 student fits the training minibatch examples early in training. By 2K student updates, training minibatch accuracy reaches 99.51% on PACS and 97.70% on OfficeHome. At this point, the mean CE weight and minibatch standard deviation are 0.40001 and 0.00009 on PACS, and 0.4132 and 0.0445 on OfficeHome, respectively. In contrast, the more challenging DomainNet reaches only 76.28% training minibatch accuracy at 2K updates, with a mean CE weight mean/std of 0.4428/0.0809. However, as training progresses, CE weights become nearly uniform by 10k updates. This behaviour ultimately appears to reduce the scope for adaptive supervision. We also note that simply lowering the clipping threshold is not necessarily a solution, as the forecaster is trained over the full [0, 1] prediction range; a lower threshold drives CE weights toward zero, excessively suppressing supervised learning.
To complement this analysis, we perform an additional ablation on RN10, replacing the adaptive CE weights with a constant CE weight equal to the lower clipping threshold (w_min=0.4).
Here, as hypothesized, the constant-weight variant achieves similar performance to vanilla KD averaged across all domains. We will add these observations to the draft as a plausible explanation for the limited additional gains over vanilla KD with RN18.
c)We will add seed-wise paired results for all our main experiments in the final draft.