acceptodds
Under review as a conference paper at ICLR 2027

On the Efficacy of Intermediate Representations in Knowledge Distillation for Robust Generalization

35 views

Abstract

Domain generalization (DG) aims to learn a model that can generalize to unseen i.e. out-of-distribution (OOD) test domain. While large-capacity networks trained with sophisticated DG algorithms tend to achieve high robustness, they tend to be impractical in deployment. Typically, Knowledge distillation (KD) can alleviate this via an efficient transfer of knowledge from a robust teacher to a smaller student network. In our experiments, we observe that vanilla KD already provides strong OOD performance, often outperforming standalone DG algorithms. Motivated by this observation, we explore an adaptive distillation strategy that utilizes intermediate layer predictions in the student backbone to learn a meta network that effectively modulates the KD objective at an instance level. Our method adds no inference overhead and outperforms canonical ERM, and KD across vision, and text OOD generalization benchmarks.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
distillation, domain generalization, early exit

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mB2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes adaptive knowledge distillation for domain generalization using intermediate student predictions. A forecaster uses auxiliary logits and uncertainty statistics to predict student correctness, assigning larger supervised-loss weights to predicted difficult examples while keeping the distillation coefficient fixed. All auxiliary components are removed at inference. Experiments on three vision datasets and two text ID–OOD task pairs show improvements over vanilla KD for ResNet-10 and the reported transformer settings, but mixed results for ResNet-18.

Critical strengths

The method preserves inference efficiency. It adapts supervision using student predictions and removes all auxiliary components after training, making it practical for compact-model deployment.

The evaluation includes an informative KD baseline. Table 2 reports domain-level results over three seeds. Vanilla KD exceeds ERM and the reproduced KDDG on all six student–dataset averages, and the proposed method further improves all three ResNet-10 averages.

The ablations support the usefulness of readout signals. Adding uncertainty features improves correctness AUROC from 63.7% to 76.7% (Figure 2). Additional readout stages correlate with higher forecaster AUROC and better OOD accuracy (Figure 3(f)), although these results alone do not establish why the resulting weights improve generalization.

Critical weaknesses

The gains are inconsistent across architectures. ResNet-18 underperforms KD on nine of twelve target domains and two of three dataset averages (Table 2). The ViT gain is only 0.26 percentage points, and both ViT and text results lack variability estimates (Figures 3(a)–(b)).

The distinct contribution is insufficiently isolated. Missing controls include a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights. AMAL and DEDIER are also not evaluated, leaving the advantage over simpler and closely related approaches unclear.

The computational trade-off remains unresolved. Figure 3(c) reports approximately twice the FLOPs and 1.5 times the wall time for 10,000 student optimizer steps. Without a KD comparison under a similar computational budget, the practical value of the modest and mixed gains is uncertain.

Questions for authors

Clarification: Are the instance weights detached using stop-gradient during student updates? Please clarify the gradient paths in Figure 1 and Algorithm 1.

Clarification: For text experiments, which data are used for checkpoint selection and hyperparameter tuning? Are these separate from the gold validation splits used to train the forecaster?

Additional evidence: Can you compare against a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights? Comparisons with AMAL and DEDIER would also clarify the contribution.

Additional evidence: Can you report paired seed-level differences with uncertainty estimates and compare against KD under a similar training budget? Please also analyze why the ResNet-10 gains largely disappear for ResNet-18.

5 comments
AuthorsOpenReview ↗

Response to Reviewer mB2s (1/3)

We thank the reviewer for the thoughtful and constructive assessment of our work. We appreciate their recognition of the method’s practicality and their positive assessment of the KD comparisons and readout ablations. We also value the reviewer’s careful identification of the limitations, which we address below.


W1: The gains are inconsistent across architectures. ResNet-18 underperforms KD on nine of twelve target domains and two of three dataset averages (Table 2). The ViT gain is only 0.26 percentage points, and both ViT and text results lack variability estimates (Figures 3(a)–(b)).

Q4: Can you report paired seed-level differences with uncertainty estimates and compare against KD under a similar training budget? Please also analyze why the ResNet-10 gains largely disappear for ResNet-18.

a)We will add the variance estimates to the subfigures 3a and 3b. The table below reports the mean ± standard deviation across three seeds for the ViT and text experiments.

ExperimentKDOurs
ViT-B / DomainNet12667.84 ± 0.6468.50 ± 0.46
BERT / Paraphrase58.40 ± 4.2160.90 ± 2.92
BERT / NLI74.80 ± 0.3875.40 ± 0.03

b)The stronger RN18 student fits the training minibatch examples early in training. By 2K student updates, training minibatch accuracy reaches 99.51% on PACS and 97.70% on OfficeHome. At this point, the mean CE weight and minibatch standard deviation are 0.40001 and 0.00009 on PACS, and 0.4132 and 0.0445 on OfficeHome, respectively. In contrast, the more challenging DomainNet reaches only 76.28% training minibatch accuracy at 2K updates, with a mean CE weight mean/std of 0.4428/0.0809. However, as training progresses, CE weights become nearly uniform by 10k updates. This behaviour ultimately appears to reduce the scope for adaptive supervision. We also note that simply lowering the clipping threshold is not necessarily a solution, as the forecaster is trained over the full [0, 1] prediction range; a lower threshold drives CE weights toward zero, excessively suppressing supervised learning.

To complement this analysis, we perform an additional ablation on RN10, replacing the adaptive CE weights with a constant CE weight equal to the lower clipping threshold (w_min=0.4).

MethodAccuracy (mean ± SD)
Vanilla KD52.01 ± 0.16%
Constant CE weight52.06 ± 0.53%
Ours52.98 ± 0.54%

Here, as hypothesized, the constant-weight variant achieves similar performance to vanilla KD averaged across all domains. We will add these observations to the draft as a plausible explanation for the limited additional gains over vanilla KD with RN18.

c)We will add seed-wise paired results for all our main experiments in the final draft.

AuthorsOpenReview ↗

Response to Reviewer mB2s (2/3)

W2: The distinct contribution is insufficiently isolated. Missing controls include a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights. AMAL and DEDIER are also not evaluated, leaving the advantage over simpler and closely related approaches unclear.

Q3: Can you compare against a final-layer-only forecaster of comparable capacity, simple confidence weighting, and matched constant CE weights? Comparisons with AMAL and DEDIER would also clarify the contribution.

We present domain-wise OOD performance (averaged over 3 seeds) of DeDiER and AMAL against the existing results from the paper on DomainNet, PACS and OfficeHome for ResNet-10 below. Similar to our main results, we sweep method-specific hyperparameters and LR on DomainNet126 and transfer unchanged to PACS and OfficeHome.

DomainNet126

MethodClipartPaintingRealSketchAverage
ERM51.27 ± 0.9342.30 ± 0.4748.25 ± 0.2745.80 ± 1.2246.90 ± 0.47
KDDG54.90 ± 1.2446.63 ± 0.4952.22 ± 0.3448.89 ± 0.8250.66 ± 0.71
KD56.00 ± 0.2147.70 ± 0.3753.67 ± 0.2750.65 ± 0.4552.01 ± 0.16
AMAL52.34 ± 1.5646.34 ± 0.8653.63 ± 0.1449.72 ± 1.3450.51 ± 0.21
DeDiER43.63 ± 0.4844.19 ± 0.4141.92 ± 0.4045.73 ± 0.8243.87 ± 0.29
Ours56.93 ± 0.6148.53 ± 1.0253.73 ± 0.5752.73 ± 1.4552.98 ± 0.54

PACS

MethodArtCartoonPhotoSketchAverage
ERM60.18 ± 1.4670.40 ± 2.2179.92 ± 2.2567.93 ± 0.9669.61 ± 0.34
KDDG62.17 ± 0.5866.77 ± 0.5380.84 ± 0.1366.47 ± 3.2469.06 ± 0.82
KD67.20 ± 2.0667.73 ± 2.8282.81 ± 0.3169.33 ± 1.2571.77 ± 0.41
AMAL67.40 ± 1.1567.13 ± 2.4082.58 ± 0.3568.44 ± 0.5171.39 ± 0.71
DeDiER58.74 ± 1.4964.59 ± 1.2575.95 ± 1.5561.59 ± 3.6865.21 ± 1.19
Ours70.31 ± 2.0869.35 ± 2.3182.41 ± 1.5368.76 ± 0.7472.71 ± 0.79

OfficeHome

MethodArtClipartProductReal WorldAverage
ERM30.11 ± 1.0631.37 ± 1.7649.68 ± 0.6348.63 ± 0.6539.95 ± 0.55
KDDG31.86 ± 1.1333.62 ± 1.1653.25 ± 0.8952.75 ± 1.4942.87 ± 1.16
KD37.95 ± 1.5737.24 ± 1.3158.48 ± 0.5958.69 ± 0.8048.09 ± 0.66
AMAL37.08 ± 0.6236.74 ± 0.6759.11 ± 0.5859.14 ± 0.9148.02 ± 0.38
DeDiER28.73 ± 0.5731.60 ± 1.5950.50 ± 0.8555.09 ± 0.9941.48 ± 0.28
Ours39.65 ± 0.7338.78 ± 0.8158.62 ± 0.9957.90 ± 0.5948.74 ± 0.63

Here, AMAL serves as a representative meta-learning-based adaptive instance-specific weighting algorithm for mixing KD and CE loss at an instance level. In this approach, the instance-wise mixing weights are learned using a meta-objective on a held-out in-domain validation set. On DomainNet, ours achieves the best average (52.98%), outperforming KD by 0.97 points and AMAL by 2.47 points. On PACS we achieve the best average (72.71%), outperforming KD by 0.94 points and AMAL by 1.32 points. On OfficeHome, our approach again achieves the best average (48.74%), but the gain over KD is smaller (+0.65 points). Notably, AMAL achieves the best accuracy on Product (59.11%) and Real World (59.14%), outperforming both KD and ours on these domains.

On simpler weighting alternatives: We further clarify that KDDG and DeDiER provide complementary hand-crafted weighting baselines. KDDG uses student’s final-layer true-label confidence to suppress highly confident examples in both CE and KD loss, while DeDiER uses early readouts to upweight the KD loss for examples the early readouts confidently misclassify, without a learned forecaster. In our experiments, DeDiER underperforms KD on all three datasets, with particularly large drops on DomainNet126 and OfficeHome. DeDiER focuses on robustness to spurious feature–label correlations by identifying confidently wrong early readouts, and increasing teacher supervision on those examples. This simple weighting based on early layer confidence does not transfer to our DG setting. These comparisons therefore cover simpler rule-based hard-example weighting based on both final and intermediate student prediction confidence. AMAL additionally provides a comparison with meta-learned instance-wise loss mixing. Our method achieves the highest average OOD accuracy among these alternatives on all three ResNet-10 benchmarks as presented above. We will include these baselines for the larger student ResNet-18 in the final draft of the paper.

AuthorsOpenReview ↗

Response to Reviewer mB2s (3/3)

(W2, Q3 continued)

Contribution of Readouts, Forecaster: We provide additional ablations on DomainNet126 with a ResNet-10 student to isolate the contributions of the forecaster and intermediate readouts to the generalization ability of the student. Here, we vary the inputs available to the forecaster while retaining the adaptive weighting framework. We observe that the forecaster achieves its best performance when using all intermediate-layer readouts, compared to using only final-layer representations or both. Our approach also outperforms a matched constant CE weight baseline that eliminates both readouts and the forecaster in the distillation setup.

Forecaster InputAccuracy (mean ± SD)
Constant CE weight52.06 ± 0.53%
Intermediate layers (A)52.98 ± 0.54%
Final layer (B)52.11 ± 0.61%
All layers (A + B)52.40 ± 0.37%

The motivation to use intermediate readouts is that the final-layer prediction alone does not reveal how an instance was resolved within the student’s layers. Some examples are confidently correct at shallow layers, while others have low true-class confidence at intermediate layers and become confidently correct only near the final head. Using only the final layer predictions can therefore lose this signal: whether the example was resolved early or required deeper processing. Intermediate readouts expose the changes in prediction confidence across layers, giving the forecaster additional evidence about sample difficulty beyond the final-layer alone. Consistent with this observation, intermediate-only inputs achieve the highest mean OOD accuracy, while adding final-layer inputs provides no further improvement. We will include this analysis in the revised manuscript through a figure showing the fraction of examples first becoming stably correct at each readout and their mean true-class confidence across layers.

Our additional results on DeDiER provide a complementary comparison with hand-designed weighting based on intermediate readouts. On DomainNet126, it achieves 43.87%, compared with 52.01% for vanilla KD. This suggests that intermediate readouts alone do not guarantee improved generalization when their information is incorporated through a manually specified weighting rule.


W3.The computational trade-off remains unresolved. Figure 3(c) reports approximately twice the FLOPs and 1.5 times the wall time for 10,000 student optimizer steps. Without a KD comparison under a similar computational budget, the practical value of the modest and mixed gains is uncertain.

MethodStudent updatesOOD Accuracy
KD10K55.72%
Ours10K56.42%
KD (FLOPs-matched)19K58.23%
KD (wall-time matched)13K56.25%

However, this delta improvement appears to be dataset- and model-dependent. With a ResNet-18 student, on OfficeHome (Seed 0, Art), our method achieves 54.84%, compared with 54.17% for vanilla KD at 10K updates, while FLOPs-matched KD achieves 55.08%, a narrower improvement at 2X student optimizer steps. We will make this additional observation in the Limitations section more explicit in the revised draft.


Q1: Are the instance weights detached using stop-gradient during student updates? Please clarify the gradient paths in Figure 1 and Algorithm 1.

Yes. During student updates, instance weights are detached, so the student loss does not update the forecaster. The forecaster is trained separately using detached readout outputs, so its loss does not update the student or readouts. Similarly, the auxiliary readouts get detached intermediate representations from the student and their weight updates do not alter the student backbone. We will explicitly mark these stop-gradient paths in Figure 1 and Algorithm 1 in the revision.


Q2: For text experiments, which data are used for checkpoint selection and hyperparameter tuning? Are these separate from the gold validation splits used to train the forecaster?

Hyperparameters are selected using the MRPC test split, which is separate from the MRPC validation split used to train the forecaster, and MRPC train set used to train the student backbone. QQP is the held-out OOD test set. Hyperparameters once chosen are retained for MNLI.

Reviewer_7tAQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an adaptive knowledge distillation method for improving the out-of-domain generalization of compact models. Auxiliary classifiers on intermediate student layers provide signals to a forecaster, which estimates the correctness of the student’s final prediction and assigns per-example weights to the supervised loss; the distillation loss remains unchanged. The method is evaluated against ERM, vanilla KD, and KDDG on vision and text OOD benchmarks.

Critical strengths

1.The paper addresses a relevant problem: improving the OOD generalization of compact models without adding inference-time overhead. Its forecaster uses intermediate student predictions and uncertainty estimates to adjust the supervised loss for individual examples, and the auxiliary components are removed after training.

2.The evaluation spans vision and text tasks and several student architectures. The paper also analyzes uncertainty signals, readout depth, loss-weight clipping, and training cost, providing useful evidence about the method’s behavior.

Critical weaknesses

1.The contribution over existing adaptive distillation methods is not sufficiently established. The paper discusses related approaches such as AMAL and DEDIER, but does not compare against them or a simpler instance-wise loss-weighting baseline. Thus, the experiments do not show whether the gains come from the proposed forecaster and intermediate readouts, or from adaptive sample weighting more generally.

2.The comparison with domain generalization methods is limited. The main experiments include ERM and KDDG, but do not evaluate other DG-oriented distillation approaches discussed in the paper, such as DAFT or methods using multimodal or domain-specific teachers. This leaves the method’s performance relative to a broader range of DG and DG-KD approaches unclear.

Questions for authors

1.How does the proposed method differ from AMAL, DEDIER, and a simpler instance-wise loss-weighting baseline?

2.The current ablations do not isolate the contributions of the forecaster and intermediate readouts to the final OOD performance.

3.Other DG-oriented distillation methods discussed in the paper, such as DAFT and methods using multimodal or domain-specific teachers, are not included in the main comparisons.

4.Table 2 shows that the ResNet-18 student performs below vanilla KD on the DomainNet and OfficeHome averages, but the paper does not discuss these results.

5.The discussion of prior DG-oriented distillation methods is incomplete. [1][2]. [1] Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation-CVPR2026 [2] DomED: Redesigning Ensemble Distillation for Domain Generalization-ICML2026

6 comments
AuthorsOpenReview ↗

Response to Reviewer 7tAQ (1/3)

We thank the reviewer for the thoughtful and constructive feedback. We appreciate their recognition of the relevance of our problem setting, and the breadth of our evaluation. We address their questions below.


W1.How does the proposed method differ from AMAL, DEDIER, and a simpler instance-wise loss-weighting baseline?

We present domain-wise OOD performance (avg. over 3 seeds) of DeDiER and AMAL on DomainNet, PACS and OfficeHome for ResNet-10 below. Similar to our main results, we sweep method-specific hyperparameters and LR on DomainNet126 and transfer unchanged to PACS and OfficeHome.

DomainNet126

MethodClipPaint.RealSketchAvg
ERM51.27 ± 0.9342.30 ± 0.4748.25 ± 0.2745.80 ± 1.2246.90 ± 0.47
KDDG54.90 ± 1.2446.63 ± 0.4952.22 ± 0.3448.89 ± 0.8250.66 ± 0.71
KD56.00 ± 0.2147.70 ± 0.3753.67 ± 0.2750.65 ± 0.4552.01 ± 0.16
AMAL52.34 ± 1.5646.34 ± 0.8653.63 ± 0.1449.72 ± 1.3450.51 ± 0.21
DeDiER43.63 ± 0.4844.19 ± 0.4141.92 ± 0.4045.73 ± 0.8243.87 ± 0.29
Ours56.93 ± 0.6148.53 ± 1.0253.73 ± 0.5752.73 ± 1.4552.98 ± 0.54

PACS

MethodArtCartoonPhotoSketchAvg
ERM60.18 ± 1.4670.40 ± 2.2179.92 ± 2.2567.93 ± 0.9669.61 ± 0.34
KDDG62.17 ± 0.5866.77 ± 0.5380.84 ± 0.1366.47 ± 3.2469.06 ± 0.82
KD67.20 ± 2.0667.73 ± 2.8282.81 ± 0.3169.33 ± 1.2571.77 ± 0.41
AMAL67.40 ± 1.1567.13 ± 2.4082.58 ± 0.3568.44 ± 0.5171.39 ± 0.71
DeDiER58.74 ± 1.4964.59 ± 1.2575.95 ± 1.5561.59 ± 3.6865.21 ± 1.19
Ours70.31 ± 2.0869.35 ± 2.3182.41 ± 1.5368.76 ± 0.7472.71 ± 0.79

OfficeHome

MethodArtClip.Prod.RealAvg
ERM30.11 ± 1.0631.37 ± 1.7649.68 ± 0.6348.63 ± 0.6539.95 ± 0.55
KDDG31.86 ± 1.1333.62 ± 1.1653.25 ± 0.8952.75 ± 1.4942.87 ± 1.16
KD37.95 ± 1.5737.24 ± 1.3158.48 ± 0.5958.69 ± 0.8048.09 ± 0.66
AMAL37.08 ± 0.6236.74 ± 0.6759.11 ± 0.5859.14 ± 0.9148.02 ± 0.38
DeDiER28.73 ± 0.5731.60 ± 1.5950.50 ± 0.8555.09 ± 0.9941.48 ± 0.28
Ours39.65 ± 0.7338.78 ± 0.8158.62 ± 0.9957.90 ± 0.5948.74 ± 0.63

Here, AMAL serves as a meta-learning-based adaptive weighting algorithm for mixing KD and CE loss at an instance level. In this approach, the mixing weights are learned using a meta-objective on a held-out in-domain val. set. On DomainNet, ours achieves the best average (52.98%), outperforming KD by 0.97 points and AMAL by 2.47 points. On PACS we achieve the best average (72.71%), outperforming KD by 0.94 points and AMAL by 1.32 points. On OfficeHome, our approach again achieves the best average (48.74%), but the gain over KD is smaller (+0.65 points). Notably, AMAL achieves the best accuracy on Product (59.11%) and Real World (59.14%), outperforming both KD and ours on these domains.

We further clarify that KDDG and DeDiER provide complementary hand-crafted weighting baselines. KDDG uses the student's final-layer logits to modulate the distillation objective, whereas DeDiER uses early readout representations to construct instance-specific weights. In our experiments, DeDiER underperforms KD on all three datasets, with particularly large drops on DomainNet126 and OfficeHome. DeDiER focuses on robustness to spurious feature–label correlations by identifying confidently wrong early readouts, and increasing teacher supervision on those examples. This simple weighting based on early layer confidence does not transfer to our DG setting. These observations suggest that instance-wise loss weighting alone does not necessarily improve KD for DG. In conclusion, the above observations show that instance-wise weighting alone can be insufficient or dataset dependent, and weighting strategy also plays a role. Our method learns the weighting strategy through a neural forecaster trained to predict student correctness.

AuthorsOpenReview ↗

Response to Reviewer 7tAQ (2/3)

(W1 continued)

On the concern for simpler instance-level weighting baselines, our evaluation already includes complementary baselines that examine whether simpler weighting strategies can achieve similar generalization. KDDG uses student’s final-layer true-label confidence to suppress highly confident examples in both CE and KD loss, while DeDiER uses early readouts to upweight the KD loss for examples the early readouts confidently misclassify, without a learned forecaster. These comparisons therefore cover simpler rule-based hard-example weighting based on both final and intermediate student prediction confidence. AMAL additionally provides a comparison with meta-learned instance-wise loss mixing. Our method achieves the highest average OOD accuracy among these alternatives on all three ResNet-10 benchmarks as presented above. We will include quantitative figures for the larger student ResNet-18 in the final draft of the paper.


W2.The current ablations do not isolate the contributions of the forecaster and intermediate readouts to the final OOD performance.

We provide additional ablations on DomainNet126 with a ResNet-10 student to isolate the contributions of the forecaster and intermediate readouts to the generalization ability of the student. Here, we vary the inputs available to the forecaster while retaining the adaptive weighting framework. We observe that the forecaster achieves its best performance when using only intermediate-layer readouts, compared to using only final-layer representations or both.

Forecaster inputsAccuracy (mean ± SD)
Intermediate layers (A)52.98 ± 0.54%
Final layer (B)52.11 ± 0.61%
All layers (A + B)52.40 ± 0.37%

The motivation to use intermediate readouts is that the final-layer prediction alone does not reveal how an instance was resolved within the student’s layers. Some examples are confidently correct at shallow layers, while others have low true-class confidence at intermediate layers and become correctly classified only near the final head. Using only the final layer predictions can therefore lose this signal: whether the example was resolved early or required deeper processing. Intermediate readouts expose the changes in prediction confidence across layers, giving the forecaster additional evidence about sample difficulty beyond the final-layer alone. Consistent with this observation, our experiments show that intermediate-only inputs achieve the highest mean OOD accuracy, while adding final-layer inputs provides no further improvement. We will include this analysis in the revised manuscript through a figure showing the fraction of examples first becoming stably correct at each readout and their mean true-class confidence across layers.

Our additional results on DeDiER above provide a complementary comparison with hand-designed weighting based on intermediate readouts. On DomainNet126, it achieves 43.87%, compared with 52.01% for vanilla KD. This suggests that intermediate readouts alone do not guarantee improved generalization when their information is incorporated through a manually specified weighting function.


W3.Other DG-oriented distillation methods discussed in the paper, such as DAFT and methods using multimodal or domain-specific teachers, are not included in the main comparisons.

DAFT, RISE and BOLD change teacher supervision and the distillation protocol through construction of adversarially fine-tuned teachers, language-guided regularization loss, and using multiple domain-specific teacher experts, respectively. Our contribution instead concerns student-side intervention for instance-level weighting in KD for DG with a fixed teacher logit distribution. Our baselines share the same supervision to isolate our contribution. We already discuss specialized teacher training and additional teacher signals in “Limitations and Future Work.” These approaches are complementary extensions, rather than direct baselines for instance-level learning in KD for DG.

AuthorsOpenReview ↗

Response to Reviewer 7tAQ (3/3)

W4.Table 2 shows that the ResNet-18 student performs below vanilla KD on the DomainNet and OfficeHome averages, but the paper does not discuss these results.

The stronger RN18 student fits the training minibatch examples early in training. By 2K student updates, training minibatch accuracy reaches 99.51% on PACS and 97.70% on OfficeHome. At this point, the mean CE weight and minibatch standard deviation are 0.40001 and 0.00009 on PACS, and 0.4132 and 0.0445 on OfficeHome, respectively. In contrast, the more challenging DomainNet reaches only 76.28% training minibatch accuracy at 2K updates, with a mean CE weight mean/std of 0.4428/0.0809. However, as training progresses, forecaster predicted CE weights become nearly uniform by 10K updates. This behaviour ultimately appears to reduce the scope for adaptive supervision. We note that simply lowering the clipping threshold is not necessarily a solution, as the forecaster is trained over the full [0, 1] prediction range; a lower threshold drives CE weights toward zero with minimal variance within a minibatch, excessively suppressing supervised learning.

To complement this analysis, we perform an additional ablation on RN10, replacing the adaptive CE weights with a constant CE weight equal to the lower clipping threshold ((w_min=0.4)).

MethodAccuracy (mean ± SD)
Vanilla KD52.01 ± 0.16%
Constant CE weight52.06 ± 0.53%
Ours52.98 ± 0.54%

Here, as hypothesized, the constant-weight variant achieves similar performance to vanilla KD averaged across all held-out domains. We will add these observations to the draft as a plausible explanation for the limited additional gains over vanilla KD with RN18.


W5.The discussion of prior DG-oriented distillation methods is incomplete. [1][2]. [1] Generalizable Knowledge Distillation from Vision Foundation Models for Semantic Segmentation-CVPR2026 [2] DomED: Redesigning Ensemble Distillation for Domain Generalization-ICML2026

We thank the reviewer for pointing out these relevant works. We will incorporate both papers into the Related Work section of the revised manuscript and discuss their contributions to DG, and convey their differences from our approach.


Reviewer_jcz2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an adaptive knowledge distillation framework for domain generalization. The key idea is to use intermediate-layer predictions from the student network to estimate the difficulty of each training example. The authors attach auxiliary classifiers to intermediate student features and use their logits and confidence statistics to train a lightweight forecaster that predicts whether the student's final prediction is correct. The estimated correctness is then used to assign instance-dependent weights to the supervised classification loss, while the distillation loss retains a fixed coefficient. The auxiliary classifiers and forecaster are discarded at inference time. The method is evaluated on three vision domain generalization benchmarks (DomainNet, PACS, and OfficeHome) and two text classification settings based on GLUE-X. The results show improvements over vanilla knowledge distillation in some settings, with no additional inference-time cost. However, the gains are relatively small and inconsistent for the larger student models, and the evaluation does not establish that the proposed forecaster is the key reason for the observed improvements.

Critical strengths
  1. The paper explores a reasonable direction. Intermediate predictions can provide information about how a model processes an example, and using this information to guide training is a sensible idea. The framework is also relatively easy to integrate into conventional teacher–student distillation.

  2. The method does not introduce inference-time overhead. The auxiliary classifiers and forecaster are used only during training, so the final student architecture remains unchanged. This is a practical advantage over approaches that require additional components at deployment.

  3. The paper evaluates both vision and text tasks. The experiments cover multiple student architectures and several domain-shift settings. The ablations on intermediate readouts, confidence statistics, and the training schedule provide some useful insight into the proposed design.

Critical weaknesses
  1. The main methodological contribution is not sufficiently differentiated from existing instance-wise loss reweighting approaches. The proposed forecaster predicts sample correctness from intermediate-layer logits and confidence statistics, then converts this estimate into a supervised-loss weight. This is closely related to existing work on adaptive sample weighting, meta-learned loss mixing, and early-readout-based distillation. The paper discusses DEDIER and AMAL, but does not directly compare against these closely related approaches. In particular, the main comparison with a hand-crafted weighting method is not enough to establish that the learned forecaster is a meaningful advance over existing alternatives. The current evidence supports the usefulness of adaptive weighting more clearly than it supports a distinct contribution from the proposed design.

  2. The method does not actually adapt the distillation loss, despite how the contribution is framed. In Equation (8), the student objective is w~iLCE[i]+λKDLKD[i].\widetilde{w}_i\mathcal{L}_{\mathrm{CE}}[i]+\lambda_{\mathrm{KD}}\mathcal{L}_{\mathrm{KD}}[i]. The forecaster only reweights the ground-truth cross-entropy term; the distillation term receives a fixed coefficient for every sample. Thus, the method learns to adjust the balance of hard-label supervision at the instance level, rather than deciding when or how strongly the student should trust the teacher. This distinction matters because the paper motivates the method partly through concerns about teacher-specific biases and the limitations of uniform knowledge transfer. The proposed objective does not directly address those issues. A more accurate characterization would be adaptive supervised-loss weighting on top of standard KD.

  3. The teacher setup does not adequately test the paper's central motivation. The introduction argues that a robust teacher can transfer OOD generalization to a smaller student, but the experiments primarily use teachers trained with ERM. The resulting comparisons therefore establish whether the proposed weighting scheme can improve conventional KD, not whether it helps distill robustness from a teacher trained with a specialized domain generalization or adversarial training method. The authors acknowledge this limitation and leave robust-teacher distillation for future work. This is not a minor missing experiment: it leaves a gap between the motivating problem and the setting actually evaluated.

  4. The baseline selection is too limited to support the paper's claims about the proposed mechanism. The main comparisons are against ERM, vanilla KD, and KDDG. This leaves several straightforward alternatives untested, including focal-loss-style hard-example weighting, simple weighting based on final-layer confidence, weighting based on intermediate confidence without a learned forecaster, and closely related learned reweighting methods. These are particularly important controls because the proposed method's main effect is to increase the contribution of examples that the forecaster considers likely to be misclassified. Without such comparisons, it is difficult to determine whether the intermediate readouts and meta-network provide a meaningful advantage over simpler ways of emphasizing difficult examples.

Questions for authors

See weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer jcz2 (1/2)

We thank the reviewer for the thoughtful assessment. We appreciate the positive assessment of the method's practicality. We address each of the major concerns below.


W1: The main methodological contribution is not sufficiently differentiated from existing instance-wise loss reweighting approaches. The proposed forecaster predicts sample correctness from intermediate-layer logits and confidence statistics, then converts this estimate into a supervised-loss weight. This is closely related to existing work on adaptive sample weighting, meta-learned loss mixing, and early-readout-based distillation. The paper discusses DeDiER and AMAL, but does not directly compare against these closely related approaches. In particular, the main comparison with a hand-crafted weighting method is not enough to establish that the learned forecaster is a meaningful advance over existing alternatives. The current evidence supports the usefulness of adaptive weighting more clearly than it supports a distinct contribution from the proposed design.

We present OOD evaluations (averaged over 3 seeds) of DeDiER and AMAL on DomainNet, PACS and OfficeHome for ResNet-10 below. Similar to our main table results, we sweep method-specific hyperparameters and LR on DomainNet126 and transfer them to PACS and OfficeHome.

DomainNet126

MethodClipartPaintingRealSketchAverage
ERM51.27 ± 0.9342.30 ± 0.4748.25 ± 0.2745.80 ± 1.2246.90 ± 0.47
KDDG54.90 ± 1.2446.63 ± 0.4952.22 ± 0.3448.89 ± 0.8250.66 ± 0.71
KD56.00 ± 0.2147.70 ± 0.3753.67 ± 0.2750.65 ± 0.4552.01 ± 0.16
AMAL52.34 ± 1.5646.34 ± 0.8653.63 ± 0.1449.72 ± 1.3450.51 ± 0.21
DeDiER43.63 ± 0.4844.19 ± 0.4141.92 ± 0.4045.73 ± 0.8243.87 ± 0.29
Ours56.93 ± 0.6148.53 ± 1.0253.73 ± 0.5752.73 ± 1.4552.98 ± 0.54

PACS

MethodArtCartoonPhotoSketchAverage
ERM60.18 ± 1.4670.40 ± 2.2179.92 ± 2.2567.93 ± 0.9669.61 ± 0.34
KDDG62.17 ± 0.5866.77 ± 0.5380.84 ± 0.1366.47 ± 3.2469.06 ± 0.82
KD67.20 ± 2.0667.73 ± 2.8282.81 ± 0.3169.33 ± 1.2571.77 ± 0.41
AMAL67.40 ± 1.1567.13 ± 2.4082.58 ± 0.3568.44 ± 0.5171.39 ± 0.71
DeDiER58.74 ± 1.4964.59 ± 1.2575.95 ± 1.5561.59 ± 3.6865.21 ± 1.19
Ours70.31 ± 2.0869.35 ± 2.3182.41 ± 1.5368.76 ± 0.7472.71 ± 0.79

OfficeHome

MethodArtClipartProductReal WorldAverage
ERM30.11 ± 1.0631.37 ± 1.7649.68 ± 0.6348.63 ± 0.6539.95 ± 0.55
KDDG31.86 ± 1.1333.62 ± 1.1653.25 ± 0.8952.75 ± 1.4942.87 ± 1.16
KD37.95 ± 1.5737.24 ± 1.3158.48 ± 0.5958.69 ± 0.8048.09 ± 0.66
AMAL37.08 ± 0.6236.74 ± 0.6759.11 ± 0.5859.14 ± 0.9148.02 ± 0.38
DeDiER28.73 ± 0.5731.60 ± 1.5950.50 ± 0.8555.09 ± 0.9941.48 ± 0.28
Ours39.65 ± 0.7338.78 ± 0.8158.62 ± 0.9957.90 ± 0.5948.74 ± 0.63

Here, AMAL serves as a representative meta-learning-based adaptive instance-specific weighting algorithm for mixing KD and CE loss at an instance level. In this approach, the instance-wise mixing weights are learned using a meta-objective on a held-out in-domain validation set. On DomainNet, ours achieves the best average (52.98%), outperforming KD by 0.97 points and AMAL by 2.47 points. On PACS we achieve the best average (72.71%), outperforming KD by 0.94 points and AMAL by 1.32 points. On OfficeHome, our approach again achieves the best average (48.74%), but the gain over KD is smaller (+0.65 points). Notably, AMAL achieves the best accuracy on Product (59.11%) and Real World (59.14%), outperforming both KD and our approach on these domains.

We further clarify that KDDG and DeDiER provide complementary hand-crafted weighting baselines. KDDG uses the student's final-layer logits to modulate the distillation objective, whereas DeDiER uses early readout representations to construct instance-specific weights. In our experiments, DeDiER underperforms KD on all three datasets, with particularly large drops on DomainNet126 and OfficeHome. DeDiER, as a method, focuses on robustness to spurious feature–label correlations by identifying confidently wrong early readouts, and increasing teacher supervision on those examples. This simple weighting based on early layer confidence does not transfer to our DG setting. These observations suggest that confidence-based instance-level weighting alone can be insufficient or dataset/task dependent, and weighting strategy also plays a role. Our method learns the weighting strategy through a neural forecaster trained to predict student correctness.


AuthorsOpenReview ↗

Response to Reviewer jcz2 (2/2)


W2: The method does not actually adapt the distillation loss, despite how the contribution is framed. In Equation (8) The forecaster only reweights the ground-truth cross-entropy term; the distillation term receives a fixed coefficient for every sample. Thus, the method learns to adjust the balance of hard-label supervision at the instance level, rather than deciding when or how strongly the student should trust the teacher. This distinction matters because the paper motivates the method partly through concerns about teacher-specific biases and the limitations of uniform knowledge transfer. The proposed objective does not directly address those issues. A more accurate characterization would be adaptive supervised-loss weighting on top of standard KD.

We clarify that our method adaptively weights the supervised CE term while keeping the KD coefficient fixed. The forecaster adjusts the relative contributions of supervised and distillation losses through instance-level weighting of the supervised term and does not model teacher reliability explicitly. We agree that this distinction is important and will revise the abstract and introduction to correctly pose our contribution.


W3: The teacher setup does not adequately test the paper's central motivation. The introduction argues that a robust teacher can transfer OOD generalization to a smaller student, but the experiments primarily use teachers trained with ERM. The resulting comparisons therefore establish whether the proposed weighting scheme can improve conventional KD, not whether it helps distill robustness from a teacher trained with a specialized domain generalization or adversarial training method. The authors acknowledge this limitation and leave robust-teacher distillation for future work. This is not a minor missing experiment: it leaves a gap between the motivating problem and the setting actually evaluated.

DomainBed [1] shows that a correctly implemented ERM is a strong DG algorithm and is competitive against specialized DG algorithms. More importantly, OOD generalization depends on factors beyond the training objective, including model architecture and capacity [2]. These points motivate our use of larger-capacity, ERM-trained ResNet-50 and ViT-L/16 as sufficiently robust teachers in our experiments. While we already explicitly identify distillation from teachers trained on specialized DG algorithms as an extension in the paper's “Limitations and Future Work” section, we will adjust the write-up to more accurately reflect the scope of our experimental setup.

[1] Ishaan Gulrajani and David Lopez-Paz In Search of Lost Domain Generalization https://arxiv.org/abs/2007.01434

[2] Naganuma et al., An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration https://arxiv.org/abs/2307.08187v4


W4: The baseline selection is too limited to support the paper's claims about the proposed mechanism. The main comparisons are against ERM, vanilla KD, and KDDG. This leaves several straightforward alternatives untested, including focal-loss-style hard-example weighting, simple weighting based on final-layer confidence, weighting based on intermediate confidence without a learned forecaster, and closely related learned reweighting methods. These are particularly important controls because the proposed method's main effect is to increase the contribution of examples that the forecaster considers likely to be misclassified. Without such comparisons, it is difficult to determine whether the intermediate readouts and meta-network provide a meaningful advantage over simpler ways of emphasizing difficult examples.

Focal loss emphasizes difficult examples by downweighting the CE contribution of confidently correct predictions. Our evaluation already includes complementary baselines that examine whether simpler weighting strategies can achieve similar generalization. KDDG uses student’s final-layer true-label confidence to suppress highly confident examples in both CE and KD loss, while DeDiER uses early readouts to upweight the KD loss for examples the early readouts confidently misclassify, without a learned forecaster. These comparisons therefore cover simpler rule-based hard-example weighting based on both final and intermediate student prediction confidence. Although their weighting functions differ from focal loss, they directly address the broader question of whether emphasizing difficult examples without our learned forecaster is sufficient. AMAL additionally provides a comparison with meta-learned instance-wise loss mixing. Our method achieves the highest average OOD accuracy among these alternatives on all three ResNet-10 benchmarks as presented above.


Dear Reviewer,

With the author response deadline approaching, we wanted to ask whether you have any remaining questions or concerns. We’d be happy to clarify any points while the discussion is still open.

If our responses have addressed your concerns, we would appreciate it if you could let us know whether you would consider raising your score.

Thank you again for your time and thoughtful feedback.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Genereal Response (Summary of Contributions and Rebuttal Phase Discussion)

General Response

We thank the reviewers for their constructive feedback.

Strengths highlighted by the reviewers

Reviewer jcz2 found that the paper "explores a reasonable direction" and described using intermediate predictions to guide training as "a sensible idea". The reviewer noted that the method is "relatively easy to integrate into conventional teacher–student distillation" and that removing training-time components is "a practical advantage over approaches that require additional components at deployment".

Reviewer 7tAQ recognised that the paper "addresses a relevant problem: improving the OOD generalization of compact models without adding inference-time overhead". The reviewer valued that the evaluation "spans vision and text tasks and several student architectures", with analyses "providing useful evidence about the method's behavior"

Reviewer mB2s noted that the method "preserves inference efficiency" and removes all auxiliary components after training, "making it practical for compact-model deployment". The reviewer found that "the evaluation includes an informative KD baseline" and that "the ablations support the usefulness of readout signals".

Importantly, reviewer 7tAQ has acknowledged that our responses and revisions to the manuscript reasonably addressed their main concerns and raised their rating from Weak Reject (2) to Weak Accept (3).


The revised PDF (changes in blue) includes our additional contributions and addresses reviewers concerns as follows.

1. Related adaptive KD baselines (all reviewers)

We added AMAL, which meta-learns per-instance CE/KD mixing weights, and DeDiER, which upweights KD on examples that early readouts confidently misclassify (Table 1, ResNet-10)

MethodDomainNet126PACSOfficeHome
KD52.0171.7748.09
AMAL50.5171.3948.02
DeDiER43.8765.2141.48
Ours52.9872.7148.74

Above comparisons show that the gains are not due to adaptive weighting alone and the choice of instance-level weighting signal matters. See Sec. 7 of the revised PDF.

2. Source of the gain (all reviewers)

Table 2 showcases ablations to identify components that drive our gain over vanilla KD. On DomainNet, a constant CE weight (no readouts, no forecaster) results in 52.06% OOD accuracy, about the same as KD (52.01%). A forecaster using only the final layer confidence gives 52.11%. A forecaster using intermediate readouts results in 52.98%, biggest improvement over KD. Final logits do not show how a sample was resolved inside the student. Some samples are correct from shallow layers, while others stay low-confidence until the final head. Fig. 3 shows that intermediate readouts expose this difficulty across depth, and the learned forecaster turns it into instance weights. DeDiER uses the same readouts with a fixed rule and falls below KD, so the learned mapping matters (Sec. 4.1, Sec. 7)

3. Improvement over KD reduces on Larger Student (mB2s, 7tAQ)

Sec. 7 now presents an analysis of these findings. ResNet-18 fits the training data early: by 2K steps, minibatch accuracy is 99.51% on PACS and 97.70% on OfficeHome. As a result, the forecaster's correctness targets become highly skewed, and w~\tilde w collapses to wmin⁡w_{\min}. The method then reduces to KD, and avg. gain (PACS) stays within 0.5 points of KD.

4. Framing of “adaptive” in our approach (jcz2)

We now state clearly that only the CE term is reweighted per instance and that λKD\lambda_{KD} stays fixed in the student’s training objective in the revised draft.

5. Cost and Scope (mB2s, 7tAQ, jcz2)

Section 9 now reports the training cost: 1.93×1.93\times FLOPs and 1.48×1.48\times wall time, with no change in inference. It also includes the budget-matched KD comparison. DomainBed[1] shows that well-tuned ERM is competitive with specialized DG algorithms, and OOD robustness also grows with model capacity [2]. We therefore use larger ERM-trained teachers, ResNet-50 and ViT-L/16. DG-KD methods such as DAFT (adversarially fine-tuned teachers), language-guided distillation (RISE) and DomED (domain-specific teacher ensembles) change the teacher or the source of supervision. Our method intervenes only on the student side, with fixed teacher logits. All baselines share the same teacher, which isolates our contribution, and these methods remain complementary to ours (Sec. 2, Sec. 9). Figure 1 and Algorithm 1 (App. B) now mark stop-gradients, for forecaster and student training and Sec. 6 clarifies the text dataset splits.

For the camera-ready version, we will add AMAL and DeDiER results for ResNet-18, and seed-wise results for all datasets to the appendix.

[1] Ishaan Gulrajani and David Lopez-Paz In Search of Lost Domain Generalization

[2] Naganuma et al., An Empirical Study of Pre-trained Model Selection for Out-of-Distribution Generalization and Calibration

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.