acceptodds
Under review as a conference paper at ICLR 2027

Self-Ensemble Post Learning for Noisy Domain Generalization

Abstract

Distribution shift and label noise remain major challenges to model robustness. In domain generalization (DG), noisy source labels can impair performance on unseen domains. We observe that intermediate representations retain discriminative information and attend to different image regions. Based on these observations, we propose Self-Ensemble Post Learning (SEPL) to improve the robustness of existing DG methods. SEPL consists of two phases: feature probing training and prediction ensemble inference. It trains multiple classifiers on intermediate features from a frozen pretrained model and uses semi-supervised learning to mitigate label noise. Their predictions are then combined through a crowdsourcing inference approach that accounts for differences in probe reliability. Extensive experiments across image classification benchmarks and medical-image settings show that SEPL improves robustness under label noise across DG methods.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Domain Generalization, Noisy Label Learning, Healthcare, Ensemble Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_m8mkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Distribution shift and label noise severely threaten model robustness and degrade domain generalization (DG) performance on unseen domains. The authors observe that intermediate representations preserve discriminative cues focusing on diverse image regions. This work introduces Self-Ensemble Post Learning (SEPL), a two-stage framework built upon a frozen pretrained model: it first probes intermediate features with multiple classifiers under semi-supervised learning to alleviate label noise, then aggregates predictions via a crowdsourcing-style inference that weights probe reliability. Experiments on standard classification and medical image benchmarks validate that SEPL consistently boosts the label-noise robustness of existing DG approaches.

Critical strengths
  1. The authors observe and leverage discriminative information preserved in intermediate layers beyond the final classifier for layer-wise ensemble, and avoids the performance degradation of naive equal-weight averaging by acknowledging varying reliability across probes.

  2. The experimental results are plentiful and the authors provide source codes which will help the community.

Critical weaknesses
  1. The writing quality is poor.

  2. The Grad-CAM visualization seems useless. The observation of diverse discriminative information preserved in intermediate layers seems to be enough for mativating the aggregation. Such visualiztaion does not help for determin the weight in aggreation. It does not help to select low entropy in semi-supervised learning.

  3. There are much more details not provided. Please refer to Questions For Authors.

Questions for authors
  1. What does N means in Equation (7) ? Does it means the total number of all source domain samples? So I_l contains the \gammaN samples with low entropies? I think the statement is hard to understand. If means the total number of all source domain samples, what if I select \gammaN/C samples with low entropies per class? I think such selection would help for balancing. Besides, how to determine \gamma? How the trained model performs if \gamma changes?

  2. Overall, I think that adaptive methods use unlabeled target batches is irreational, so I prefer fixed method. Then, can the authors explain why Fixed SoftDS performs worse that baseline in Table 5?

  3. What does K=3 means in Figure 5? Does it means Layer 1-2 + final prediction or Layer 5-6 +final prediction?

  4. Why not compare MixMatch in Table 3 (b)? MixMatch is adopted in the proposed method so such comparison is more valuable. Besides, DivideMix + Ensemble and MixMatch + Ensemble should be also evaludated.

3 comments
AuthorsOpenReview ↗

Response to Reviewer m8mk

Response to W1: Writing and Presentation

Thank you for pointing this out. We acknowledge that the presentation needs improvement. We will revise the introduction and Section 3 to make the reasoning clearer: layer-dependent differences in discrimination and noise fitting motivate multiple probes; noisy supervision motivates confidence-guided refinement; and unequal probe reliability motivates aggregation. We will also clarify the distinction between probe training and inference, explicitly separate target-batch DS from fixed-source inference, and standardize terminology and notation. Finally, we will simplify dense passages, improve figure captions, and distinguish empirical observations from causal claims throughout the paper.

Response to W2: Purpose of the Grad-CAM Visualization

Thank you for this clarification. We agree that Grad-CAM does not determine aggregation weights or select low-entropy samples. Its purpose is qualitative: it helps readers understand how predictions from different depths may rely on different spatial evidence, making the motivation for exploring intermediate representations more intuitive. Aggregation instead uses prediction-based reliability estimates, while sample selection uses the entropy of aggregated predictions. We will clarify this distinction and present Grad-CAM as an illustration of the motivation, without treating it as proof of complementarity or a component of the algorithm.

Response to W3: Missing Implementation Details

Thank you for highlighting the need for additional details. We address the specific questions individually in our responses to Q1–Q4 below.

Response to Q1: Definition of N, Classwise Selection, and Sensitivity to γ

Thank you for this insightful suggestion. As stated in Section 3.2, NN denotes the number of examples in the aggregation set. However, we agree that its scope in Eq. (7) was insufficiently explicit. During refinement, selection is performed separately within each source domain. Thus, for a domain containing NdN_d training samples, Iℓ,dI_{\ell,d} contains the indices of the ⌊γNd⌋\lfloor\gamma N_d\rfloor lowest-entropy samples. Their observed labels are retained; the remaining samples participate as unlabeled data. We will make this domain-wise notation explicit.

Your class-balanced selection suggestion is valuable. We tested a related variant that retains the lowest-entropy half within each predicted class. On the same 12 PACS checkpoints, it achieves 84.52%, compared with 84.25% for the original selection rule. This modest difference supports further exploration, although it does not establish statistical significance or directly evaluate equal per-class quotas.

SelectionPACS accuracy (%)Retained-label noise (%)
Entropy84.25 ± 0.5012.29 ± 0.36
Classwise entropy84.52 ± 0.629.92 ± 0.47

We fix γ=0.5\gamma=0.5, as specified in Section 3.2 and Table 1, without assuming the noise rate. Figure 5(c) examines its sensitivity: accuracy varies, but both DS variants remain above the plotted baseline across the tested values.

Our focus is the framework motivated by layer-dependent discrimination and noise robustness. Its selection module is replaceable, and improved class balancing and adaptive selection ratios are valuable directions for future work.

AuthorsOpenReview ↗

Response to Reviewer m8mk (2)

Response to Q2: Why Fixed SoftDS Can Underperform the Backbone

Thank you for raising this concern. We appreciate the importance of fixed, independent inference when target batches are unavailable. Table 5 indeed shows that Fixed SoftDS achieves 83.2%, below the backbone’s 84.9%, and we should explain this limitation more clearly.

A plausible explanation is that fixed aggregation transfers source-estimated probe reliabilities and class priors to the target domain. Under combined distribution shift and label noise, these estimates may be inaccurate: probe reliability can change across domains, while noisy source labels can distort its estimation. SoftDS also uses probability magnitudes, making it potentially sensitive to changes in confidence calibration. Furthermore, probes share a backbone, so DS’s conditional-independence assumption is approximate and correlated evidence may be overcounted. These are possible mechanisms, rather than experimentally established causes.

Our additional evaluation on 12 PACS checkpoints reinforces this limitation. Under the same fixed-source inference protocol, refinement improves accuracy from 80.79% to 82.00%, but remains below the 82.52% backbone. Thus, better probe training alone does not guarantee an improvement under fixed aggregation. Table 5’s Fixed HardDS result of 85.2% also indicates that the aggregation choice matters.

Estimating transferable probe reliability under both distribution shift and label noise remains challenging. These limitations are acknowledged in Sections 3.2 and 7. We will make their implications for Fixed SoftDS more explicit and investigate noise-aware, dependence-aware reliability estimation for fixed inference as future work.

Response to Q3: Meaning of K=3 in Figure 5

Thank you for pointing out this ambiguity. Section 3.2 defines KK as the total number of prediction sources, including the original classifier. However, we did not clearly specify the probe selection order in Figure 5. Probes are numbered from shallow to deep, and smaller ensembles retain the deepest probes first. Thus, K=3K=3 means Probes 5 and 6 plus the original classifier’s prediction. We will explicitly state this ordering in the figure caption.

Response to Q4: MixMatch, MixMatch + Ensemble, and DivideMix + Ensemble

Thank you for this helpful suggestion. We agree that standalone MixMatch and ensemble comparisons help clarify SEPL’s contribution. Our independently trained, full-model MixMatch baseline achieves 82.54 ± 1.16% on PACS.

Since SEPL refines probes on a frozen backbone, we additionally evaluate a single-backbone ensemble of six final-layer probes, matching SEPL’s backbone checkpoints, probe counts, MixMatch refinement rounds, and SoftDS aggregation:

DatasetERM++Final-layer ensembleSEPL
PACS82.52 ± 0.34%83.64 ± 0.32%84.25 ± 0.50%

SEPL improves over this matched final-layer ensemble by 0.61 percentage points, supporting the usefulness of information across depths beyond applying MixMatch and aggregation to final-layer features.

MixMatch + Ensemble and DivideMix + Ensemble, which combine multiple independently trained full networks, achieve 83.71 ± 0.40% and 81.41 ± 1.61% on PACS, respectively. Both incur additional backbone training and inference costs, while their accuracies remain numerically below SEPL’s 84.25 ± 0.50% under their respective experimental protocols. Nevertheless, combining SEPL with DiWA-inspired model averaging may further improve performance; we leave this exploration to future work.

Reviewer_m8mkOpenReview ↗

response

Parts of my concerns are addressed and I appreciate the efforts you put. I think experiments on how the trained model performs if \gamma changes would be helpful. Nevertheless, I have read other reviewers' concern of limited novelty which is not addressed and I decide to maintain my rating.

Reviewer_1kJ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Self-Ensemble Post Learning (SEPL), a post-training framework for improving domain generalization under noisy source labels. SEPL trains multiple classifiers on intermediate representations of a frozen backbone, uses entropy-based sample selection and semi-supervised learning to reduce label noise, and aggregates predictions through reliability-aware Dawid–Skene estimation. Experiments on several image classification and medical imaging benchmarks demonstrate improvements over existing DG baselines.

Critical strengths

1. Consistent improvements across multiple benchmarks.

The paper evaluates SEPL on five datasets, including standard DG benchmarks and medical imaging tasks. The results show consistent improvements over ERM++ under noisy-label settings. Additional experiments with different noise levels, asymmetric noise, and ViT backbones further support the effectiveness of the proposed framework.

2. Practical post-training framework with useful empirical analysis.

SEPL can be applied to existing DG models without modifying or retraining the backbone, making it a potentially useful extension to existing methods. The paper also provides ablation studies, layer-wise representation analysis, and comparisons of different aggregation strategies, offering useful insights into the behavior and limitations of the approach.

Critical weaknesses

1. Limited methodological novelty.

SEPL mainly combines existing techniques, including intermediate-layer classifiers [1, 2], semi-supervised learning for noisy labels [3], and Dawid–Skene aggregation [4]. While applying these techniques to noisy DG is reasonable, the paper does not clearly establish the new insights from their combination. Stronger comparisons with simpler alternatives would help justify the contribution.

2. Insufficient evidence for the necessity of the proposed components.

Table 3(a) shows that linear probing with DS aggregation already improves accuracy from 76.7% to 80.1%, which is close to the gain achieved by full SEPL (81.0%). Moreover, the simpler AVG+DS configuration achieves 81.2%, slightly outperforming SEPL. These results leave the necessity of the proposed refinement strategy unclear. A more complete ablation on ERM++, which is used in the main experiments, would be helpful.

3. Potentially unfair comparisons due to transductive inference.

SEPL uses adaptive DS aggregation on unlabeled target batches, whereas the DG baselines generally perform independent inference without target-batch adaptation [5]. Table 5 shows that adaptive SoftDS achieves 86.2% accuracy, while fixed SoftDS reaches only 83.2%, below the original backbone (84.9%). This raises concerns about how much of the reported improvement comes from access to target-batch information. Comparisons under matched inference protocols are needed.

4. Insufficient validation of intermediate-layer complementarity.

The paper argues that intermediate representations provide complementary information, but the Grad-CAM visualizations and correlation analysis do not establish that this complementarity drives the performance gains. In particular, Probes 5 and 6 show a high prediction correlation of 0.967. A useful baseline would be an ensemble of classifiers trained on the same final-layer features, with matched probe counts and aggregation methods, to isolate the benefits of using intermediate representations.

5. Limited comparisons with recent noisy DG methods.

The paper lacks direct comparisons with several recent noisy DG approaches, such as D2AP [6] and DL4ND [7]. The A3W [8] results are also taken from a separate study without matched experimental settings. Although these methods may differ in their evaluation protocols, more carefully controlled comparisons would help establish the competitiveness of SEPL.

References

[1] Lee et al., "Deeply-Supervised Nets." AISTATS, 2015. https://proceedings.mlr.press/v38/lee15a.html

[2] Teerapittayanon et al., "BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks." ICPR, 2016. https://arxiv.org/abs/1709.01686

[3] Li et al., "DivideMix: Learning with Noisy Labels as Semi-Supervised Learning." ICLR, 2020. https://arxiv.org/abs/2002.07394

[4] Dawid and Skene, "Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm." JRSS-C, 1979. https://doi.org/10.2307/2346806

[5] Gulrajani and Lopez-Paz, "In Search of Lost Domain Generalization." ICLR, 2021. https://arxiv.org/abs/2007.01434

[6] Wang et al., "D2AP: Double Debiasing with Adaptive Proxies for Domain Generalization in Noisy Environment." Knowledge-Based Systems, 2025. https://doi.org/10.1016/j.knosys.2025.113458

[7] Wang et al., "Noise-Aware Generalization: Robustness to In-Domain Noise and Out-of-Domain Generalization." ICLR, 2026. https://proceedings.iclr.cc/paper_files/paper/2026/hash/6b90ec5be9adbaf8d6316da2211f745d-Abstract-Conference.html

[8] Dai et al., "A Language Anchor-Guided Method for Robust Noisy Domain Generalization." 2025. https://arxiv.org/abs/2503.17211

Questions for authors

I encourage authors to answer the questions in the Weaknesses part.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1kJ9

Response to W1: Methodological Novelty

Thank you for this constructive comment. We agree that the individual techniques are established, and we will clarify the insight motivating their adaptation and integration. Under label noise and distribution shift, different depths exhibit different trade-offs among discrimination, noise fitting, and prediction diversity. This motivates SEPL’s framework: intermediate probes expose these differences, confidence-guided refinement limits learning from noisy labels, and reliability-aware aggregation combines unequal prediction sources. The backbone remains frozen, making the framework applicable to existing models.

We strengthened the comparisons on the same 12 PACS checkpoints. Simple alternatives achieve 83.16% with flip TTA and 83.13% with ridge readout, compared with 84.25% for SEPL. More directly, supervised layer-wise probes trained for the same 6,000 optimization updates, with the same final aggregation, achieve 82.07%; adding SEPL’s refinement raises accuracy to 84.25%. A six-head final-layer ensemble with matched refinement and aggregation achieves 83.64%, providing additional support for using intermediate representations.

These comparisons support the framework’s design beyond simply adding classifiers or extending training.

Response to W2: Necessity of Refinement and Aggregation Choices

Thank you for this helpful suggestion. AVG+DS is also a SEPL variant, explicitly supported by the “DS/AVG” notation in our framework diagram. It uses AVG during refinement and DS during inference, while retaining semi-supervised refinement.

We conducted matched PACS ablations on 12 frozen ERM++ checkpoints, keeping DS inference fixed:

Probe trainingTotal updatesAccuracy
Initial supervised probing1,00082.81%
Longer supervised probing6,00082.07%
SEPL1,000 + 5×1,00084.25%

SEPL adds 1.44 points over initialization and 2.18 points over supervised probing with the same number of updates. However, Figure 5(b) shows that more refinement rounds do not necessarily improve accuracy, motivating a modest refinement budget.

DS also does not uniformly outperform AVG. Averaged over three matched configurations per domain, AVG+DS leads on 3/4 domains (A, C, S) with all six probes, while DS+DS leads on P. Retaining the prespecified last three probes only at inference reverses this pattern: DS+DS leads on 3/4 domains (A, P, S), while AVG+DS leads on C. These varying rankings support including both aggregation choices within SEPL.

Response to W3: Transductive Inference and Matched Inference Protocols

Thank you for raising this concern. We agree that independent and target-adaptive inference should be reported separately. Table 5’s Fixed SoftDS result—83.2%, below the 84.9% backbone—is an important limitation.

Source-estimated probe reliability and class priors may not transfer under distribution shift and label noise. Changes in confidence calibration and correlated errors among probes sharing a backbone may also affect SoftDS. These are plausible mechanisms, not experimentally established causes.

Our additional evaluation uses 12 PACS checkpoints across four domains; the backbone mean is 82.52%. With six probes and matched training updates, refinement improves Fixed SoftDS from 80.79% to 82.00%, but remains below the backbone. We also tested DS during source refinement followed by independent AVG inference:

Retained probesAVGFixed-source SoftDSAdaptive SoftDS
All79.70%82.00%84.25%
Last 382.87%82.96%84.29%
Source-selected82.78%83.09%84.19%

Source-selected counts use only noisy source-validation labels. Reduced-probe results prune the same trained bank at inference. These results show modest gains under independent inference, while the larger adaptive results use additional target-batch information.

Table 5’s Fixed HardDS result of 85.2% further highlights the importance of aggregation choice. We will clarify these limitations in Sections 3.2 and 7 and investigate noise-aware, dependence-aware source calibration and joint sparse probe selection for independent inference as future work.

AuthorsOpenReview ↗

Response to Reviewer 1kJ9 (2)

Response to W4: Evidence for Intermediate-Layer Complementarity

Thank you for this suggestion. We agree that Grad-CAM and correlation analysis alone do not establish that complementarity drives the gains. Although Probes 5 and 6 have a high prediction correlation of 0.967, high overall agreement does not rule out useful complementary information on the examples where they disagree.

We added six independently initialized final-layer classifiers, matching SEPL’s backbone checkpoints, probe count, refinement rounds, and SoftDS aggregation. On PACS, this ensemble achieves 83.64%, compared with 84.25% for SEPL, supporting the usefulness of intermediate representations in this setting.

DatasetERM++Final-layer ensembleSEPL
PACS82.52 ± 0.34%83.64 ± 0.32%84.25 ± 0.50%

Across all 20 three-layer subsets with the same backbone anchor, source-only correlation pruning achieves 83.10%, versus 84.25% with all six probes and 84.29% with the last three. Thus, removing correlated probes can also discard useful evidence; correlation alone does not determine utility. These comparisons support our interpretation but do not constitute strict causal proof, and we will revise the claims accordingly.

Response to W5: Comparisons with Recent Noisy DG Methods

Thank you for this suggestion. To address this concern, we reimplemented D2AP, DL4ND, and A3W following their papers, as official implementations were unavailable to us. We carefully tuned these baselines and evaluated them under a common experimental setting. Their average accuracies (%) are:

MethodPACSVLCSOfficeHome
DL4ND82.7278.1365.74
D2AP77.2474.6664.54
A3W76.4273.3957.83
SEPL83.778.767.5

SEPL retains the manuscript results; the other entries are our reimplementations. Thus, the additional A3W comparison comes from our own evaluation, supplementing the previously cited result.

SEPL achieves the highest mean accuracy across these three benchmarks, providing additional evidence of its competitiveness against recent noisy DG approaches. We will include these supplementary comparisons and explicitly document the implementation details and remaining protocol differences in the revised manuscript.

Thanks for the detailed rebuttal and the extra experiments. I appreciate the effort you put into addressing my comments—especially the matched-budget ablations and the final-layer ensemble comparison, which help clarify what each part of the method contributes.

That said, my main concerns still remain:

  • Limited novelty: The overall framework still feels largely like a combination of existing techniques.
  • Unmatched inference setup: The new results actually suggest that much of the gain relies heavily on transductive target-batch inference, and we still lack enough comparisons under the exact same inference protocol.
  • Questionable baseline numbers: The reproduced results for recent noisy DG methods also raise some doubts about whether the comparisons are completely fair.

Overall, while the rebuttal cleared up several points, these core issues aren't resolved enough for me to change my assessment.

Reviewer_8fZzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Self-Ensemble Post Learning (SEPL), a post-training framework for noisy domain generalization. SEPL trains multiple classification probes on intermediate features of a frozen backbone, employs entropy-based sample selection and MixMatch to mitigate label noise, and uses Dawid–Skene aggregation to combine predictions according to estimated probe reliability. Experiments on three domain generalization benchmarks and two medical image datasets demonstrate improvements over several existing DG baselines under synthetic label noise. I have the following strengths and concerns regarding the paper.

Critical strengths
  1. Clear motivation and practical framework. Section 3.1 provides empirical observations of layer-wise discrimination, spatial attention, and noise fitting. The proposed framework exploits intermediate representations without updating the pretrained backbone, making it applicable to different DG models.
  2. Extensive empirical evaluation. Section 4 evaluates SEPL on multiple benchmarks and DG methods. The experiments include ablation studies, different noise settings, a ViT backbone, and statistical comparisons, providing useful evidence of its effectiveness.
  3. Reliability-aware prediction aggregation. Section 3.2 considers differences in prediction reliability across intermediate classifiers rather than assigning equal weights. The analyses of probe correlations, inference protocols, and class imbalance provide additional insights into the method.
Critical weaknesses
  1. The proposed framework mainly combines existing techniques, including intermediate-layer probing, entropy-based sample selection, MixMatch, and Dawid–Skene aggregation. Although their integration is practically useful, the paper does not sufficiently establish what is methodologically novel beyond this combination. More importantly, the experiments do not demonstrate whether intermediate-layer complementarity is essential to the improvements. A comparison with multiple classifiers trained exclusively on final-layer features under the same aggregation strategy is missing. Moreover, AVG+DS achieves 81.2%, slightly outperforming the complete SEPL configuration at 81.0% in Table 3(a).
  2. The proposed method selects the lowest-entropy 50% of samples as labeled data, assuming that lower prediction entropy indicates more reliable supervision. However, incorrect predictions can also have low entropy, potentially retaining noisy labels during probe refinement. Although Table 4(c) evaluates the overall selection quality, the paper does not sufficiently investigate whether this strategy introduces class-wise or domain-wise selection bias. Additional comparisons with alternative sample selection strategies would strengthen the justification of this design.
  3. The main experiments primarily compare SEPL with general DG baselines. The A3W results are taken from a separate study without matched data splits, corruption masks, or experimental runs, while recent noisy DG methods such as D2AP and DL4ND are not directly evaluated. Although statistical significance is examined on PACS-A, corresponding analyses across other benchmarks and target domains remain limited. More comparisons under matched experimental protocols are needed to establish the effectiveness of SEPL relative to recent noisy DG methods.
  4. On HAM10000 Group0, AVG aggregation improves overall accuracy over ERM++ but reduces balanced accuracy from 63.1% to 59.6% and macro-F1 from 61.4% to 60.6%, raising concerns about minority-class performance. In addition, Figure 6(d) lacks a zero-noise control, making it difficult to distinguish general post-training improvements from noise-specific robustness gains. Most experiments rely on synthetic label corruption, leaving robustness under instance-dependent noise and naturally occurring annotation errors insufficiently explored.
  5. Although SEPL avoids updating the backbone, it introduces additional probe training, multiple refinement rounds, and iterative Dawid–Skene aggregation. Table 5 reports substantial additional post-training time and inference overhead, while the average accuracy improvement over ERM++ is relatively modest on several benchmarks. Although computational costs are reported, the paper lacks a systematic accuracy–efficiency comparison with simpler post-training alternatives. Additional experiments would help establish whether the performance gains justify the extra computational cost.
Questions for authors
  1. Could the authors compare SEPL with an ensemble of classifiers trained exclusively on final-layer features, using the same number of probes and aggregation strategy, to verify the necessity of intermediate-layer complementarity?
  2. How much of SEPL's improvement comes from adaptive target-batch aggregation rather than semi-supervised probe refinement? How sensitive is Dawid–Skene aggregation to target-batch size and probe correlations? Could the authors provide matched comparisons under fixed-source inference and investigate removing highly correlated probes?
  3. Could the authors provide additional comparisons with recent noisy DG methods, particularly A3W, D2AP, and DL4ND where their settings are compatible, under matched experimental protocols? Could statistical significance analyses also be extended to other benchmarks and target domains?
  4. How does SEPL perform under zero-noise, instance-dependent noise, or naturally occurring annotation errors? Could the authors report balanced accuracy, macro-F1, and class-wise recall across all medical domains to better evaluate minority-class performance?
  5. The entropy-based splitting strategy in Section 3.2 retains the lowest-entropy 50% of samples as labeled data. Could this global selection strategy lead to underrepresentation of difficult classes or domains? Could the authors analyze the class-wise and domain-wise sample distributions after splitting and compare with class-balanced or domain-balanced selection strategies?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8fZz

Response to W1: Methodological Novelty and Intermediate-Layer Complementarity

Thank you for this constructive comment. Our central observation is that, under label noise and distribution shift, different depths exhibit different noise robustness and discriminative ability, producing predictors with different error patterns and reliabilities. SEPL’s contribution is an observation-driven framework that exploits this depth-dependent diversity through coupled confidence-guided refinement and reliability-aware crowdsourcing. To make the framework readily applicable to existing DG models, it operates after backbone training: the backbone remains frozen, and only the added probes are trained.

Your suggested final-layer comparison is valuable. We added six independently initialized final-layer probes, matching SEPL’s backbone checkpoints, probe count, refinement rounds, and SoftDS aggregation.

DatasetERM++Final-layer ensembleSEPL
PACS82.52 ± 0.34%83.64 ± 0.32%84.25 ± 0.50%

Values are mean ± configuration SD across three four-domain averages, with hyperparameters and seeds varying. The final-layer ensemble improves over ERM++, but SEPL achieves another 0.61 percentage points. One possible explanation is that sharing the same final-layer features limits diversity among independently initialized heads and yields correlated errors. The additional gain from SEPL is consistent with the value of complementary information across depths in this setting.

We used three fixed configurations per target domain with different seeds and evaluated paired post-training gains against their corresponding backbones. These results provide further evidence that our post-processing framework is effective across different backbone configurations, supporting its broader applicability without extensive backbone tuning.

We acknowledge that AVG+DS achieves 81.2%, slightly exceeding our default’s 81.0% in Table 3(a). SEPL is a flexible framework: as illustrated in our framework diagram, AVG can replace DS as an aggregation module. DS estimates probe reliability under simplifying assumptions, and correlated probe errors may limit the accuracy of these estimates, offering a plausible explanation for this difference. Our default is therefore not universally optimal, and developing more accurate reliability estimators is an important direction for future work.

Finally, final-layer probe diversity shares the diversity motivation of DiWA, although DiWA averages weights while our control aggregates predictions. We had considered adding further final-layer probes to SEPL, but kept the submitted ensemble compact given additional costs and potentially redundant, correlated votes. Combining initialization diversity with depth diversity may further strengthen SEPL and is a direction for future work.

Response to W2: Entropy-Based Selection and Class/Domain Bias

Thank you for raising this point. Entropy is a common confidence criterion in self-training, but low entropy does not guarantee correct supervision. Our audit found a retained prediction error of 0.10 ± 0.01%.

We recorded selection at every refinement round. Entropy filtering generally under-selected dog and horse and over-selected guitar and house. The maximum gap between a class’s retention rate and the overall source-pool retention rate was 38.17 percentage points, reaching 50.00 points within class–domain groups. Selection is performed separately within each source domain, preserving approximately 50% coverage per domain. However, source-domain mean retained-label noise ranged from 7.34% to 16.57%, showing that similar coverage does not ensure equally reliable supervision.

Using the same 12 ERM++ checkpoints and refinement protocol, we compared:

SelectionPACS accuracy (%)Retained-label noise (%)
Entropy84.25 ± 0.5012.29 ± 0.36
Random84.03 ± 0.6924.74 ± 0.10
Classwise entropy84.52 ± 0.629.92 ± 0.47
UniCon-inspired JSD selection [1]84.17 ± 0.518.00 ± 0.17

Values are mean ± SD across three configuration groups with different hyperparameters and seeds; accuracy is averaged over four domains. Classwise entropy reduced the maximum class-retention gap to 3.38 points, with a slightly higher mean accuracy. These results support a class-aware alternative, without establishing statistical significance or the optimality of entropy filtering.

Finally, unselected images remain in MixMatch’s unlabeled branch, contributing through augmented-view pseudo-labels, consistency loss, and MixUp. The split determines which observed annotations provide direct supervision; it does not discard the remaining training images.

[1] Karim N, Rizve M N, Rahnavard N, et al. Unicon: Combating label noise through uniform selection and contrastive learning[C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022: 9666-9676.

AuthorsOpenReview ↗

Response to Reviewer 8fZz (2)

Response to W3: Comparisons with Recent Noisy DG Methods and Statistical Evidence

We thank the reviewer for this suggestion. We have added direct evaluations of A3W, D2AP, and DL4ND. At the time of our reproduction, we could not obtain publicly accessible official implementations of these three methods. We therefore independently reimplemented them based on their published descriptions, with documented implementation adaptations and source-domain validation checks. For a fair comparison, we evaluated the reimplemented methods under a consistent experimental setup, with hyperparameters calibrated using source-domain validation. The results are summarized below.

MethodPACSVLCSOfficeHome
A3W76.42 ± 3.2173.39 ± 0.7557.83 ± 0.09
D2AP77.24 ± 2.2274.66 ± 0.2664.54 ± 0.51
DL4ND82.72 ± 0.7878.13 ± 0.4465.74 ± 0.39
SEPL83.778.767.5

SEPL’s reported mean accuracy is higher on all three benchmarks. These additional experiments broaden the comparison beyond general DG baselines and provide direct evaluations of recent noisy DG approaches.

We additionally extended the checkpoint-paired analysis to the remaining three PACS domains and all four OfficeHome domains, with six independently trained checkpoints per target. The domain-averaged improvements over the same ERM++ backbones are:

Dataset coveragePaired improvement (pp, mean ± SD)
PACS+1.25 ± 0.67
OfficeHome+0.90 ± 0.65

Mean improvements are positive on all seven targets. Three targets achieve an exact unadjusted p = 0.03125. These results extend the paired evidence beyond PACS-A, while we distinguish positive average effects from statistical significance after correction.

Response to W4: Minority-Class Performance and Additional Noise Settings

Thank you for highlighting this important trade-off. We acknowledge AVG’s decline in BA and macro-F1 despite higher OA. Table 6(a) also shows metric-dependent trade-offs: SoftDS raises BA from 63.1% to 68.7%, while HardDS raises macro-F1 from 61.4% to 62.6%.

SEPL also supports an imbalance-aware extension through the class-prior term in DS aggregation. Section 6 and Table 6(b) provide a focused illustration. Jointly varying class imbalance, label noise and distribution shift introduces interacting factors that warrant a dedicated systematic study. To keep the present work focused on noisy domain generalization, we included this lightweight prior-aware analysis without an extensive joint evaluation across these factors.

These gains are over the corresponding uniform-prior DS variants and are confined to Group0. Since Table 6(b) reports OA without the corresponding BA/F1, we regard it as evidence of the framework’s extensibility, while broader validation of minority-class performance remains necessary. We will make this scope and limitation clearer.

Second, the original study includes symmetric corruption and asymmetric noise in Table 3(c), where ERM++ improves from 80.2% to 84.3%. We appreciate the reviewer’s concern that these synthetic settings remain limited; each stress test uses one configuration and three runs. To address the missing control in Figure 6(d), we added a separate zero-added-noise control and broader supplementary tests:

ConditionERM++SEPL
PACS, 0% added noise87.25 ± 1.3387.79 ± 1.18
PACS, 25% symmetric noise82.52 ± 0.3484.25 ± 0.50
PACS, projection IDN80.43 ± 3.2184.23 ± 1.49
CIFAR-10N → CIFAR-10.186.79 ± 0.2687.44 ± 0.11

Values are mean ± sample SD across three configurations varying hyperparameters, training seed and holdout split; PACS first averages four target domains.

Here, projection IDN means our synthetic instance-dependent label corruption. Each image is resized to 8×8 RGB and converted to a centered, normalized 192-dimensional pixel vector. Class-specific Gaussian random projections determine its logistic flip probability, calibrated to an average of 25%; another projection and softmax sample an incorrect class, excluding the original class. Thus, both whether a label changes and which wrong class is assigned depend on the image. Only source labels are corrupted; target labels remain unchanged. This remains a controlled pixel-based stress test.

Finally, pairing CIFAR-10N aggregate human annotations (9.01% disagreement with reference labels) with CIFAR-10.1 forms a single-source, mild-shift test, using 1,600 target images per configuration. Observed-label source validation selects three new probes plus the original classifier, one refinement round, probe learning rate ×3 and AVG, reusing the ERM++ checkpoints. BA/macro-F1 improve from 86.76/86.70% to 87.44/87.41%; three fixed-setting replays reproduce its weights and predictions exactly. We retain Section 7’s limitation concerning realistic and clinical annotation errors.

AuthorsOpenReview ↗

Response to Reviewer 8fZz (3)

Response to W5: Accuracy–Efficiency Trade-off

Thank you for this suggestion. We agree that SEPL's gains should be weighed against its additional computation.

We compared six lightweight baselines on the same 12 PACS ERM++ checkpoints, selecting hyperparameters using noisy source validation only. Readout and nearest-neighbor baselines adapt the cited methodologies to this setting.

MethodMean PACS accuracy (%)Post-training stage (s)
ERM++82.520.0
Temperature scaling [1]82.525.3
Flip TTA [2]83.160.0
Logistic readout [3]82.6756.7
Ridge readout [4]83.1356.3
Cosine kNN [5]82.9127.7
NCM [5]82.8824.1
SEPL (0 refinement rounds) + SoftDS82.81239.5
SEPL (5 refinement rounds) + SoftDS84.251219.2

Both SEPL variants use the same frozen backbone, six layer-wise probes, and final SoftDS aggregation. The 0-round variant stops after 1,000 supervised updates; the 5-round variant adds five 1,000-step refinement rounds.

SEPL improves over ERM++ by 1.73 percentage points, the best lightweight baseline (TTA) by 1.09 points, and supervision-only probes by 1.44 points. These gains require a larger offline budget.

Times are observed stage costs under shared load, with different initialization/evaluation boundaries; they do not establish end-to-end speed ratios. TTA has no fitting stage but requires two forward passes; SoftDS uses the unlabeled target collection.

SEPL is a general post-training framework; the manuscript's ERM ablation gains 4.3 points (76.7% to 81.0%). Our vectorized CPU SoftDS reduces the aggregation component from 33.32 to 0.629 ms/image with identical posteriors.

Future work could explore more efficient backbones, compact probe architectures, or distillation into a single network to reduce computation. These directions remain to be evaluated.

[1] Guo C, Pleiss G, Sun Y, et al. On calibration of modern neural networks[C]//International conference on machine learning. PMLR, 2017: 1321-1330.

[2] Shanmugam D, Blalock D, Balakrishnan G, et al. Better aggregation in test-time augmentation[C]//2021 IEEE/CVF international conference on computer vision (ICCV). IEEE, 2021: 1194-1203.

[3] Kirichenko P, Izmailov P, Wilson A G. Last layer re-training is sufficient for robustness to spurious correlations. ICLR, 2023.

[4] Hoerl A E, Kennard R W. Ridge regression: Biased estimation for nonorthogonal problems[J]. Technometrics, 1970, 12(1): 55-67.

[5] Wang Y, Chao W L, Weinberger K Q, et al. Simpleshot: Revisiting nearest-neighbor classification for few-shot learning[J]. arXiv preprint arXiv:1911.04623, 2019.

Response to Q1: Final-Layer Ensemble Control

Please see our response to W1 above for the matched final-layer ensemble comparison. With the same backbone checkpoints, probe count, refinement rounds, and SoftDS aggregation, the final-layer ensemble achieves 83.64 ± 0.32% on PACS, compared with 84.25 ± 0.50% for SEPL.

Response to Q2: Refinement, Target-Batch Adaptation, and Probe Correlations

Thank you for this helpful suggestion. Extending Table 5, we evaluated 12 PACS ERM++ checkpoints. Fixed-source calibration uses only noisy source-validation labels.

Probe trainingFixed-sourceAdaptive
Supervised80.79%82.07%
SEPL82.00%84.25%

Refinement improves accuracy by 1.21 points under fixed-source inference and 2.18 under adaptive inference. With refined probes unchanged, the inference-protocol contrast is 2.25 points. These effects interact, so a unique percentage attribution is inappropriate. Fixed-source SEPL remains below the 82.52% backbone.

Using ten label-blind target partitions, batch sizes 8 / 128 / Full achieve 71.43 / 83.36 / 84.25%, showing substantial sensitivity to small batches. Fixed-source inference is batch invariant.

Across all 20 three-layer subsets with the same backbone anchor, source-only correlation pruning achieves 83.10%, versus 84.25% with all six probes and 84.29% with the last three. Thus, removing correlated probes can also discard useful evidence; correlation alone does not determine utility.

Response to Q3: Recent Noisy DG Baselines and Extended Statistical Analysis

Please see our response to W3 above for the additional A3W, D2AP, and DL4ND evaluations and the extended checkpoint-paired analysis across PACS and OfficeHome.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Responses to Reviewers

Dear Reviewers and Area Chair,

Thank you for your valuable feedback. We have uploaded a revised manuscript and provided detailed responses to all three reviewers' comments.

We would greatly appreciate it if you could review our responses and the updated manuscript. Please let us know if any concerns remain.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.