- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Distribution shift and label noise severely threaten model robustness and degrade domain generalization (DG) performance on unseen domains. The authors observe that intermediate representations preserve discriminative cues focusing on diverse image regions. This work introduces Self-Ensemble Post Learning (SEPL), a two-stage framework built upon a frozen pretrained model: it first probes intermediate features with multiple classifiers under semi-supervised learning to alleviate label noise, then aggregates predictions via a crowdsourcing-style inference that weights probe reliability. Experiments on standard classification and medical image benchmarks validate that SEPL consistently boosts the label-noise robustness of existing DG approaches.
Critical strengths
-
The authors observe and leverage discriminative information preserved in intermediate layers beyond the final classifier for layer-wise ensemble, and avoids the performance degradation of naive equal-weight averaging by acknowledging varying reliability across probes.
-
The experimental results are plentiful and the authors provide source codes which will help the community.
Critical weaknesses
-
The writing quality is poor.
-
The Grad-CAM visualization seems useless. The observation of diverse discriminative information preserved in intermediate layers seems to be enough for mativating the aggregation. Such visualiztaion does not help for determin the weight in aggreation. It does not help to select low entropy in semi-supervised learning.
-
There are much more details not provided. Please refer to Questions For Authors.
Questions for authors
-
What does N means in Equation (7) ? Does it means the total number of all source domain samples? So I_l contains the \gammaN samples with low entropies? I think the statement is hard to understand. If means the total number of all source domain samples, what if I select \gammaN/C samples with low entropies per class? I think such selection would help for balancing. Besides, how to determine \gamma? How the trained model performs if \gamma changes?
-
Overall, I think that adaptive methods use unlabeled target batches is irreational, so I prefer fixed method. Then, can the authors explain why Fixed SoftDS performs worse that baseline in Table 5?
-
What does K=3 means in Figure 5? Does it means Layer 1-2 + final prediction or Layer 5-6 +final prediction?
-
Why not compare MixMatch in Table 3 (b)? MixMatch is adopted in the proposed method so such comparison is more valuable. Besides, DivideMix + Ensemble and MixMatch + Ensemble should be also evaludated.
3 comments
Response to Reviewer m8mk (2)
Response to Q2: Why Fixed SoftDS Can Underperform the Backbone
Thank you for raising this concern. We appreciate the importance of fixed, independent inference when target batches are unavailable. Table 5 indeed shows that Fixed SoftDS achieves 83.2%, below the backbone’s 84.9%, and we should explain this limitation more clearly.
A plausible explanation is that fixed aggregation transfers source-estimated probe reliabilities and class priors to the target domain. Under combined distribution shift and label noise, these estimates may be inaccurate: probe reliability can change across domains, while noisy source labels can distort its estimation. SoftDS also uses probability magnitudes, making it potentially sensitive to changes in confidence calibration. Furthermore, probes share a backbone, so DS’s conditional-independence assumption is approximate and correlated evidence may be overcounted. These are possible mechanisms, rather than experimentally established causes.
Our additional evaluation on 12 PACS checkpoints reinforces this limitation. Under the same fixed-source inference protocol, refinement improves accuracy from 80.79% to 82.00%, but remains below the 82.52% backbone. Thus, better probe training alone does not guarantee an improvement under fixed aggregation. Table 5’s Fixed HardDS result of 85.2% also indicates that the aggregation choice matters.
Estimating transferable probe reliability under both distribution shift and label noise remains challenging. These limitations are acknowledged in Sections 3.2 and 7. We will make their implications for Fixed SoftDS more explicit and investigate noise-aware, dependence-aware reliability estimation for fixed inference as future work.
Response to Q3: Meaning of K=3 in Figure 5
Thank you for pointing out this ambiguity. Section 3.2 defines as the total number of prediction sources, including the original classifier. However, we did not clearly specify the probe selection order in Figure 5. Probes are numbered from shallow to deep, and smaller ensembles retain the deepest probes first. Thus, means Probes 5 and 6 plus the original classifier’s prediction. We will explicitly state this ordering in the figure caption.
Response to Q4: MixMatch, MixMatch + Ensemble, and DivideMix + Ensemble
Thank you for this helpful suggestion. We agree that standalone MixMatch and ensemble comparisons help clarify SEPL’s contribution. Our independently trained, full-model MixMatch baseline achieves 82.54 ± 1.16% on PACS.
Since SEPL refines probes on a frozen backbone, we additionally evaluate a single-backbone ensemble of six final-layer probes, matching SEPL’s backbone checkpoints, probe counts, MixMatch refinement rounds, and SoftDS aggregation:
| Dataset | ERM++ | Final-layer ensemble | SEPL |
|---|---|---|---|
| PACS | 82.52 ± 0.34% | 83.64 ± 0.32% | 84.25 ± 0.50% |
SEPL improves over this matched final-layer ensemble by 0.61 percentage points, supporting the usefulness of information across depths beyond applying MixMatch and aggregation to final-layer features.
MixMatch + Ensemble and DivideMix + Ensemble, which combine multiple independently trained full networks, achieve 83.71 ± 0.40% and 81.41 ± 1.61% on PACS, respectively. Both incur additional backbone training and inference costs, while their accuracies remain numerically below SEPL’s 84.25 ± 0.50% under their respective experimental protocols. Nevertheless, combining SEPL with DiWA-inspired model averaging may further improve performance; we leave this exploration to future work.
response
Parts of my concerns are addressed and I appreciate the efforts you put. I think experiments on how the trained model performs if \gamma changes would be helpful. Nevertheless, I have read other reviewers' concern of limited novelty which is not addressed and I decide to maintain my rating.
Response to Reviewer m8mk
Response to W1: Writing and Presentation
Thank you for pointing this out. We acknowledge that the presentation needs improvement. We will revise the introduction and Section 3 to make the reasoning clearer: layer-dependent differences in discrimination and noise fitting motivate multiple probes; noisy supervision motivates confidence-guided refinement; and unequal probe reliability motivates aggregation. We will also clarify the distinction between probe training and inference, explicitly separate target-batch DS from fixed-source inference, and standardize terminology and notation. Finally, we will simplify dense passages, improve figure captions, and distinguish empirical observations from causal claims throughout the paper.
Response to W2: Purpose of the Grad-CAM Visualization
Thank you for this clarification. We agree that Grad-CAM does not determine aggregation weights or select low-entropy samples. Its purpose is qualitative: it helps readers understand how predictions from different depths may rely on different spatial evidence, making the motivation for exploring intermediate representations more intuitive. Aggregation instead uses prediction-based reliability estimates, while sample selection uses the entropy of aggregated predictions. We will clarify this distinction and present Grad-CAM as an illustration of the motivation, without treating it as proof of complementarity or a component of the algorithm.
Response to W3: Missing Implementation Details
Thank you for highlighting the need for additional details. We address the specific questions individually in our responses to Q1–Q4 below.
Response to Q1: Definition of N, Classwise Selection, and Sensitivity to γ
Thank you for this insightful suggestion. As stated in Section 3.2, N denotes the number of examples in the aggregation set. However, we agree that its scope in Eq. (7) was insufficiently explicit. During refinement, selection is performed separately within each source domain. Thus, for a domain containing Nd training samples, Iℓ,d contains the indices of the ⌊γNd⌋ lowest-entropy samples. Their observed labels are retained; the remaining samples participate as unlabeled data. We will make this domain-wise notation explicit.
Your class-balanced selection suggestion is valuable. We tested a related variant that retains the lowest-entropy half within each predicted class. On the same 12 PACS checkpoints, it achieves 84.52%, compared with 84.25% for the original selection rule. This modest difference supports further exploration, although it does not establish statistical significance or directly evaluate equal per-class quotas.
We fix γ=0.5, as specified in Section 3.2 and Table 1, without assuming the noise rate. Figure 5(c) examines its sensitivity: accuracy varies, but both DS variants remain above the plotted baseline across the tested values.
Our focus is the framework motivated by layer-dependent discrimination and noise robustness. Its selection module is replaceable, and improved class balancing and adaptive selection ratios are valuable directions for future work.