acceptodds
Under review as a conference paper at ICLR 2027

Selective prediction forgetting for adaptation of weakly supervised localization of histology images

Abstract

Under strong cross-domain shifts, weakly supervised object localization (WSOL) models often lose their capacity to discriminate and develop a strong prediction bias toward a subset of classes. As a result, the majority of target samples are assigned to these dominant classes. Since source-free (unsupervised) domain adaptation (SFDA) methods typically rely on pseudo-labels, inaccurate pseudo-label estimation causes suboptimal adaptation and poor classification and localization performance. This issue is particularly challenging in computational histopathology, where shifts in organ-specific morphology, staining protocols, and acquisition conditions can significantly impact class-discriminative visual cues. This paper introduces SFDA with selective prediction forgetting (SFDA-SPF), a method designed to mitigate excessive concentration of predictions for dominant classes during target-domain adaptation. SFDA-SPF identifies prediction-dominant classes based on the model's target predictions and selectively revises uncertain samples assigned to them. Importantly, this procedure does not require knowledge of the true target class distribution. In contrast with previous SFDA methods, SFDA-SPF introduces a “forget set” composed of uncertain samples from dominant classes near their decision boundaries, enabling the classification boundaries to shift away from biased predictions. A second “retain set” is used to anchor certain and accurate samples. To forget the predictions of dominant classes, SFDA-SPF introduces a complementary optimization objective that reduces their probability while allowing alternative class assignments to emerge. This iterative optimization process reassesses target images for forget and retain sets at regular intervals. In addition to improving image-level pseudo-label estimation, the pixel-level classifier is jointly optimized, further enhancing localization. Experiments on cross-organ and cross-center histopathology benchmarks (GLAS, C16, and C17) show that SFDA-SPF consistently outperforms state-of-the-art SFDA methods in both classification and localization, including across different levels of target-domain class imbalance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Domain Adaptation, Histology Images, Weakly Supervised Object Localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FPwdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SFDA-SPF, a source-free domain adaptation framework for weakly supervised object localization (WSOL) in histopathology. The method addresses prediction bias under severe domain shifts, where source-trained models tend to assign a disproportionate number of target samples to a small subset of classes. SFDA-SPF identifies prediction-dominant classes using the empirical distribution of target predictions. It then selects high-entropy samples from these classes to form a forget set, while the remaining samples constitute a retain set. A complementary forgetting loss reduces the probability assigned to the current predicted classes of forget samples, whereas a cross-entropy objective preserves the predictions of retained samples. The method also incorporates an auxiliary pixel-level localization loss using CAM-derived partial pseudo-labels from the frozen source model. Experiments on GlaS, CAMELYON16, CAMELYON17, and an additional EBHI corruption benchmark evaluate the method under cross-organ, cross-center, class-imbalance, and multi-class settings. The proposed method reports substantial localization improvements over several existing SFDA baselines.

Critical strengths
  1. A meaningful problem formulation for source-free adaptation under prediction bias.

The paper addresses an important challenge in source-free domain adaptation: unreliable pseudo-labels may reinforce dominant-class prediction bias under severe domain shifts.

Instead of relying exclusively on confident predictions or global diversity regularization, the proposed approach explicitly identifies uncertain samples from prediction-dominant classes and attempts to reconsider their current assignments. This provides a coherent motivation for addressing prediction collapse in histopathology localization.

  1. Encouraging localization performance across multiple WSOL architectures and domain shifts.

The experimental evaluation covers multiple histopathology datasets and three different WSOL architectures, including CNN- and transformer-based models.

The reported localization improvements are substantial in several challenging settings, particularly when conventional SFDA methods suffer from strong prediction bias. The paper also provides component ablations, analyses of dynamic set reconstruction, class-imbalance experiments, and supplementary evaluations under multi-class corruptions.

These experiments provide useful evidence that the proposed adaptation framework can improve localization under certain challenging domain shifts.

Critical weaknesses
  1. The evaluation relies on labeled target-domain validation, limiting the strength of the unsupervised adaptation claim.

Although the adaptation objective uses unlabeled target training data, Appendix B.2.2 explicitly states that hyperparameters and checkpoints are selected using a separate labeled target-domain validation set.

More importantly, Appendix A.1 reports that localization-validation images and their pixel-level annotations are used for localization model selection. This is particularly relevant because the proposed approach introduces multiple tunable components, including the forgetting ratio, the reconstruction period, and the weights of the classification and localization losses.

The authors clearly disclose this validation-assisted protocol, and the use of target validation data does not itself indicate test-set leakage. Nevertheless, the complete experimental procedure is not fully target-label-free, which substantially limits its applicability to settings where target-domain annotations are unavailable.

This distinction matters because source-free unsupervised domain adaptation is generally motivated by the absence of target labels. Moreover, using target localization annotations for model selection may favor configurations that improve pixel-level localization, making it difficult to determine whether similar improvements would be obtained under a fully unsupervised model-selection protocol.

A comparison using target-label-free hyperparameter and checkpoint selection would substantially strengthen the experimental evidence and clarify the practical applicability of SFDA-SPF.

  1. The specific contribution and generality of selective prediction forgetting are insufficiently isolated.

The proposed approach assumes that uncertain samples assigned to prediction-dominant classes are particularly likely to contain incorrect pseudo-labels. However, high prediction entropy does not necessarily imply misclassification, and prediction dominance does not necessarily indicate erroneous over-prediction.

The criterion based on prediction frequency exceeding 1/K cannot distinguish genuine target-domain class imbalance from prediction collapse without additional assumptions about the target label distribution. Consequently, the forgetting objective may suppress correct predictions in naturally imbalanced domains.

Although the paper includes component ablations, the experiments do not sufficiently isolate the benefit of selectively forgetting uncertain samples from dominant classes. In particular, comparisons against random forgetting within dominant classes, class-agnostic high-entropy forgetting, and confidence-based sample reweighting under matched training conditions would help determine whether the proposed selection principle is essential.

Furthermore, the auxiliary localization loss contributes substantially to the final localization performance. For example, Table 4 shows that combining the image-level forget and retain objectives yields only 13.1 PxAP, whereas adding the localization loss increases PxAP to 49.9. The comparisons would be more convincing if existing SFDA baselines were also evaluated with the same auxiliary localization supervision.

The generalization claims also require qualification. The proposed method does not consistently achieve the best classification accuracy, and its performance is less competitive in reverse cross-organ adaptation and certain multi-class corruption settings. On EBHI, it underperforms SHOT in average classification F1 and exhibits substantial degradation under severe blur and contrast shifts.

Finally, the absence of repeated-seed performance statistics makes it difficult to assess the stability of the reported improvements.

Together, these limitations weaken the evidence that the selective forgetting strategy itself is responsible for the reported gains and that its benefits generalize across different adaptation settings.

Questions for authors

Q1. Target-label-free model selection.

Appendix B.2.2 states that labeled target-domain validation data are used for hyperparameter and checkpoint selection, and Appendix A.1 indicates that pixel-level target annotations are available for localization model selection.

Could the authors clarify exactly which validation metrics are used to select each hyperparameter and checkpoint? How much performance degradation occurs when model selection is performed without any target-domain labels or localization masks?

Such an experiment would materially strengthen the paper's source-free unsupervised adaptation claims.

Q2. Validity of the dominant-class forgetting criterion.

How does the proposed method distinguish erroneous dominant-class prediction bias from genuine target-domain class imbalance?

Could the authors compare their entropy-based selection strategy against random forgetting within dominant classes and class-agnostic high-entropy forgetting, while keeping the forgetting ratio and other optimization settings identical?

These comparisons would help establish whether the proposed selection mechanism is necessary rather than merely the application of negative learning to uncertain target predictions.

Q3. Contribution of the auxiliary localization objective.

Table 4 indicates that the localization loss accounts for a substantial improvement in PxAP.

Could the authors evaluate representative SFDA baselines augmented with the same CAM-based pixel-level localization objective? This would allow a more controlled comparison and help distinguish the contribution of selective prediction forgetting from the benefits of additional localization supervision.

Q4. Robustness under severe multi-class shifts and repeated training runs.

The EBHI experiments show that SFDA-SPF is less effective under severe blur and contrast shifts, where some classes receive almost no source-model predictions.

Could the authors further analyze the failure mode in which classes disappear from the predicted target distribution? Additionally, could they report repeated-seed results with standard deviations or confidence intervals for representative cross-organ and cross-center experiments?

These results would help assess the robustness of the proposed method.

3 comments

Q1 — Hyperparameter selection and validation protocol (Sec. 4.1 and Appendix B.1–B.2.2)

We thank the reviewer for raising this important practical question. Our evaluation follows the validation-assisted protocol described in Sec. 4.1 and Appendix B.1–B.2.2. Hyperparameter selection remains an open practical challenge in SFDA [1,2,3,4]. Different approaches have therefore been explored. For example, Xu et al. [1, Appendix C.5] use a validation set with 300 manually annotated target samples; [2] report the best performance across different learning rates for each method on the test set. [3] also mentioned using a validation set based on classification for WSOL adaptation. To remain consistent with the state of the art, our main experiments follow the validation-assisted evaluation framework for SFDA of WSOL models introduced in prior work [3], with histology dataset construction following [4].

Selection metric: For our method and every baseline, all adaptation hyperparameters and checkpoints are selected with one criterion: image-level classification accuracy on the same held-out target validation split (Macro-F1 on EBHI). No target pixel-level annotation or PxAP score is used for hyperparameter tuning. The test set remains strictly independent and is never used for model selection, preventing data leakage between training, validation, and testing. This follows the validation-assisted protocol of [3] for SFDA of WSOL models. For the main experiments, the hyperparameter search ranges are reported in App. B.1–B.2. Configurations and checkpoints are selected using the image-level validation criterion described above. The five parameters are selected through grid search using the classification-based validation criterion: λR,λF∈{0.5,1.0}\lambda_R, \lambda_F\in\{0.5,1.0\}, L∈{0.5,1.0,5.0}L\in\{0.5,1.0,5.0\}, ρ∈{0.05,0.15,0.25}\rho\in\{0.05,0.15,0.25\} and m∈{5,15,35}m\in\{5,15,35\} for CAMELYON17 or {3,5}\{3,5\} for CAMELYON16 (Sec. 4.1 and Appendix B.1–B.2.2). Reduced search spaces are used for the additional experiments, as detailed in Appendix B.1.

Validation-set sizes: The image-level validation splits contain 18 samples for GlaS, 8,850 for C16, and 38–146 for the C17 centers (Table 7). For EBHI, the validation set contains 223 images, with 6–79 per class (Table 8).

Additional evaluation without target-label-based selection. Selection without any target label. To assess the performance of our method against baselines assuming no validation set for hyperparameter setup, we reran GlaS→C17-0 (PixelCAM) with a single fixed configuration for all methods: all loss weights set to 1, learning rate 10−410^{-4} decayed by 0.9 every 20 epochs, batch size 32, 100 epochs. SFDA-SPF uses default values ρ=0.1\rho=0.1 and m=5m=5, not tuned on this task. Test results, evaluated every 10 epochs:

Classification (CL)

Method102030405060708090100
SHOT40.4640.8441.2241.6041.2241.6041.2241.2241.6041.60
SFDA-DE40.4632.4429.3925.1925.1937.7945.8045.8045.8045.80
CDCL41.6041.2241.9840.8440.8441.2241.2241.2241.2241.22
ERL40.8441.6040.8441.2241.9841.2241.6041.2240.8441.22
RGV46.9550.0050.0050.0050.0050.0050.0050.0050.0050.00
Ours71.7679.7779.3976.7281.3070.2372.9073.6671.3764.50

Localization (PxAP)

Method102030405060708090100
SHOT22.4021.4721.1620.6320.4520.4019.5619.4319.8819.30
SFDA-DE19.1313.2314.2616.3624.3724.2623.0923.0923.0923.09
CDCL22.3821.1320.6120.3420.1219.9619.9019.8219.7519.79
ERL20.2219.8419.3319.0618.8918.5418.3118.1017.8617.94
RGV34.7633.6633.5833.6533.7533.8433.9033.9433.9633.97
Ours54.0457.8053.7946.9146.2742.6444.7744.5742.0641.58

References

  1. Xu et al., Revisiting Source-Free Domain Adaptation: A New Perspective via Uncertainty Control, ICLR 2025.
  2. Patel et al., Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement, ICLR 2025.
  3. Guichemerre et al., Source-Free Domain Adaptation of Weakly-Supervised Object Localization Models for Histology, CVPRW 2024.
  4. Rony et al., Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey, MELBA 2023.

W2 — Prediction dominance and target-class imbalance (Sec .3, Tab .3, 6 and 15 and Appendix E)

In practice, the true target class proportions are unknown, so SFDA-SPF relies only on observable prediction cues. As described in Sec. 3, the 1/K1/K threshold identifies prediction-dominant classes without estimating the true target priors or explicitly enforcing uniform predictions. Entropy serves as an empirical indicator of unreliability, allowing SFDA-SPF to prioritize assignments that are more likely to be incorrect.

Empirical evidence on severe shifts. App. E.1–E.2 support this choice under both cross-organ and inter-center shifts, across target class priors from 50:50 to 10:90. Misclassified samples concentrate in the mid- and high-entropy bins, whereas low-entropy predictions are mostly correct (Fig. 24), and accuracy decreases with entropy in most settings (Fig. 25). The controlled-imbalance experiments are conducted on GlaS→C17-0, combining a substantial histological domain shift with varying target class proportions. Figure 6 further shows that retained pseudo-labels become more accurate during adaptation, while the forget set contains a higher proportion of incorrect assignments.

Robustness to genuine imbalance. Tables 3 and 15 evaluate GlaS→C17-0 under both Normal-majority and Cancer-majority target distributions, up to 10:90, with a balanced test set. Across these imbalanced settings, SFDA-SPF maintains 79.0–86.2 CL and 43.7–58.8 PxAP, supporting its robustness to varying target class proportions.

To isolate the selection criterion, we additionally compare SFDA-SPF against random forgetting within dominant classes and class-agnostic high-entropy forgetting on GlaS→C17-0 (PixelCAM), using the same losses, forgetting ratio (ρ=0.1\rho=0.1). Results are reported for different Cancer:Normal ratios of the target training set. Added in Sec .3 Tab. 6.

Selection strategy50:5040:6030:7020:8010:90
Class-agnostic high entropy79.0 / 28.955.7 / 25.750.7 / 25.750.7 / 14.477.9 / 23.2
Random within dominant classes76.3 / 35.478.2 / 32.880.5 / 42.278.2 / 30.478.2 / 37.9
SFDA-SPF86.2 / 49.979.0 / 43.782.8 / 55.179.4 / 46.082.8 / 58.8

Class-agnostic forgetting may suppress correct but uncertain predictions in less frequently predicted classes, potentially harming adaptation. Random forgetting within dominant classes can still select incorrect assignments when prediction errors concentrate in these classes. However, entropy-based selection performs better across all evaluated class ratios, suggesting more effective identification of unreliable assignments.

W1 Q2 — Entropy/diversity objectives combined with localization (Sec 4.3)

The complementary contributions of prediction correction and localization supervision are examined in Sec. 4.3 (Table 4 and Fig. 5(a)). To directly address the comment, we additionally added Table 5 to Sec. 4.3 of the main paper, applying the same LlocL_{\mathrm{loc}} (CAM-derived foreground/background pseudo-labels from the frozen source model, without target pixel-level annotations) to all five SFDA baselines on C17-0→C17-2 (PixelCAM). SHOT combines entropy minimization and prediction diversity, while ERL uses entropy-based regularization.

MethodCLPXAP
SHOT77.031.6
SHOT + LlocL_{\mathrm{loc}}56.425.1
CDCL74.730.6
CDCL + LlocL_{\mathrm{loc}}72.135.8
SFDA-DE50.018.7
SFDA-DE + LlocL_{\mathrm{loc}}49.423.9
ERL58.445.6
ERL + LlocL_{\mathrm{loc}}71.542.2
RGV50.041.2
RGV + LlocL_{\mathrm{loc}}51.224.9
Ours81.658.1

Overall, SFDA-SPF exceeds the strongest localization-augmented baseline by 9.5 CL and 15.9 PxAP points, considering the best baseline separately for each metric.

Contribution of localization supervision. Adding LlocL_{\mathrm{loc}} does not consistently improve the baselines. It increases localization performance for CDCL and SFDA-DE but degrades it for SHOT and RGV. This behavior is consistent with the dependence of LlocL_{\mathrm{loc}} on image-level pseudo-label quality: when a prediction is incorrect, the CAM of the wrong class may introduce misleading pixel-level supervision.

Table 4 shows that combining LR+LFL_R+L_F without localization supervision achieves 84.7 CL but only 13.1 PxAP, compared with 86.1 CL / 49.9 PxAP for the complete objective. This highlights the importance of CAM-derived localization in cross-organ adaptation, where substantial differences in tissue architecture and cellular morphology make accurate localization particularly challenging.

W2 Q4 — Failure mode under severe prediction collapse (Appendix C.4)

Failure analysis. Appendix C.4 (Tables 19–20) reveals two regimes. When the source model predicts all six classes (Target Clean, Brightness), SFDA-SPF achieves the highest F1. Under Defocus Blur and Contrast, predictions collapse: High-grade IN, representing only 8.2% of target samples, receives 80.2% and 92.1% of predictions, respectively, while Low-grade IN (28.6%) receives none.

Our forgetting objective suppresses the current dominant-class prediction but does not identify the correct alternative class. Moreover, classes absent from the initial predictions receive no positive pseudo-label supervision at initialization, making their recovery particularly challenging. SHOT may be less affected because its diversity objective discourages prediction collapse.

Additional diversity ablation. To further investigate this limitation, we added the same marginal prediction-diversity regularizer used in SHOT to SFDA-SPF. Results are reported as F1 / PxAP (%).

Target shiftSFDA-SPFSFDA-SPF + KLSHOT
Stain-heavy57.9/82.766.0/83.1570.29/86.74
Dust72.4/86.781.0/81.1279.39/82.93
Air Bubble66.5/78.475.0/73.9278.07/77.84
Defocus Blur21.6/76.543.0/71.2842.24/73.25
Motion Blur29.3/72.935.0/70.9721.13/70.68
Gaussian Noise57.2/88.264.0/85.5760.94/82.92
Shot Noise52.6/86.857.0/86.4146.51/84.29
Brightness79.4/84.682.0/89.0370.88/81.09
Contrast12.1/62.625.0/66.7030.80/71.10

Adding diversity improves F1 across all nine evaluated conditions, particularly under Defocus Blur (21.6 → 43.0) and Contrast (12.1 → 25.0), and outperforms SHOT in F1 on six of nine conditions.

To assess robustness, we report the mean ± standard deviation over five random seeds on GlaS→C17-0 and C17-0→C17-2. We used a fixed learning rate of 10−410^{-4} and λ∈{0.5,1.0}\lambda \in \{0.5, 1.0\}, following the same hyperparameter selection protocol as in the main paper, based solely on classification accuracy on the target validation set.

C17-0 → C17-2 (5 seeds)

MethodCLPxAP
SHOT75.4 ± 1.231.2 ± 0.4
SFDA-DE48.8 ± 3.017.2 ± 3.1
CDCL74.9 ± 1.541.9 ± 11.4
ERL57.9 ± 1.947.8 ± 3.9
RGV50.0 ± 0.241.3 ± 0.1
Ours80.1 ± 1.150.4 ± 5.1

GlaS → C17-0 (5 seeds)

MethodCLPxAP
SHOT64.9 ± 15.618.8 ± 10.6
SFDA-DE46.3 ± 5.123.9 ± 9.1
CDCL50.0 ± 0.132.9 ± 0.2
ERL44.2 ± 0.621.4 ± 1.2
RGV50.0 ± 0.133.9 ± 0.1
Ours84.2 ± 2.344.0 ± 4.8
Reviewer_K7SAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SFDA-SPF, a source-free domain adaptation method for weakly supervised localization (WSOL) in histopathology that targets prediction collapse toward dominant classes under domain shift. It dynamically partitions target predictions into a "forget set" (uncertain, dominant-class predictions optimized via negative learning) and a "retain set" (confident predictions preserved via standard cross-entropy), while integrating an auxiliary CAM-derived pixel-level supervision loss. Evaluations across datasets (GlaS, CAMELYON16, CAMELYON17) using DeepMIL, SAT, and PixelCAM report improvements in classification accuracy and PxAP localization over several SFDA baselines.

Critical strengths
  1. The paper addresses a relevant and practical failure mode of pseudo-label-based SFDA, particularly class-prediction collapse under domain shift, moving beyond global diversity regularizers toward targeted instance selection.

  2. The adaptation strategy is successfully demonstrated across both CNN-based (ResNet-50) and transformer-based WSOL models (DeepMIL, SAT, PixelCAM), making it backbone agnostic.

  3. The empirical gains in the binary cross-organ/cross-center settings are pronounced, particularly for PixelCAM on benchmark shifts.

Critical weaknesses
  1. SFDA-SPF essentially combines entropy-based uncertainty selection, pseudo-label preservation, negative learning on selected pseudo-labels, and periodic sample reselection, all of which are closely related to established ideas in SFDA/noisy-label learning. The main novelty is therefore the specific dominant-class-driven composition of these components rather than a fundamentally new adaptation principle.

  2. Defining a "dominant" class strictly by a uniform frequency threshold (1/K1/K) contradicts the reality that true histopathology class proportions need not be uniform. As a result, genuinely prevalent target classes can be actively and erroneously suppressed.

  3. Softmax entropy is notoriously uncalibrated under severe distribution shift. That is why contemporary adaptation methods increasingly exploit feature-space consistency, neighborhood structure, augmentation agreement, prototypes, or calibrated uncertainty.

  4. Minimizing negative learning objective (−log⁡(1−py^-\log(1 - p_{\hat{y}})) only suppresses the current prediction without providing evidence for which alternative class is semantically correct. This is relatively benign for binary classification, but becomes substantially more ambiguous as the number of possible alternative classes increases.

  5. The ablation in Fig. 5(a) shows that combining the forget and retain objectives yields only 13.1 PxAP, whereas adding �0��0� increases localization to 49.9 PxAP. Much of the localization improvement comes from the auxiliary PixelCAM-derived pixel supervision loss (LLL_L) rather than the selective forgetting mechanism itself.

  6. The periodic re-mining of forget/retain sets every mm epochs relies on unstable set selection accuracies (refer Fig. 6), introducing unquantified optimization variance and sensitivity to the interval parameter.

  7. The proposed method specifically targets uncertainty and pseudo-label reliability. Hence, controlled comparison against stronger uncertainty-aware/consistency-based SFDA strategies is very important to showcase the effect of forgetting principle.

Questions for authors
  1. How does SFDA-SPF prevent the active suppression of a genuinely prevalent class when an inherently imbalanced target domain naturally exceeds the uniform 1/K1/K dominance threshold?

  2. Can stronger reliability signals (feature-space neighborhood consistency, augmentation agreement, prototype distance, or ensemble uncertainty) better identify erroneous predictions than entropy for SAT?

  3. Can the authors disentangle how much of the PxAP improvement stems from the forget/retain objectives versus the auxiliary PixelCAM localization loss �0��0�?

  4. In multi-class scenarios where severe shift totally eliminates a class from the pseudo-label pool, how does negative learning recover that semantic category?

  5. Clarify why negative learning is preferable to i) contemporary consistency or ii) representation-based correction of uncertain pseudo-labels.

  6. Given that Figure 6 shows forget-set selection accuracy fluctuating widely during training, how sensitive is final performance to the reconstruction period mm?

5 comments

W1 — Main novelty (Sec .3 and 4.3)

Under strong domain shifts, pseudo-label errors can concentrate in a small set of prediction-dominant classes, particularly among uncertain assignments. Conventional self-training reinforces these incorrect predictions. SFDA-SPF addresses this issue through class-conditional selective forgetting: prediction dominance identifies where potentially unreliable assignments accumulate, entropy determines which samples should be reconsidered, and LF=−log⁡(1−py^)L_F=-\log(1-p_{\hat y}) directly penalizes their current predicted labels. Meanwhile, LRL_R preserves the remaining assignments to limit model drift.

Difference from prior SFDA methods. SHOT [3], uses labels from aggregates of the model's own predictions, but under single-class dominance these aggregates are themselves dominated by that class and confirm the bias. Uncertainty-aware methods re-weight [1] or relax [2] the supervision of uncertain samples, changing how strongly they are supervised. [1] uses negative learning but always negates a class other than the pseudo-label whereas UCon-SFDA relaxes supervision through partial labels. In contrast, SFDA-SPF directs negative supervision at the current predicted class of selected uncertain samples within prediction-dominant classes.

The distinct contributions of the loss components and selection criteria are examined in Sec. 4.3 (Tables 4 and 6).

  1. Litrico et al., Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation, CVPR 2023.

  2. Xu et al., Revisiting Source-Free Domain Adaptation: A New Perspective via Uncertainty Control, ICLR 2025.

  3. Liang et al., Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation, ICML 2020.

W2 — Prediction dominance and target-class imbalance (Sec .3, Tab .3, 6 and 15 and Appendix E)

In practice, the true target class proportions are unknown, so SFDA-SPF relies only on observable prediction cues. As described in Sec. 3, the 1/K1/K threshold identifies prediction-dominant classes without estimating the true target priors or explicitly enforcing uniform predictions. Entropy serves as an empirical indicator of unreliability, allowing SFDA-SPF to prioritize assignments that are more likely to be incorrect.

Empirical evidence on severe shifts. App. E.1–E.2 support this choice under both cross-organ and inter-center shifts, across target class priors from 50:50 to 10:90. Misclassified samples concentrate in the mid- and high-entropy bins, whereas low-entropy predictions are mostly correct (Fig. 24), and accuracy decreases with entropy in most settings (Fig. 25). The controlled-imbalance experiments are conducted on GlaS→C17-0, combining a substantial histological domain shift with varying target class proportions. Figure 6 further shows that retained pseudo-labels become more accurate during adaptation, while the forget set contains a higher proportion of incorrect assignments.

Robustness to genuine imbalance. Tables 3 and 15 evaluate GlaS→C17-0 under both Normal-majority and Cancer-majority target distributions, up to 10:90, with a balanced test set. Across these imbalanced settings, SFDA-SPF maintains 79.0–86.2 CL and 43.7–58.8 PxAP, supporting its robustness to varying target class proportions.

To isolate the selection criterion, we additionally compare SFDA-SPF against random forgetting within dominant classes and class-agnostic high-entropy forgetting on GlaS→C17-0 (PixelCAM), using the same losses, forgetting ratio (ρ=0.1\rho=0.1). Results are reported for different Cancer:Normal ratios of the target training set. Added in Sec .3 Tab. 6.

Selection strategy50:5040:6030:7020:8010:90
Class-agnostic high entropy79.0 / 28.955.7 / 25.750.7 / 25.750.7 / 14.477.9 / 23.2
Random within dominant classes76.3 / 35.478.2 / 32.880.5 / 42.278.2 / 30.478.2 / 37.9
SFDA-SPF86.2 / 49.979.0 / 43.782.8 / 55.179.4 / 46.082.8 / 58.8

Class-agnostic forgetting may suppress correct but uncertain predictions in less frequently predicted classes, potentially harming adaptation. Random forgetting within dominant classes can still select incorrect assignments when prediction errors concentrate in these classes. However, entropy-based selection performs better across all evaluated class ratios, suggesting more effective identification of unreliable assignments.

W5 Q3. — Entropy/diversity objectives combined with localization (Sec 4.3)

The complementary contributions of prediction correction and localization supervision are examined in Sec. 4.3 (Table 4 and Fig. 5(a)). To directly address the comment, we additionally added Table 5 to Sec. 4.3 of the main paper, applying the same LlocL_{\mathrm{loc}} (CAM-derived foreground/background pseudo-labels from the frozen source model, without target pixel-level annotations) to all five SFDA baselines on C17-0→C17-2 (PixelCAM). SHOT combines entropy minimization and prediction diversity, while ERL uses entropy-based regularization.

MethodCLPxAP
SHOT77.031.6
SHOT + LlocL_{\mathrm{loc}}56.425.1
CDCL74.730.6
CDCL + LlocL_{\mathrm{loc}}72.135.8
SFDA-DE50.018.7
SFDA-DE + LlocL_{\mathrm{loc}}49.423.9
ERL58.445.6
ERL + LlocL_{\mathrm{loc}}71.542.2
RGV50.041.2
RGV + LlocL_{\mathrm{loc}}51.224.9
Ours81.658.1

Overall, SFDA-SPF exceeds the strongest localization-augmented baseline by 9.5 CL and 15.9 PxAP points, considering the best baseline separately for each metric.

W4 Q4 — Failure mode under severe prediction collapse (Appendix C.4)

Failure analysis. Appendix C.4 (Tables 19–20) reveals two regimes. When the source model predicts all six classes (Target Clean, Brightness), SFDA-SPF achieves the highest F1. Under Defocus Blur and Contrast, predictions collapse: High-grade IN, representing only 8.2% of target samples, receives 80.2% and 92.1% of predictions, respectively, while Low-grade IN (28.6%) receives none.

Our forgetting objective suppresses the current dominant-class prediction but does not identify the correct alternative class. Moreover, classes absent from the initial predictions receive no positive pseudo-label supervision at initialization, making their recovery particularly challenging. SHOT may be less affected because its diversity objective discourages prediction collapse.

Additional diversity ablation. To further investigate this limitation, we added the same marginal prediction-diversity regularizer used in SHOT to SFDA-SPF. Results are reported as F1 / PxAP (%).

Target shiftSFDA-SPFSFDA-SPF + KLSHOT
Stain-heavy57.9/82.766.0/83.1570.29/86.74
Dust72.4/86.781.0/81.1279.39/82.93
Air Bubble66.5/78.475.0/73.9278.07/77.84
Defocus Blur21.6/76.543.0/71.2842.24/73.25
Motion Blur29.3/72.935.0/70.9721.13/70.68
Gaussian Noise57.2/88.264.0/85.5760.94/82.92
Shot Noise52.6/86.857.0/86.4146.51/84.29
Brightness79.4/84.682.0/89.0370.88/81.09
Contrast12.1/62.625.0/66.7030.80/71.10

Adding diversity improves F1 across all nine evaluated conditions, particularly under Defocus Blur (21.6 → 43.0) and Contrast (12.1 → 25.0), and outperforms SHOT in F1 on six of nine conditions.

Q5 — Why forgetting rather than consistency- or representation-based correction

SFDA-SPF targets a specific failure mode: under strong domain shifts, source-model predictions concentrate in a few dominant classes, making pseudo-label supervision unreliable. Consistency- and representation-based methods exploit signals from augmented views, feature neighbourhoods, or class prototypes. While effective when these signals are reliable, they may preserve an existing prediction bias: augmented views can agree on the same incorrect dominant-class prediction, while neighbourhoods and prototypes may reflect the same bias.

Rather than relying on these signals to infer reliable supervision, SFDA-SPF selects high-entropy samples within prediction-dominant classes and directly penalizes their current assignments through LFL_F, without explicitly constructing replacement pseudo-labels. Meanwhile, LRL_R preserves the remaining predictions to limit model drift.

Neighborhood pseudo-label reliability (SAT, [GlaS → C17-1]). To assess whether neighborhood-based correction can mitigate prediction bias, we evaluated majority voting using the pseudo-labels of the kk nearest neighbors on the train set.

kOverall CLNormal CLCancer CL
351.6911.2692.12
551.4111.6391.18
1051.0311.2690.81

The source model achieves 51.97% CL, with 89.2% of target samples predicted as Cancer despite a balanced target distribution on the train set. Neighborhood voting does not improve accuracy (51.03–51.69%) and preserves a strong class bias, with only 11.26–11.63% accuracy for Normal versus 90.81–92.12% for Cancer. These findings illustrate that aggregating neighbouring pseudo-labels can preserve dominant-class prediction bias rather than correct it. SFDA-SPF instead suppresses selected uncertain predictions without relying on potentially biased neighbor-derived replacement labels.

In addition to the SFDA baselines already evaluated, we include NRC [1] (neighbourhood consistency) and FAUST [2] (augmentation agreement). C17-0 as source, PixelCAM, CL / PxAP:

PixelCAM (C17-0 source to all C17)

MethodC17-1C17-2C17-3C17-4
NRC29.1 / 24.556.2 / 39.893.6 / 47.858.1 / 52.8
FAUST49.4 / 23.949.3 / 25.392.0 / 50.774.1 / 51.4
Ours74.4 / 48.581.6 / 58.193.8 / 60.483.2 / 46.1

SAT (GlaS source to C17)

MethodC17-1C17-2C17-3C17-4
NRC44.1 / 19.172.9 / 31.565.4 / 16.256.0 / 16.4
FAUST44.2 / 20.156.5 / 29.453.2 / 21.041.3 / 20.7
Ours74.8 / 24.957.6 / 26.973.2 / 43.777.4 / 27.8

References

  1. Yang et al., Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation, NeurIPS 2021.
  2. Lee et al., Feature Alignment by Uncertainty and Self-Training for Source-Free Unsupervised Domain Adaptation, Neural Networks 2023.

Q6 — Sensitivity to the reconstruction interval (Figs. 4–6).

The retain and forget sets serve different purposes: the retain set provides positive supervision from relatively reliable predictions, whereas the forget set targets uncertain predictions that may be either correct or incorrect. Consequently, forget-set accuracy can fluctuate as model predictions evolve. These fluctuations do not necessarily indicate optimization instability, but motivate periodic re-mining to avoid relying on outdated assignments.

Interpretation of set fluctuations: On GlaS→C17-4 (940 samples), the forget set decreases from 221 samples (23.5%) at epoch 1 to 152 (16.2%) at epoch 10, with subsequent fluctuations (109 at epoch 90 and 163 at epoch 100). Similarly, on the larger CAMELYON16 dataset (26,564 samples), it decreases from 1,067 (4.0%) to 676 (2.5%) between epochs 1 and 15. These observations illustrate how the selected subsets evolve during adaptation.

To quantify sensitivity to the reconstruction interval, we evaluate m∈{5,15,35}m \in \{5,15,35\} with two learning rates (10−510^{-5} and 10−410^{-4}) on C17-0→C17-2 (Fig. 4). Both m=5m=5 and m=15m=15 achieve similar final classification accuracy (approximately 76%), with faster initial adaptation for m=5m=5, whereas m=35m=35 reduces accuracy to approximately 60–62%. Additionally, dynamic reconstruction improves classification accuracy from 57.9% to 80.6% and PxAP from 39.3% to 53.8% compared with static sets on GlaS→C17-3 (Fig. 5b).

Reviewer_jA3eOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SFDA-SPF, a source-free domain adaptation method for weakly supervised object localization (WSOL) in histopathology images. It identifies classes that receive disproportionately many target predictions and selectively forgets uncertain samples assigned to these dominant classes, while retaining other predictions to limit model drift. An auxiliary pixel-level loss further supports localization using CAM-derived pseudo-labels from the frozen source model. Experiments on GlaS, CAMELYON16, and CAMELYON17 show improvements in classification and localization over several SFDA baselines

Critical strengths
  • Promising empirical improvements and useful ablations: SFDA-SPF achieves the best average localization performance across the three evaluated WSOL models and reports substantial gains on several target domains. The ablation studies also suggest that the forgetting objective and pixel-level localization loss serve different purposes, providing evidence that both components contribute to the final performance.
Critical weaknesses
  • The novelty over existing SFDA methods needs to be established more clearly: The core idea of identifying uncertain samples from over-predicted classes and suppressing their current predictions is intuitive, but the paper does not sufficiently establish how fundamentally different this strategy is from existing uncertainty-aware adaptation and prediction-diversification methods. The contribution appears to lie mainly in combining dominant-class identification, selective forgetting, and retention, yet the paper does not clearly isolate which part represents the main methodological advance.

  • The method relies on a potentially unreliable assumption, and the validation protocol limits its practical appeal: A class receiving more than its uniform share of predictions is not necessarily over-predicted in the presence of genuine class imbalance. Likewise, uncertain samples assigned to dominant classes are not necessarily misclassified. Forgetting these predictions could therefore introduce additional errors, particularly under severe domain shifts or multiclass imbalance. Moreover, the supplementary material explicitly uses a separate labeled target validation split for hyperparameter and checkpoint selection. Although the target adaptation data remain unlabeled, this requirement introduces annotation costs and raises questions about how well the method would work in a strictly source-free, annotation-free deployment setting.

Questions for authors
  • How are the hyperparameters ρ\rho, mm, λR\lambda_R, λF\lambda_F, and λL\lambda_L selected in practice? The supplementary material states that a separate labeled validation split is used. How many labeled target samples are required, what metric determines the final configuration, and is exactly the same validation protocol applied to all baselines?

  • How does SFDA-SPF compare with a baseline that applies a standard prediction-diversity or entropy-based objective alongside the same localization loss? Such a comparison would help determine whether the gains come specifically from selecting and forgetting uncertain samples in dominant classes, rather than from the auxiliary localization objective or general prediction regularization.


After the rebuttal, some points in the results are still not clear to me. In the end, the method does not handle multi-class prediction collapse by itself and needs SHOT’s diversity loss to work in this setting. In the binary setting, it is mainly a sample selection rule for label flipping on top of existing losses. The novelty of the proposed method is limited. Thus, I keep my rating.

6 comments

Q1 — Hyperparameter selection and validation protocol (Sec. 4.1 and Appendix B.1–B.2.2)

We thank the reviewer for raising this important practical question. Our evaluation follows the validation-assisted protocol described in Sec. 4.1 and Appendix B.1–B.2.2. Hyperparameter selection remains an open practical challenge in SFDA [1,2,3,4]. Different approaches have therefore been explored. For example, Xu et al. [1, Appendix C.5] use a validation set with 300 manually annotated target samples; [2] report the best performance across different learning rates for each method on the test set. [3] also mentioned using a validation set based on classification for WSOL adaptation. To remain consistent with the state of the art, our main experiments follow the validation-assisted evaluation framework for SFDA of WSOL models introduced in prior work [3], with histology dataset construction following [4].

Selection metric: For our method and every baseline, all adaptation hyperparameters and checkpoints are selected with one criterion: image-level classification accuracy on the same held-out target validation split (Macro-F1 on EBHI). No target pixel-level annotation or PxAP score is used for hyperparameter tuning. The test set remains strictly independent and is never used for model selection, preventing data leakage between training, validation, and testing. This follows the validation-assisted protocol of [3] for SFDA of WSOL models. For the main experiments, the hyperparameter search ranges are reported in App. B.1–B.2. Configurations and checkpoints are selected using the image-level validation criterion described above. The five parameters are selected through grid search using the classification-based validation criterion: λR,λF∈{0.5,1.0}\lambda_R, \lambda_F\in\{0.5,1.0\}, L∈{0.5,1.0,5.0}L\in\{0.5,1.0,5.0\}, ρ∈{0.05,0.15,0.25}\rho\in\{0.05,0.15,0.25\}, and m∈{5,15,35}m\in\{5,15,35\} for CAMELYON17 or {3,5}\{3,5\} for CAMELYON16 (Sec. 4.1 and Appendix B.1–B.2.2). Reduced search spaces are used for the additional experiments, as detailed in Appendix B.1.

Validation-set sizes: The image-level validation splits contain 18 samples for GlaS, 8,850 for C16, and 38–146 for the C17 centers (Table 7). For EBHI, the validation set contains 223 images, with 6–79 per class (Table 8).

Additional evaluation without target-label-based selection. Selection without any target label. To assess the performance of our method against baselines assuming no validation set for hyperparameter setup, we reran GlaS→C17-0 (PixelCAM) with a single fixed configuration for all methods: all loss weights set to 1, learning rate 10−410^{-4} decayed by 0.9 every 20 epochs, batch size 32, 100 epochs. SFDA-SPF uses default values ρ=0.1\rho=0.1 and m=5m=5, not tuned on this task. Test results, evaluated every 10 epochs:

Classification (CL)

Method102030405060708090100
SHOT40.4640.8441.2241.6041.2241.6041.2241.2241.6041.60
SFDA-DE40.4632.4429.3925.1925.1937.7945.8045.8045.8045.80
CDCL41.6041.2241.9840.8440.8441.2241.2241.2241.2241.22
ERL40.8441.6040.8441.2241.9841.2241.6041.2240.8441.22
RGV46.9550.0050.0050.0050.0050.0050.0050.0050.0050.00
Ours71.7679.7779.3976.7281.3070.2372.9073.6671.3764.50

Localization (PxAP)

Method102030405060708090100
SHOT22.4021.4721.1620.6320.4520.4019.5619.4319.8819.30
SFDA-DE19.1313.2314.2616.3624.3724.2623.0923.0923.0923.09
CDCL22.3821.1320.6120.3420.1219.9619.9019.8219.7519.79
ERL20.2219.8419.3319.0618.8918.5418.3118.1017.8617.94
RGV34.7633.6633.5833.6533.7533.8433.9033.9433.9633.97
Ours54.0457.8053.7946.9146.2742.6444.7744.5742.0641.58

References

  1. Xu et al., Revisiting Source-Free Domain Adaptation: A New Perspective via Uncertainty Control, ICLR 2025.
  2. Patel et al., Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement, ICLR 2025.
  3. Guichemerre et al., Source-Free Domain Adaptation of Weakly-Supervised Object Localization Models for Histology, CVPRW 2024.
  4. Rony et al., Deep Weakly-Supervised Learning Methods for Classification and Localization in Histology Images: A Survey, MELBA 2023.

W1 Q2 — Entropy/diversity objectives combined with localization (Sec 4.3)

The complementary contributions of prediction correction and localization supervision are examined in Sec. 4.3 (Table 4 and Fig. 5(a)). To directly address the comment, we additionally added Table 5 to Sec. 4.3 of the main paper, applying the same LlocL_{\mathrm{loc}} (CAM-derived foreground/background pseudo-labels from the frozen source model, without target pixel-level annotations) to all five SFDA baselines on C17-0→C17-2 (PixelCAM). SHOT combines entropy minimization and prediction diversity, while ERL uses entropy-based regularization.

MethodCLPXAP
SHOT77.031.6
SHOT + LlocL_{\mathrm{loc}}56.425.1
CDCL74.730.6
CDCL + LlocL_{\mathrm{loc}}72.135.8
SFDA-DE50.018.7
SFDA-DE + LlocL_{\mathrm{loc}}49.423.9
ERL58.445.6
ERL + LlocL_{\mathrm{loc}}71.542.2
RGV50.041.2
RGV + LlocL_{\mathrm{loc}}51.224.9
Ours81.658.1

Overall, SFDA-SPF exceeds the strongest localization-augmented baseline by 9.5 CL and 15.9 PxAP points, considering the best baseline separately for each metric.

Contribution of localization supervision. Adding LlocL_{\mathrm{loc}} does not consistently improve the baselines. It increases localization performance for CDCL and SFDA-DE but degrades it for SHOT and RGV. This behavior is consistent with the dependence of LlocL_{\mathrm{loc}} on image-level pseudo-label quality: when a prediction is incorrect, the CAM of the wrong class may introduce misleading pixel-level supervision.

Table 4 shows that combining LR+LFL_R+L_F without localization supervision achieves 84.7 CL but only 13.1 PxAP, compared with 86.1 CL / 49.9 PxAP for the complete objective. This highlights the importance of CAM-derived localization in cross-organ adaptation, where substantial differences in tissue architecture and cellular morphology make accurate localization particularly challenging.

Thank you for your responses. Some of my concerns have been addressed. However, my concern about novelty hasn't been fully addressed. Can you compare your method with Litrico et al. [1] and UCon-SFDA [2]?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.