- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Reliability-Aware Environment Discovery (RAE), a method for improving group robustness under subpopulation shift without requiring group annotations. RAE extends error-based environment discovery by incorporating margin-based prediction uncertainty to identify correctly classified but potentially vulnerable samples. It employs twin discovery networks with uncertainty-guided dynamic label updates and auxiliary regularization, then constructs surrogate groups using prediction errors and uncertainty scores for subsequent GroupDRO training. Experiments on four benchmarks evaluate the method against existing approaches in terms of worst-group accuracy.
Critical strengths
The paper addresses an important limitation of error-based environment discovery by considering correctly classified but uncertain samples. The combination of prediction errors and uncertainty achieves reasonable worst-group accuracy across several benchmarks.
Critical weaknesses
-
(Limited methodological novelty and insufficient differentiation from prior work.) RAE largely builds upon the XRM framework, retaining its twin-network discovery architecture and downstream robust learning pipeline. Its main modifications involve incorporating margin-based uncertainty into environment construction, introducing additional regularization terms, and modifying the dynamic-label update mechanism. Given this close resemblance to XRM, the experimental comparisons should also include a simpler baseline that retains XRM's original discovery training procedure and modifies only the final environment assignment by combining prediction errors with high uncertainty, as in Eq. (17). Moreover, leveraging predictive uncertainty to improve robustness against spurious correlations has already been explored in prior work, such as END [1] and Evidential Alignment [2]. In particular, Evidential Alignment also utilizes uncertainty to improve group robustness without group annotations. Although the specific methodologies differ from RAE, the general idea of incorporating uncertainty to mitigate spurious correlations is not novel. Furthermore, the limitation discussed in Section 3.2 is specific to methods relying on binary misclassification indicators, whereas existing robust learning methods such as LfF [3] and AFR [4] already employ continuous loss-based signals that can distinguish among correctly classified examples.
-
(Insufficient justification for the proposed discovery training procedure.) While incorporating uncertainty into error-based environment discovery is a reasonable motivation, the necessity of the additional training mechanisms in Eqs. (11)–(15) is not sufficiently justified. In particular, it is unclear why uncertainty regularization and stochastic dynamic-label updates should improve the identification of samples vulnerable to spurious correlations. The dynamic-label update is especially difficult to motivate. Since the interpolated predictions are converted into hard labels via argmax and , the update effectively selects between the sample's own opposite-branch prediction and a randomly paired sample's prediction with equal probability whenever they differ. It is unclear how this random label assignment relates to spurious vulnerability. Although Table 5 shows performance improvements from these components, it does not establish the underlying mechanism.
-
(Unclear hyperparameter tuning and model selection under the annotation-free setting.) The paper emphasizes that RAE requires no group annotations during training or validation, yet the hyperparameter tuning and model selection procedures are not adequately explained. In Phase I, the selection criteria for , , , and are unclear. In particular, directly determines the uncertainty threshold used for environment assignment and also affects the training objectives. The paper uses dataset-specific values (e.g., 0.05 for Waterbirds and 0.25 for CelebA) without explaining how they were selected or providing a sensitivity analysis. Additionally, in Phase II, the paper does not specify how the final GroupDRO checkpoint is selected without validation group annotations. Since selecting the checkpoint with the highest validation worst-group accuracy typically requires group information, this omission makes it difficult to verify the claimed annotation-free evaluation protocol.
-
(Inconsistent experimental results and unexplained baseline discrepancies.) The reported results for RAE are inconsistent across Tables 3, 4, and 5. Notably, the Waterbirds worst-group accuracy is 88.0% in Table 3 but 87.54% in Tables 4 and 5. This discrepancy is particularly concerning because it changes the apparent comparison with XRM (87.7%), with RAE outperforming XRM in Table 3 (88.0%) but underperforming it in Tables 4 and 5 (87.54%). The authors should clarify whether these discrepancies arise from different experimental settings, random seeds, or evaluation procedures. More importantly, the reproduced XRM results in Table 3 are substantially lower than those originally reported in the XRM paper [5]. For example, the worst-group accuracies on CelebA and MultiNLI decrease from 89.1% to 82.4% and from 72.1% to 60.3%, respectively. Although Appendix A.2 states that the authors reran the original implementations, these substantial discrepancies are not explained. The authors should clarify the reasons for the degraded baseline performance and report both the original and reproduced results to ensure a transparent and fair comparison. Furthermore, Table 3 states that standard deviations are reported in "Appendix ??", but no such values are provided. Consistent experimental results and standard deviations are necessary to assess the reliability of the reported improvements.
-
(Insufficiently specified experimental protocol for Figure 1.) Figure 1 provides the main empirical motivation for using prediction reliability, yet its experimental setup is not sufficiently explained. In particular, the precise meaning of the conflict parameter α and how it modifies the input are not defined in the main text or figure caption, nor is the reader explicitly directed to Appendix B. This makes the comparisons between anchor values α₀ and evaluation values α₁ difficult to interpret. Furthermore, Appendix B describes a dataset construction using α ∈ {0, 0.25, 0.5}, whereas Figure 1 and Table 1 use values such as 0.1, 0.2, 0.3, and 0.4. The relationship between these settings is not explained. Given the central role of Figure 1 in motivating the proposed uncertainty criterion, the experimental protocol and the distinction between these α settings should be clarified.
-
(Insufficient consideration of computational efficiency.) The paper overlooks the well-established line of efficient group-robust learning methods based on last-layer retraining, such as AFR [4] and SELF [6], both introduced in 2023. RAE requires training twin discovery networks followed by a separate GroupDRO training stage, introducing additional computational overhead. Although Appendix A.2 provides approximate training times, it does not establish a controlled end-to-end computational comparison against these efficient alternatives. Given that RAE does not consistently outperform existing methods, a rigorous evaluation of its computational cost and performance trade-offs against efficient robust learning approaches is necessary.
Minor Comments
- The Related Work section focuses primarily on environment discovery, with limited discussion of recent advances in mitigating spurious correlations. For example, recent studies have explored data pruning [7] and extensions of DRO ambiguity sets [8]. Discussing these different approaches would provide a more comprehensive overview of recent developments in this field.
- In Eq. (3), is used without being explicitly defined. The notation should be clarified or made consistent with the previously introduced .
- The implementation code is not provided in the supplementary materials.
Overall, I find the paper insufficiently convincing in terms of methodological novelty, justification of the proposed approach, and empirical validation. These concerns substantially undermine the paper's overall contribution and reliability. I therefore recommend clear rejection.
References
[1] Oh et al., Improving Group Robustness under Noisy Labels Using Predictive Uncertainty, arXiv:2212.07026, 2022.
[2] Ye et al., Improving Group Robustness on Spurious Correlation via Evidential Alignment, KDD 2025.
[3] Nam et al., Learning from Failure: De-biasing Classifier from Biased Classifier, NeurIPS 2020.
[4] Qiu et al., Simple and Fast Group Robustness by Automatic Feature Reweighting, ICML 2023.
[5] Pezeshki et al., Discovering Environments with XRM, ICML 2024.
[6] LaBonte et al., Towards Last-layer Retraining for Group Robustness with Fewer Annotations, NeurIPS 2023.
[7] Mulchandani and Kim, Severing Spurious Correlations with Data Pruning, ICLR 2025.
[8] Jo et al., Mitigating Spurious Correlation via Distributionally Robust Learning with Hierarchical Ambiguity Sets, ICLR 2026.
Questions for authors
Please see the weaknesses.
9 comments
Response to Reviewer uL8c (2/5)
W2. Motivation and Effectiveness of Reliability-Guided Discovery
Response. We agree that component ablations alone do not explain how reliability-guided label updates improve environment discovery. As described in Sec. 3.3, RAE selectively updates uncertain examples using opposite-branch predictions (Eqs. 14–15). This aims to prevent potentially vulnerable examples from being confidently fitted to their original labels, preserving prediction errors or high uncertainty as signals for subsequent environment discovery. The uncertainty quantile penalty and soft margin regularizer (Eqs. 11–13) shape the uncertainty scores used to guide these updates.
To further examine the update mechanisms beyond the component ablations in Table 5, we evaluated three variants (Table R3): w/o prediction interpolation (λ = 1) removes interpolation with randomly paired predictions; Uniform update selection replaces uncertainty-guided selection with uniform random selection; and w/o dynamic-label updates retains the original training labels throughout discovery training.
Table R3. Ablation of dynamic-label update mechanisms (test worst-group accuracy; mean ± standard deviation over three seeds).
| Dynamic-label update variant | Waterbirds | CelebA |
|---|---|---|
| RAE (full) | 88.0 ± 0.0 | 87.9 ± 0.7 |
| w/o prediction interpolation (λ = 1) | 87.0 ± 0.7 | 89.0 ± 0.4 |
| Uniform update selection | 86.9 ± 1.4 | 81.8 ± 4.5 |
| w/o dynamic-label updates | 75.2 ± 2.1 | 72.0 ± 2.2 |
Removing dynamic-label updates reduces worst-group accuracy by 12.9 points on Waterbirds and 15.9 points on CelebA. Uncertainty-guided selection outperforms uniform selection by 1.1 and 6.0 points, respectively, whereas prediction interpolation has mixed effects across datasets. The corresponding dynamic-label ablations are included in Appendix E (Table 12) of the revised manuscript.
To further examine the discovered environments, we analyzed the Phase-I networks using ground-truth groups only for post-hoc analysis (Table R4).
Table R4. Characteristics of inferred vulnerable environments across ablation variants (Phase-I). Flag denotes the percentage of training examples assigned to the vulnerable environment, and precision denotes the minority proportion within that environment (computed within the relevant class for CelebA). All values are percentages.
| Variant | Waterbirds Flag | Waterbirds Precision | CelebA Flag | CelebA Precision |
|---|---|---|---|---|
| RAE (full) | 11.3 | 40.9 | 41.6 | 47.9 |
| w/o prediction interpolation (λ = 1) | 9.7 | 46.0 | 37.2 | 29.1 |
| Uniform update selection | 11.0 | 40.9 | 39.3 | 44.2 |
| w/o dynamic-label updates | 7.0 | 44.2 | 27.5 | 15.4 |
| w/o uncertainty regularizers | 22.5 | 18.1 | 42.0 | 48.9 |
Without dynamic-label updates, the inferred environment becomes smaller on both datasets (Waterbirds: 11.3% → 7.0%; CelebA: 41.6% → 27.5%), with a substantial reduction in minority precision on CelebA (47.9% → 15.4%). Uniform selection yields similar environment sizes and minority precision to RAE, despite lower worst-group accuracy. On CelebA, both selection strategies also yield comparable label-change rates (minority: 44.5% vs. 44.3%; majority: 29.8% vs. 30.5%), suggesting that performance differences are not explained by the frequency of label changes alone.
Removing both uncertainty regularizers increases the flagged fraction (11.3% → 22.5%) and reduces minority precision (40.9% → 18.1%) on Waterbirds, whereas the corresponding environment statistics remain similar on CelebA. These findings support the contribution of reliability-guided discovery training while indicating that the effects of prediction interpolation and regularization vary across datasets. We have added these analyses in Appendix E (Tables 12 and 14) and clarified the motivation in Section 3.3 of the revised manuscript.
Response to Reviewer uL8c (3/5)
W3. Hyperparameter and Checkpoint Selection
Response. Ground-truth group labels were not used for training, hyperparameter tuning, or checkpoint selection in either phase. They were used only for test worst-group evaluation and post-hoc analysis. Optimization hyperparameters, including the GroupDRO step size η, follow XRM's dataset-specific defaults (Table 6). All RAE-specific hyperparameters except are fixed across datasets ( = 0.01, = 1, k = 20, = 0.5).
is selected from {0.02, 0.05, 0.1, 0.25} based on worst-group accuracy over the validation groups inferred in Phase I. This yields = 0.05 for Waterbirds and MultiNLI and = 0.25 for CelebA and CivilComments. The final GroupDRO checkpoint is selected by the highest validation worst-group accuracy over the same inferred groups. We report the sensitivity to below and have included it in Appendix E (Table 13) of the revised manuscript.
Table R4. Sensitivity to (test worst-group accuracy; mean ± standard deviation over three seeds).
| Waterbirds | CelebA | |
|---|---|---|
| 0.02 | 84.3 ± 1.4 | 74.1 ± 0.9 |
| 0.05 | 88.0 ± 0.0 | 77.0 ± 0.9 |
| 0.1 | 85.8 ± 0.2 | 84.1 ± 2.9 |
| 0.25 | 81.9 ± 1.0 | 87.9 ± 0.7 |
Response to Reviewer uL8c (1/5)
We thank the reviewer for the thoughtful feedback. The comments helped us strengthen the paper, and we address each with new experiments and clarifications below.
W1. Methodological Novelty Beyond Existing Robustness Methods
Response. RAE shares XRM's twin-network architecture and GroupDRO training on inferred groups, but differs in how prediction reliability is incorporated into discovery training and environment construction.
Compared with XRM, which constructs environments from prediction errors, RAE uses the prediction margin (s = 1 − (p₁ − p₂)) to guide label updates and regularization during discovery training (Eqs. 11–15), and includes correctly classified low-margin examples in the vulnerable environment (Eq. 17). To isolate these contributions, we apply RAE's environment assignment to XRM's discovery networks without modifying their training procedure (Table R1).
Table R1. Discovery training × environment assignment (test worst-group accuracy, three seeds)
Modifying XRM's assignment alone does not reproduce RAE's performance. On XRM's networks, the new assignment lowers worst-group accuracy (Waterbirds 87.7 → 82.8; CelebA 82.4 → 81.3). Under XRM's discovery training, the error set already has a high share of minority examples (64% of the error set on Waterbirds; 38% of the blond-class errors on CelebA), so the added low-margin examples are mostly majority examples (18% minority on Waterbirds, 1.3% on CelebA), which may lower the share of minority examples in the vulnerable environment. With RAE's discovery training, the same assignment gives 88.0 and 87.9.
Compared with uncertainty-based methods, END [1] oversamples high-entropy examples for retraining, while Evidential Alignment [2] uses uncertainty to upweight misclassified examples. RAE instead incorporates reliability-guided regularization and label updates into discovery training, then constructs environments using both prediction errors and correctly classified low-margin examples. On RAE's discovery networks, uncertainty-only assignment achieves 61.1 worst-group accuracy on Waterbirds, compared with 88.0 when combined with errors (Table 4, W4). RAE also outperforms Evidential Alignment in worst-group accuracy across all four datasets (79.0 vs. 67.3 on average).
Compared with loss-based methods, LfF [3] and AFR [4] use continuous loss signals for sample reweighting. In binary classification, true-class loss and the prediction margin rank correctly classified examples identically. Thus, RAE's distinction is not the ranking signal itself, but its use in discovery training and environment construction. A held-out loss quantile at matched sample counts selects nearly identical examples (Jaccard ≥ 0.97 on binary datasets) and achieves comparable worst-group accuracy on Waterbirds (87.3 ± 1.9 vs. 88.0). We have clarified these methodological distinctions in Section 2 and Appendix A (Additional Related Work) of the revised manuscript.