acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Aware Environment Discovery for Subpopulation Robustness

Abstract

Machine learning models often fail under subpopulation shift, where subgroup proportions differ between training and test environments. In this setting, empirical risk minimization may rely on spurious correlations and underperform on minority subgroups. Existing group-robust methods address this issue by optimizing worst-group risk, but typically require subgroup annotations or rely on error-based environment discovery. These approaches identify vulnerable samples through observed prediction errors, overlooking correctly classified examples that may become misclassified when spurious cues change. To address this limitation, we propose Reliability-Aware Environment Discovery (RAE), an annotation-free framework that supplements prediction errors with margin-based uncertainty to discover candidate vulnerable environments. RAE employs twin discovery networks with reliability-guided label updates and auxiliary regularization, and combines prediction errors with uncertainty-based selection to construct environment partitions. Experiments on vision and language benchmarks demonstrate competitive worst-group performance. Further analyses show that prediction reliability complements error-based discovery by identifying currently correct but unreliable examples that would otherwise be excluded.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Subpopulation shift, Subgroup robustness, Spurious Correlation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uL8cOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Reliability-Aware Environment Discovery (RAE), a method for improving group robustness under subpopulation shift without requiring group annotations. RAE extends error-based environment discovery by incorporating margin-based prediction uncertainty to identify correctly classified but potentially vulnerable samples. It employs twin discovery networks with uncertainty-guided dynamic label updates and auxiliary regularization, then constructs surrogate groups using prediction errors and uncertainty scores for subsequent GroupDRO training. Experiments on four benchmarks evaluate the method against existing approaches in terms of worst-group accuracy.

Critical strengths

The paper addresses an important limitation of error-based environment discovery by considering correctly classified but uncertain samples. The combination of prediction errors and uncertainty achieves reasonable worst-group accuracy across several benchmarks.

Critical weaknesses
  1. (Limited methodological novelty and insufficient differentiation from prior work.) RAE largely builds upon the XRM framework, retaining its twin-network discovery architecture and downstream robust learning pipeline. Its main modifications involve incorporating margin-based uncertainty into environment construction, introducing additional regularization terms, and modifying the dynamic-label update mechanism. Given this close resemblance to XRM, the experimental comparisons should also include a simpler baseline that retains XRM's original discovery training procedure and modifies only the final environment assignment by combining prediction errors with high uncertainty, as in Eq. (17). Moreover, leveraging predictive uncertainty to improve robustness against spurious correlations has already been explored in prior work, such as END [1] and Evidential Alignment [2]. In particular, Evidential Alignment also utilizes uncertainty to improve group robustness without group annotations. Although the specific methodologies differ from RAE, the general idea of incorporating uncertainty to mitigate spurious correlations is not novel. Furthermore, the limitation discussed in Section 3.2 is specific to methods relying on binary misclassification indicators, whereas existing robust learning methods such as LfF [3] and AFR [4] already employ continuous loss-based signals that can distinguish among correctly classified examples.

  2. (Insufficient justification for the proposed discovery training procedure.) While incorporating uncertainty into error-based environment discovery is a reasonable motivation, the necessity of the additional training mechanisms in Eqs. (11)–(15) is not sufficiently justified. In particular, it is unclear why uncertainty regularization and stochastic dynamic-label updates should improve the identification of samples vulnerable to spurious correlations. The dynamic-label update is especially difficult to motivate. Since the interpolated predictions are converted into hard labels via argmax and λ∼Beta(1,1)\lambda \sim \mathrm{Beta}(1,1), the update effectively selects between the sample's own opposite-branch prediction and a randomly paired sample's prediction with equal probability whenever they differ. It is unclear how this random label assignment relates to spurious vulnerability. Although Table 5 shows performance improvements from these components, it does not establish the underlying mechanism.

  3. (Unclear hyperparameter tuning and model selection under the annotation-free setting.) The paper emphasizes that RAE requires no group annotations during training or validation, yet the hyperparameter tuning and model selection procedures are not adequately explained. In Phase I, the selection criteria for β\beta, λq\lambda_q, λr\lambda_r, and kk are unclear. In particular, β\beta directly determines the uncertainty threshold used for environment assignment and also affects the training objectives. The paper uses dataset-specific values (e.g., 0.05 for Waterbirds and 0.25 for CelebA) without explaining how they were selected or providing a sensitivity analysis. Additionally, in Phase II, the paper does not specify how the final GroupDRO checkpoint is selected without validation group annotations. Since selecting the checkpoint with the highest validation worst-group accuracy typically requires group information, this omission makes it difficult to verify the claimed annotation-free evaluation protocol.

  4. (Inconsistent experimental results and unexplained baseline discrepancies.) The reported results for RAE are inconsistent across Tables 3, 4, and 5. Notably, the Waterbirds worst-group accuracy is 88.0% in Table 3 but 87.54% in Tables 4 and 5. This discrepancy is particularly concerning because it changes the apparent comparison with XRM (87.7%), with RAE outperforming XRM in Table 3 (88.0%) but underperforming it in Tables 4 and 5 (87.54%). The authors should clarify whether these discrepancies arise from different experimental settings, random seeds, or evaluation procedures. More importantly, the reproduced XRM results in Table 3 are substantially lower than those originally reported in the XRM paper [5]. For example, the worst-group accuracies on CelebA and MultiNLI decrease from 89.1% to 82.4% and from 72.1% to 60.3%, respectively. Although Appendix A.2 states that the authors reran the original implementations, these substantial discrepancies are not explained. The authors should clarify the reasons for the degraded baseline performance and report both the original and reproduced results to ensure a transparent and fair comparison. Furthermore, Table 3 states that standard deviations are reported in "Appendix ??", but no such values are provided. Consistent experimental results and standard deviations are necessary to assess the reliability of the reported improvements.

  5. (Insufficiently specified experimental protocol for Figure 1.) Figure 1 provides the main empirical motivation for using prediction reliability, yet its experimental setup is not sufficiently explained. In particular, the precise meaning of the conflict parameter α and how it modifies the input are not defined in the main text or figure caption, nor is the reader explicitly directed to Appendix B. This makes the comparisons between anchor values α₀ and evaluation values α₁ difficult to interpret. Furthermore, Appendix B describes a dataset construction using α ∈ {0, 0.25, 0.5}, whereas Figure 1 and Table 1 use values such as 0.1, 0.2, 0.3, and 0.4. The relationship between these settings is not explained. Given the central role of Figure 1 in motivating the proposed uncertainty criterion, the experimental protocol and the distinction between these α settings should be clarified.

  6. (Insufficient consideration of computational efficiency.) The paper overlooks the well-established line of efficient group-robust learning methods based on last-layer retraining, such as AFR [4] and SELF [6], both introduced in 2023. RAE requires training twin discovery networks followed by a separate GroupDRO training stage, introducing additional computational overhead. Although Appendix A.2 provides approximate training times, it does not establish a controlled end-to-end computational comparison against these efficient alternatives. Given that RAE does not consistently outperform existing methods, a rigorous evaluation of its computational cost and performance trade-offs against efficient robust learning approaches is necessary.

Minor Comments

  1. The Related Work section focuses primarily on environment discovery, with limited discussion of recent advances in mitigating spurious correlations. For example, recent studies have explored data pruning [7] and extensions of DRO ambiguity sets [8]. Discussing these different approaches would provide a more comprehensive overview of recent developments in this field.
  2. In Eq. (3), DtrD_{\mathrm{tr}} is used without being explicitly defined. The notation should be clarified or made consistent with the previously introduced PtrP_{\mathrm{tr}}.
  3. The implementation code is not provided in the supplementary materials.

Overall, I find the paper insufficiently convincing in terms of methodological novelty, justification of the proposed approach, and empirical validation. These concerns substantially undermine the paper's overall contribution and reliability. I therefore recommend clear rejection.


References

[1] Oh et al., Improving Group Robustness under Noisy Labels Using Predictive Uncertainty, arXiv:2212.07026, 2022.

[2] Ye et al., Improving Group Robustness on Spurious Correlation via Evidential Alignment, KDD 2025.

[3] Nam et al., Learning from Failure: De-biasing Classifier from Biased Classifier, NeurIPS 2020.

[4] Qiu et al., Simple and Fast Group Robustness by Automatic Feature Reweighting, ICML 2023.

[5] Pezeshki et al., Discovering Environments with XRM, ICML 2024.

[6] LaBonte et al., Towards Last-layer Retraining for Group Robustness with Fewer Annotations, NeurIPS 2023.

[7] Mulchandani and Kim, Severing Spurious Correlations with Data Pruning, ICLR 2025.

[8] Jo et al., Mitigating Spurious Correlation via Distributionally Robust Learning with Hierarchical Ambiguity Sets, ICLR 2026.

Questions for authors

Please see the weaknesses.

9 comments
AuthorsOpenReview ↗

Response to Reviewer uL8c (1/5)

We thank the reviewer for the thoughtful feedback. The comments helped us strengthen the paper, and we address each with new experiments and clarifications below.

W1. Methodological Novelty Beyond Existing Robustness Methods

Response. RAE shares XRM's twin-network architecture and GroupDRO training on inferred groups, but differs in how prediction reliability is incorporated into discovery training and environment construction.

Compared with XRM, which constructs environments from prediction errors, RAE uses the prediction margin (s = 1 − (p₁ − p₂)) to guide label updates and regularization during discovery training (Eqs. 11–15), and includes correctly classified low-margin examples in the vulnerable environment (Eq. 17). To isolate these contributions, we apply RAE's environment assignment to XRM's discovery networks without modifying their training procedure (Table R1).

Table R1. Discovery training × environment assignment (test worst-group accuracy, three seeds)

Discovery trainingWaterbirds — Errors only (Avg)Waterbirds — Errors only (Worst)Waterbirds — Errors ∪ Uncertainty (Avg)Waterbirds — Errors ∪ Uncertainty (Worst)CelebA — Errors only (Avg)CelebA — Errors only (Worst)CelebA — Errors ∪ Uncertainty (Avg)CelebA — Errors ∪ Uncertainty (Worst)
XRM89.2 ± 0.887.7 ± 1.185.2 ± 0.982.8 ± 1.588.2 ± 0.382.4 ± 0.489.4 ± 0.381.3 ± 3.3
RAE88.9 ± 0.584.6 ± 1.790.3 ± 0.188.0 ± 0.091.8 ± 1.277.6 ± 3.690.0 ± 0.287.9 ± 0.7

Modifying XRM's assignment alone does not reproduce RAE's performance. On XRM's networks, the new assignment lowers worst-group accuracy (Waterbirds 87.7 → 82.8; CelebA 82.4 → 81.3). Under XRM's discovery training, the error set already has a high share of minority examples (64% of the error set on Waterbirds; 38% of the blond-class errors on CelebA), so the added low-margin examples are mostly majority examples (18% minority on Waterbirds, 1.3% on CelebA), which may lower the share of minority examples in the vulnerable environment. With RAE's discovery training, the same assignment gives 88.0 and 87.9.

Compared with uncertainty-based methods, END [1] oversamples high-entropy examples for retraining, while Evidential Alignment [2] uses uncertainty to upweight misclassified examples. RAE instead incorporates reliability-guided regularization and label updates into discovery training, then constructs environments using both prediction errors and correctly classified low-margin examples. On RAE's discovery networks, uncertainty-only assignment achieves 61.1 worst-group accuracy on Waterbirds, compared with 88.0 when combined with errors (Table 4, W4). RAE also outperforms Evidential Alignment in worst-group accuracy across all four datasets (79.0 vs. 67.3 on average).

Compared with loss-based methods, LfF [3] and AFR [4] use continuous loss signals for sample reweighting. In binary classification, true-class loss and the prediction margin rank correctly classified examples identically. Thus, RAE's distinction is not the ranking signal itself, but its use in discovery training and environment construction. A held-out loss quantile at matched sample counts selects nearly identical examples (Jaccard ≥ 0.97 on binary datasets) and achieves comparable worst-group accuracy on Waterbirds (87.3 ± 1.9 vs. 88.0). We have clarified these methodological distinctions in Section 2 and Appendix A (Additional Related Work) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer uL8c (2/5)

W2. Motivation and Effectiveness of Reliability-Guided Discovery

Response. We agree that component ablations alone do not explain how reliability-guided label updates improve environment discovery. As described in Sec. 3.3, RAE selectively updates uncertain examples using opposite-branch predictions (Eqs. 14–15). This aims to prevent potentially vulnerable examples from being confidently fitted to their original labels, preserving prediction errors or high uncertainty as signals for subsequent environment discovery. The uncertainty quantile penalty and soft margin regularizer (Eqs. 11–13) shape the uncertainty scores used to guide these updates.

To further examine the update mechanisms beyond the component ablations in Table 5, we evaluated three variants (Table R3): w/o prediction interpolation (λ = 1) removes interpolation with randomly paired predictions; Uniform update selection replaces uncertainty-guided selection with uniform random selection; and w/o dynamic-label updates retains the original training labels throughout discovery training.

Table R3. Ablation of dynamic-label update mechanisms (test worst-group accuracy; mean ± standard deviation over three seeds).

Dynamic-label update variantWaterbirdsCelebA
RAE (full)88.0 ± 0.087.9 ± 0.7
w/o prediction interpolation (λ = 1)87.0 ± 0.789.0 ± 0.4
Uniform update selection86.9 ± 1.481.8 ± 4.5
w/o dynamic-label updates75.2 ± 2.172.0 ± 2.2

Removing dynamic-label updates reduces worst-group accuracy by 12.9 points on Waterbirds and 15.9 points on CelebA. Uncertainty-guided selection outperforms uniform selection by 1.1 and 6.0 points, respectively, whereas prediction interpolation has mixed effects across datasets. The corresponding dynamic-label ablations are included in Appendix E (Table 12) of the revised manuscript.

To further examine the discovered environments, we analyzed the Phase-I networks using ground-truth groups only for post-hoc analysis (Table R4).

Table R4. Characteristics of inferred vulnerable environments across ablation variants (Phase-I). Flag denotes the percentage of training examples assigned to the vulnerable environment, and precision denotes the minority proportion within that environment (computed within the relevant class for CelebA). All values are percentages.

VariantWaterbirds FlagWaterbirds PrecisionCelebA FlagCelebA Precision
RAE (full)11.340.941.647.9
w/o prediction interpolation (λ = 1)9.746.037.229.1
Uniform update selection11.040.939.344.2
w/o dynamic-label updates7.044.227.515.4
w/o uncertainty regularizers22.518.142.048.9

Without dynamic-label updates, the inferred environment becomes smaller on both datasets (Waterbirds: 11.3% → 7.0%; CelebA: 41.6% → 27.5%), with a substantial reduction in minority precision on CelebA (47.9% → 15.4%). Uniform selection yields similar environment sizes and minority precision to RAE, despite lower worst-group accuracy. On CelebA, both selection strategies also yield comparable label-change rates (minority: 44.5% vs. 44.3%; majority: 29.8% vs. 30.5%), suggesting that performance differences are not explained by the frequency of label changes alone.

Removing both uncertainty regularizers increases the flagged fraction (11.3% → 22.5%) and reduces minority precision (40.9% → 18.1%) on Waterbirds, whereas the corresponding environment statistics remain similar on CelebA. These findings support the contribution of reliability-guided discovery training while indicating that the effects of prediction interpolation and regularization vary across datasets. We have added these analyses in Appendix E (Tables 12 and 14) and clarified the motivation in Section 3.3 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer uL8c (3/5)

W3. Hyperparameter and Checkpoint Selection

Response. Ground-truth group labels were not used for training, hyperparameter tuning, or checkpoint selection in either phase. They were used only for test worst-group evaluation and post-hoc analysis. Optimization hyperparameters, including the GroupDRO step size η, follow XRM's dataset-specific defaults (Table 6). All RAE-specific hyperparameters except β\beta are fixed across datasets (λq\lambda_q = 0.01, λr\lambda_r= 1, k = 20, τ0\tau_0 = 0.5).

β\beta is selected from {0.02, 0.05, 0.1, 0.25} based on worst-group accuracy over the validation groups inferred in Phase I. This yields β\beta = 0.05 for Waterbirds and MultiNLI and β\beta = 0.25 for CelebA and CivilComments. The final GroupDRO checkpoint is selected by the highest validation worst-group accuracy over the same inferred groups. We report the sensitivity to β\beta below and have included it in Appendix E (Table 13) of the revised manuscript.

Table R4. Sensitivity to β\beta (test worst-group accuracy; mean ± standard deviation over three seeds).

β\betaWaterbirdsCelebA
0.0284.3 ± 1.474.1 ± 0.9
0.0588.0 ± 0.077.0 ± 0.9
0.185.8 ± 0.284.1 ± 2.9
0.2581.9 ± 1.087.9 ± 0.7
Reviewer_Qu7fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies group robustness without using any group labels. It focuses on the limitation of the current error-based environment discovery in identifying vulnerable examples, as they only care about observed prediction errors and overlook correctly classified but uncertain examples. It proposes Reliability-Aware Environment Discovery (RAE), which combines prediction errors with margin-based uncertainty, using twin networks and reliability-guided label updates to construct environments for robust training. Experiments on vision and language benchmarks show competitive worst-group performance, while controlled experiments demonstrate that uncertain but correctly classified examples are more likely to fail when spurious cues change.

Critical strengths
  1. The paper identifies an important limitation of error-based environment discovery: correctly classified examples may still be vulnerable to changes in spurious correlations.

  2. The controlled ColoredMNIST experiment provides clear evidence that prediction uncertainty can help identify vulnerable examples that error-based methods overlook.

  3. The experiments cover comprehensive benchmarks, with useful comparisons of environment discovery strategies and ablation studies of the proposed components.

Critical weaknesses
  1. About related work discussion and novelty of the paper: it seems there are some prior studies that closely related to this paper are missing in the related work part, for example, END [1] and Evidential Alignment [2]. These papers have also explored prediction uncertainty for identifying and mitigating spurious correlations, but the paper did not discuss these closely related papers, making it difficult to assess the novelty of the proposed method.

  2. Comparisons with recent methods in experiments: the experiments included in the paper stopped at methods published in 2024. There are recent metods such as Evidential Alignment [2], DPE [3], and CCDB [4]. Comparing with recent annotation-free methods is important and would strengthen the paper.

  3. Unclear hyperparameter selection and result variability: Appendix A.2 reports dataset-specific values of β\beta without explaining how they were selected without group annotations, which is important, since the paper highlights that some existing methods require group annotations for hyperparameter selection while RAE is fully group-annotation-free. Moreover, although Section 4.2 refers to standard deviations in the Appendix, these results are missing, making it difficult to assess the reliability of small performance improvements.

Questions for authors
  1. How does RAE differ from existing uncertainty-based approaches such as END [1] and Evidential Alignment [2], particularly in identifying vulnerable samples?

  2. Could the authors provide comparisons with more recent annotation-free methods, such as Evidential Alignment [2], DPE [3], and CCDB [4], under comparable experimental settings?

  3. How were the dataset-specific hyperparameters, particularly β\beta, selected without group annotations? Could the authors also provide the missing standard deviations for the main experimental results?


[1] Oh, D., Lee, D., Byun, J., & Shin, B. (2022). Improving Group Robustness under Noisy Labels using Predictive Uncertainty. arXiv:2212.07026. https://arxiv.org/abs/2212.07026

[2] Ye, W., Zheng, G., & Zhang, A. (2025). Improving Group Robustness on Spurious Correlation via Evidential Alignment. KDD 2025. https://arxiv.org/abs/2506.11347

[3] To, Minh Nguyen Nhat, et al. (2025). Diverse Prototypical Ensembles Improve Robustness to Subpopulation Shift. ICML 2025. https://proceedings.mlr.press/v267/to25a.html

[4] Zhao, M., & Zhang, Q. (2026). Class-Conditional Distribution Balancing for Group Robust Classification. ICML 2026. https://proceedings.mlr.press/v306/zhao26bt.html

5 comments
AuthorsOpenReview ↗

Response to Reviewer Qu7f (1/3)

We thank the reviewer for the detailed and constructive feedback. We address each point below and have incorporated the changes in the revised manuscript.

W1 / Q1. Methodological Distinctions from END and Evidential Alignment

Response. We thank the reviewer for the helpful suggestions on related work. We agree that END [1] and Evidential Alignment [2] already leverage predictive uncertainty to improve subgroup robustness without group annotations.

The key distinction lies in how uncertainty is incorporated into the learning process. END uses predictive entropy from an SGLD-trained Bayesian network to oversample uncertain examples for ERM retraining, while Evidential Alignment uses Dirichlet-based uncertainty to weight misclassified examples during last-layer retraining. Both obtain uncertainty signals from separately trained models and use them for subsequent sample selection or weighting.

In contrast, RAE incorporates reliability-guided regularization and uncertainty-guided label updates directly into discovery training (Eqs. 11–15). It then uses the resulting prediction errors and correctly classified low-margin examples to construct surrogate environments for GroupDRO (Eq. 17). Thus, RAE's methodological distinction lies in coupling reliability-guided discovery training with uncertainty-based environment construction, beyond using uncertainty solely for downstream sample selection or weighting.

We have clarified these differences in Section 2 and Appendix A.1 (Distinction from Uncertainty-based Approaches) of the revised manuscript.

W2 / Q2. Comparison with Recent Robust Learning Methods

Response. We thank the reviewer for suggesting these additional baselines. We reproduced Evidential Alignment and DPE using their official implementations. Table 3 in our response to W3/Q3 reports these additional comparisons alongside the existing baselines, including standard deviations over three random seeds. We could not reproduce CCDB because its implementation is not publicly available.

RAE achieves higher worst-group accuracy than Evidential Alignment on all four datasets without group annotations (79.0 vs. 67.3 on average). Compared with DPE, which uses group-labeled validation data, RAE achieves higher worst-group accuracy on CelebA (87.9 vs. 84.2) and CivilComments (71.5 vs. 69.4) without group annotations.

These additional comparisons are reported in Table 3 and Appendix D of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer Qu7f (2/3)

W3 / Q3. Hyperparameter Selection and Experimental Consistency

Response. We apologize for omitting the standard deviations from the submitted appendix. Table 3 below reports three-seed results, including the additional baselines requested in W2/Q2. We also reran the ablations in Tables 4 and 5 using the same seeds; the revised manuscript reports their means in the main text and mean ± standard deviation in Appendix D (Tables 8 and 9).

No group labels were used for training, hyperparameter selection, or checkpoint selection, including on the validation set. β is the only RAE-specific hyperparameter that varies across datasets. We selected β\beta from {0.02, 0.05, 0.1, 0.25} based on worst-group accuracy over the validation groups inferred by RAE in Phase I, yielding β\beta = 0.05 for Waterbirds and MultiNLI and β = 0.25 for CelebA and CivilComments. All other RAE-specific hyperparameters were fixed across datasets (λq\lambda_q = 0.01, λr\lambda_r = 1, k = 20), while optimization hyperparameters followed XRM's dataset-specific defaults.

We provide a β\beta sensitivity analysis in our response to uL8c (W3) and have included it in Appendix E (Table 13) of the revised manuscript.

Additionally, we corrected a typo in the submitted Table 3: the BAM (✗/✓) MultiNLI entry should be 81.5 / 71.1, not 81.3 / 71.2. We also identified an inconsistency in the CivilComments evaluation, where XRM's implementation evaluated 10 groups instead of the intended 16. We therefore re-evaluated RAE and the relevant baselines using the 16-group definition (see GML9 Q5 for details).

Table 3 (with standard deviations and additional baselines). Performance comparison of RAE and baseline methods (mean ± standard deviation over three random seeds where available). ‘Avg’ and ‘Worst’ denote average and worst-group accuracy, respectively. ‘tr/val’ indicates whether group labels were used during training and validation.

Group Annotation tr/valAlgorithmWaterbirds (Avg / Worst)CelebA (Avg / Worst)MNLI (Avg / Worst)CivilComms (Avg / Worst)Average (Avg / Worst)
Group annotations are required
✓/✓ERM82.7 ± 0.4 / 51.5 ± 0.995.8 ± 0.1 / 44.6 ± 0.081.5 ± 0.0 / 72.4 ± 0.184.2 ± 0.0 / 73.5 ± 0.186.1 / 60.5
✓/✓GroupDRO90.0 ± 0.1 / 86.1 ± 0.493.0 ± 0.1 / 87.0 ± 2.080.6 ± 0.0 / 75.7 ± 0.183.9 ± 0.0 / 73.3 ± 1.186.9 / 80.5
✗/✓EIIL86.5 ± 0.3 / 69.3 ± 2.095.2 ± 0.1 / 43.5 ± 0.682.3 ± 0.2 / 67.5 ± 0.7––
✗/✓JTT90.5 ± 0.6 / 87.8 ± 1.290.5 ± 0.3 / 79.6 ± 4.679.8 ± 0.7 / 70.9 ± 2.392.5 ± 0.2 / 59.1 ± 2.888.3 / 74.4
✗/✓BAM91.0 ± 0.3 / 89.3 ± 0.487.0 ± 6.7 / 77.7 ± 4.081.5 ± 0.2 / 71.1 ± 0.290.1 ± 0.3 / 69.1 ± 1.787.4 / 76.8
✗/✓DPE94.1 ± 0.3 / 89.2 ± 0.492.3± 0.1 / 84.2± 0.681.3 ± 0.2 / 72.2± 0.689.1 ± 0.3 / 69.4 ± 0.789.2 / 78.7
Group annotations are not required
✗/✗EIIL87.2 ± 0.2 / 69.5 ± 1.895.2 ± 0.1 / 42.6 ± 1.382.4 ± 0.1 / 66.9 ± 0.3––
✗/✗JTT91.8 ± 0.8 / 87.4 ± 0.694.3 ± 0.3 / 48.7 ± 6.180.7 ± 0.4 / 66.8 ± 1.292.9 ± 0.1 / 46.8 ± 4.089.9 / 62.4
✗/✗BAM92.1 ± 0.5 / 89.5 ± 0.591.9 ± 7.3 / 73.1 ± 2.481.5 ± 0.4 / 68.1 ± 0.990.1 ± 0.3 / 69.1 ± 1.788.9 / 75.0
✗/✗GIC89.0 ± 0.9 / 74.1 ± 4.693.5 ± 1.2 / 67.0 ± 8.882.0 ± 0.1 / 67.2 ± 2.990.8 ± 0.6 / 73.1 ± 3.088.8 / 70.4
✗/✗XRM89.2 ± 0.8 / 87.7 ± 1.188.2 ± 0.3 / 82.4 ± 0.467.7 ± 1.5 / 60.3 ± 4.490.7 ± 0.1 / 65.9 ± 1.584.0 / 74.1
✗/✗DPE93.6 ± 0.6 / 90.6 ± 0.594.3 ± 0.1 / 65.0 ± 2.981.8 ± 0.3 / 67.2 ± 0.488.6 ± 0.2 / 61.9 ± 1.289.6 / 71.2
✗/✗Evidential Alignment92.3 ± 0.9 / 85.7 ± 3.094.3 ± 1.8 / 61.5 ± 11.682.3 ± 0.0 / 65.7 ± 1.688.6 ± 0.2 / 56.1 ± 0.689.4 / 67.3
✗/✗RAE (Ours)90.3 ± 0.1 / 88.0 ± 0.090.0 ± 0.2 / 87.9 ± 0.771.4 ± 1.1 / 68.7 ± 1.287.4 ± 0.8 / 71.5 ± 1.284.8 / 79.0

Table 4. Comparison of environment assignment strategies on the same RAE discovery networks (test worst-group accuracy; mean ± standard deviation over three seeds).

DatasetErrors only (Avg)Errors only (Worst)Uncertainty only (Avg)Uncertainty only (Worst)Errors + random (Avg)Errors + random (Worst)Errors ∪ uncertainty (RAE) (Avg)Errors ∪ uncertainty (RAE) (Worst)
Waterbirds88.9 ± 0.584.6 ± 1.779.3 ± 1.061.1 ± 8.389.6 ± 0.185.4 ± 0.490.3 ± 0.188.0 ± 0.0
CelebA91.8 ± 1.277.6 ± 3.691.3 ± 0.183.0 ± 0.992.1 ± 0.979.8 ± 6.590.0 ± 0.287.9 ± 0.7
AuthorsOpenReview ↗

Response to Reviewer Qu7f (3/3)

Table 5. Ablation of RAE components on Waterbirds and CelebA (mean ± std over the three seeds of Table 3; change from the full model in parentheses).

λq\lambda_qλr\lambda_rDynamic labelWaterbirds AvgWaterbirds WorstCelebA AvgCelebA Worst
✓✓✓90.3 ± 0.188.0 ± 0.090.0 ± 0.287.9 ± 0.7
✗✓✓89.3 ± 0.2 (↓1.1)84.8 ± 0.4 (↓3.2)90.3 ± 0.5 (↑0.3)84.3 ± 0.6 (↓3.6)
✓✗✓88.2 ± 0.5 (↓2.1)81.3 ± 1.5 (↓6.7)90.5 ± 0.3 (↑0.5)84.1 ± 1.4 (↓3.8)
✓✓✗78.7 ± 1.6 (↓11.7)75.2 ± 2.1 (↓12.9)93.4 ± 0.2 (↑3.5)72.0 ± 2.2 (↓15.9)
✗✗✓84.2 ± 0.5 (↓6.1)72.7 ± 1.2 (↓15.3)90.7 ± 0.6 (↑0.8)83.9 ± 0.6 (↓4.0)
✗✗✗78.1 ± 2.6 (↓12.2)75.3 ± 4.1 (↓12.8)93.5 ± 0.2 (↑3.5)71.3 ± 1.8 (↓16.6)
Reviewer_GML9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

RAE (Reliability-Aware Environment Discovery) is a group-robustness method for subpopulation shift that needs no group labels, built on the principle that a model's confidence on an example, along with its correctness, signals vulnerability to spurious cues. It argues that error-based environment discovery, such as JTT and XRM, treats all correctly classified examples alike and so misses those that are correct only by a small gap between the top two predicted probabilities. RAE builds on XRM's twin networks, each trained on half the data to estimate the prediction and uncertainty of the classes for environment discovery. After the discovery training, an example is placed in the vulnerable environment if either network misclassifies it or its uncertainty exceeds a dataset-specific quantile, and GroupDRO is used to train on groups formed from class and environment. Motivated by a controlled ColoredMNIST experiment, RAE is evaluated on Waterbirds, CelebA, MultiNLI and CivilComments without group labels, and the paper reports the best average worst-group accuracy among annotation-free baselines, close to GroupDRO trained with true groups.

Critical strengths
  1. Adding correct but low-margin examples to the error-based vulnerable set is a reasonable new approach. The empirical results show that adding uncertainty computation for group discovery have the most competitive performance over existing baselines that do not use training/val group labels.
  2. The motivating ColoredMNIST experiment is a clean test of the idea. It shows that correct but unreliable examples fail far more often than confident ones as the conflict grows, motivating the design of using uncertainty for group inference.
Critical weaknesses
  1. The baselines rerun are significantly below their published numbers. XRM reports 89.1 worst-group accuracy on CelebA and 72.1 on MultiNLI, against 82.4 and 60.3 here. BAM with label-free ClassDiff rule. With the published numbers, XRM's four-dataset average worst-group accuracy can be higher than RAE's, and BAM with ClassDiff beats RAE on three of four datasets.
  2. The core idea, using uncertainty and not only errors to find minority examples, is already established, and the paper does not show that this part drives its gains. SELF [1] compares selecting examples by misclassification against selecting them by model disagreement, which it describes as picking the most uncertain points.

[1] LaBonte, T., Muthukumar, V., & Kumar, A. (2023). Towards last-layer retraining for group robustness with fewer annotations. Advances in Neural Information Processing Systems, 36, 11552-11579.

Questions for authors
  1. How were the hyper parameters such as β\beta and the GroupDRO step size η\eta, the loss weights λq\lambda_q and λr\lambda_r and the sigmoid sharpness kk, and the final checkpoint chosen? Were group labels used anywhere, including on the validation set?
  2. Why the reported XRM results in the paper differs from the original paper?
  3. How does RAE compare with SELF [1]?
  4. The paper didn't discuss last-layer retraining and group reweighting methods [1][2][3], which are simple yet effective approaches.
  5. The experiments of CivilComments are done with 4 groups, but typically it is often tested with 16 groups to validate the methods performance beyond single spurious correlation. How does the method perform on 16-group version of CivilComments?

[1] LaBonte, T., Muthukumar, V., & Kumar, A. (2023). Towards last-layer retraining for group robustness with fewer annotations. Advances in Neural Information Processing Systems, 36, 11552-11579.

[2] Qiao, R., Wu, Z., Wang, J., Koh, P. W., & Low, B. K. H. (2025). Group-robust Sample Reweighting for Subpopulation Shifts via Influence Functions. arXiv preprint arXiv:2503.07315.

[3] Kirichenko, P., Izmailov, P., & Wilson, A. G. (2022). Last layer re-training is sufficient for robustness to spurious correlations. arXiv preprint arXiv:2204.02937.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response for RAE

We sincerely thank all reviewers for their careful evaluation and constructive feedback. We are encouraged that the reviewers recognized the importance of addressing the limitations of error-based environment discovery (GML9, Qu7f, uL8c), particularly the vulnerability of correctly classified but uncertain examples. The reviewers also highlighted the clear empirical motivation provided by our controlled ColoredMNIST experiments (GML9, Qu7f), the comprehensive benchmark evaluations and ablation studies (Qu7f), and the competitive worst-group performance without group annotations (GML9, uL8c).

In response to the reviewers' concerns, we have conducted additional experiments, strengthened methodological comparisons, clarified our experimental protocols, and revised the manuscript accordingly. We summarize the key revisions below.

Reviewer ConcernReviewer(s)Our Responses and Revisions
1. Methodological Novelty and MotivationGML9, Qu7f, uL8cWe clarified RAE's contribution as integrating reliability-guided regularization and uncertainty-conditioned label updates into environment discovery. Additional experiments distinguish discovery training from uncertainty-based environment assignment and examine the contribution of each component.
2. Baseline Comparisons and ReproducibilityGML9, Qu7f, uL8cWe added DPE and Evidential Alignment baselines, reported three-seed results, and clarified reproduction protocols. We also evaluated AFR and SELF under group-label-free model selection.
3. Hyperparameter Selection and SensitivityGML9, Qu7f, uL8cWe clarified the hyperparameter selection procedure and conducted additional sensitivity analyses for $\beta$, $\lambda_q$, $\lambda_r$, and $k$, acknowledging the observed sensitivity and limitations.
4. Dynamic-Label Update MechanismuL8cWe conducted ablations with variant-specific $\beta$ selection and matched flagged fractions, alongside analyses of update allocation and minority discovery to investigate stochastic updates.
5. Evaluation Consistency and Experimental ProtocolsGML9, Qu7f, uL8cWe corrected CivilComments evaluation to use 16 groups, aligned experimental seeds, corrected a baseline reporting error, and clarified the ColoredMNIST protocol.
6. Computational EfficiencyGML9, uL8cWe re-measured runtimes on the same GPU, provided a stage-wise breakdown, and revised computational claims to account for differences in training protocols.

We believe these additional experiments and clarifications have substantially strengthened the manuscript and addressed the key concerns raised by the reviewers. We sincerely appreciate the reviewers' constructive feedback and thoughtful suggestions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.