acceptodds
Under review as a conference paper at ICLR 2027

Learning Environment Distributions for Robust Image Classification

11 views

Abstract

Machine learning models often exploit spurious correlations, such as using background to recognize a bird in Waterbirds or the color of a digit to classify it in Colored MNIST. One approach to reducing this reliance is to train across multiple environments: data distributions that differ, for example, in their color-label correlation or in how training examples are grouped. However, the choice of these distributions and their contribution to training can strongly affect the resulting predictor. In this paper, we introduce Universal Adaptive Environment Discovery (UAED), a framework that jointly learns a predictor and a distribution over a specified family of environment constructions, regularized by a KL penalty toward a reference. For transformation-based training, this distribution determines how different transformations contribute to the objective; for group-based training, it weights alternative groupings of examples, which we construct from a reference predictor's prediction difficulty. Our analysis characterizes the resulting policy preferences: learned cooperatively with the predictor, the distribution tilts toward environments the predictor already handles well and collapses onto a shortcut environment below a computable KL threshold, whereas reversing the sign of its update yields a robust counterpart whose objective upper-bounds the expected training score over a KL ball of environment distributions. Experiments distinguish the effects of broader coverage from those of learning its weighting. Training over the family improves worst-case accuracy over restricted environments on Colored MNIST and worst-group accuracy over ERM on Waterbirds with ResNet-50 without group annotations, while with ViT-B/16 the grouping objective destabilizes fine-tuning at the shared learning rate. Learned policies move as predicted, away from the annotated grouping when cooperative and toward it when robust, and weak KL penalties reinforce color shortcuts; on the official GroupDRO code, where the grouping family reaches 81.8% worst-group accuracy without group annotations, cooperative learning lowers accuracy in every seed and the robust policy removes this loss, matching but not exceeding a fixed weighting. These findings identify both the potential and the limitations of jointly optimizing predictors and environment distributions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Invariant Risk Minimization, Robust Learning, Adaptive Environments

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VadWOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

Due to the presence of spurious correlations in training data, domain generalization methods make use of multiple training distributions (environments) to steel models against learning predictive rules that will not generalize. Across different methods, the choice of environments and their weight in the training loss is often chosen heuristically, despite the fact that different groups of environments and weighings over those environments can significantly impact training. In this work, the authors present UAED, a method for adaptively reweighing a family of environments’ contribution to the training loss. The work presents a variety of theoretical results predicting the nature of the learned policy over environments, and applies this method to two popular domain generalization benchmarks.

Critical strengths
  • The paper is well written and easy to follow. Figures and tables are clearly labelled. Theoretical results are well formatted and are followed by clear explanations of their significance.
  • Extensive experiments across two popular datasets provide ample evidence for the empirical findings of the work.
Critical weaknesses
  • [W1] The theoretical results, while intuitive, are also unsurprising. The results regarding the cooperative policy, namely that it will favour shortcuts if the KL penalty is not high enough and that it will defer to the reference distribution when paired with an invariant learner, are to be expected given the set up. Similarly with the robust policy providing larger weights to environments with higher scores.
  • [W2] The empirical results seem to demonstrate that the methods presented in the paper are do not provide any gains with respect to simple baselines. Common methods like IRM match UAED’s performance. Furthermore, among different UAED methods, simply using a static environment policy produces the same performance as learning the policy. The authors state that this points to the importance of the family of environments the policy refers to. However, this result begs the question as to what readers are supposed to take away from the paper.
Questions for authors
  • [Q1] Are there experiments on ColorMNIST with the robust policy?
  • [Q2] Is there an intuition as to why the cooperative and robust policies tend to attain the same performance? Is it that they both provide some weighting over all environments in the family due to the KL term with the reference?
  • [Q3] Is there a way that the findings of this work can be applied to other domain generalization methods and inform their construction?
Reviewer_v5ucOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose Universal Adaptive Environment Discovery (UAED), which learns a predictor and a KL-regularized distribution over predefined environments simultaneously. The authors present cooperative and robust formulations, describe shortcut collapse under weak regularisation, and establish a KL-ball robustness bound. Experiments with Coloured MNIST and Waterbirds show that broader environment coverage is useful for robustness, but learned weighting has limited additional utility.

Critical strengths
  • The paper provides a unified formulation connecting environment weighting, Gibbs policies, and distributionally robust optimization.

  • Proposition 3 gives an interpretable characterization of when cooperative optimization favors shortcut predictors under small β\beta.

  • The analysis helps explain why jointly learning environments and predictors may not always improve robustness.

  • The experiments explicitly separate the contributions of broader environment coverage and learned weighting, which is important for evaluating the framework.

  • The frozen grouping policy achieves 81.8% worst-group accuracy in the official protocol without training group annotations, demonstrating the usefulness of the constructed grouping family.

  • The authors transparently discuss negative results, including cooperative-policy degradation and ViT-B/16 instability, rather than reporting only favorable outcomes.

Critical weaknesses

Major

  • The core theoretical tools, including Gibbs variational optimization and entropic risk, are already well established. The only novelty primarily lies in their application to environment weighting.

  • The experiments do not demonstrate a consistent advantage from learning environment weights over using a fixed broad distribution. On Colored MNIST, learned UAED-IRM achieves 71.6% worst-case accuracy, compared with 71.7% for the frozen policy at the final checkpoint (Table 1). Under the official GroupDRO protocol, cooperative UAED decreases worst-group accuracy from 81.8% to 80.0%, while robust UAED reaches 81.9% (Table 2). Thus, the robust variant mainly recovers the performance lost by cooperative weighting rather than establishing a clear improvement over fixed weighting.

  • Evaluation is restricted to Colored MNIST and Waterbirds, limiting conclusions about broader robustness settings. The ViT-B/16 results reveal substantial instability: frozen and cooperative UAED achieve only 26.3% and 27.3% worst-group accuracy, respectively (Table 2). Although lowering the learning rate stabilizes ViT-B/16 training, the recovered performance remains comparable to class-balanced ERM.

  • The partition family depends on manually chosen grouping parameters, and comparisons with additional recent group-label-free methods are missing.

Minor

  • The Waterbirds partition family relies on a reference model and manually selected grouping quantiles and temperatures. More sensitivity analysis would clarify how robust the results are to these choices.

  • The paper provides limited practical guidance for selecting the KL coefficient β\beta beyond warning against in-distribution validation. A principled selection strategy would improve usability.

  • Waterbirds checkpoint selection uses worst-group validation accuracy, which requires group annotations at validation time. This distinction should be emphasized when discussing annotation-free robustness.

Questions for authors
  1. Since frozen environment policies achieve comparable or better performance in most experiments, under what conditions is adaptive environment weighting expected to provide a meaningful advantage?

  2. How sensitive are the Waterbirds results to the reference predictor, grouping quantiles, temperatures, and entropic aggregation parameter τ\tau?

  3. Can the authors suggest a principled strategy for selecting β\beta using training statistics or policy divergence, without relying on validation environments that may favor shortcuts?

  4. Can the authors explain why the grouping objective causes severe instability with ViT-B/16? Would alternative optimization strategies improve performance beyond class-balanced ERM?

  5. Could the authors provide results (if possible) on additional natural-image robustness benchmarks or comparisons with stronger group-label-free methods to better establish the practical significance of UAED?

Reviewer_gGkVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces UAED, which jointly learns a predictor and a KL-regularized distribution over a supplied family of environment constructions. The cooperative and adversarial objectives in Eqs. (1)–(2) respectively favor easier and harder environments. Experiments on Colored MNIST and Waterbirds in Tables 1–2 distinguish broader environment coverage from adaptive weighting: coverage provides robustness gains in several settings, whereas learning the weights generally matches or underperforms fixed weighting.

Critical strengths
  1. The experimental controls clarify the source of improvement. Table 1 includes restricted, uniform, frozen-policy, and learned-policy baselines. Learned and frozen UAED-IRM achieve nearly identical final worst-case accuracy—71.6% and 71.7%, respectively—compared with restricted IRM’s 64.1%. These comparisons support the conclusion that broader coverage accounts for most of the gain.

  2. The policy analysis provides an interpretable explanation of the observed behavior. Proposition 1 and Corollary 2 characterize opposite policy preferences for a fixed predictor. Consistent with these predictions, §4.3 reports that the cooperative policy assigns 94% of its mass to the softest groupings under the official Waterbirds protocol, while the robust policy favors tighter groupings. The agreement between the qualitative predictions and policy trajectories strengthens the diagnostic contribution.

Critical weaknesses
  1. Adaptive weighting has no established advantage over fixed weighting. Under the official Waterbirds protocol, Table 2 reports worst-group accuracies of 81.8%, 80.0%, and 81.9% for frozen, cooperative, and robust UAED. The robust–frozen difference reported in §4.3 is only +0.1 percentage points, with a 95% interval of [−0.2, +0.3]. Although the paper appropriately acknowledges these null and negative results, the practical benefit of adding policy optimization remains unestablished.

  2. The practical model-selection procedure remains incomplete. Table 5 shows that collapsed configurations obtain a higher validation criterion than the successful β=1 configuration—79.8% versus 73.6%—despite achieving 0% worst-case test accuracy. For Waterbirds, Table 3 specifies that checkpoint selection uses group-labeled validation data. This is a disclosed benchmark assumption, but the absence of training group annotations does not establish applicability without group annotations for model selection. A validated strategy for selecting β and checkpoints remains necessary.

  3. The grouping construction has limited evidence of transfer across backbones. Section 2 constructs nine candidates using a fixed quantile–temperature grid over reference-model margins, and §6 acknowledges that these choices were developed with ResNet-50. In Table 10, reducing the ViT-B/16 learning rate restores stability, but frozen UAED only matches class-balanced ERM—64.2% versus 64.4%. Thus, resolving optimization instability does not establish that the proposed groupings capture useful minority-group structure across representations.

  4. The numerical threshold analysis receives only coarse empirical validation. Appendix C obtains values of 0.059 during warm-up and 0.038 afterward by numerically optimizing a comparison between idealized predictors. These are attained lower bounds supporting shortcut preference, rather than exact predictions of a training transition. Table 5 evaluates β=0.01 and β=0.1 on either side without resolving the intervening region. The results support the qualitative mechanism, but provide limited evidence about the quantitative relationship between these bounds and joint optimization dynamics.

Questions for authors
  1. Given the near-identical frozen and robust results in Table 2, can the authors identify a reproducible setting where adaptive weighting improves robustness over fixed weighting of the same environment family?

  2. Given the validation failure in Table 5 and the group-labeled validation specified in Table 3, what procedure should practitioners use to select β and checkpoints when shifted validation environments or group annotations are unavailable?

  3. Since the stabilized ViT-B/16 result in Table 10 matches class-balanced ERM, how sensitive is performance to the reference checkpoint and grouping grid? Measuring minority-group enrichment within the low-margin strata could help distinguish inadequate group construction from optimization difficulties.

  4. Could a denser β sweep around the values calculated in Appendix C assess how closely observed collapse behavior follows the objective comparison? The analysis should distinguish the sufficient conditions for shortcut preference from transitions produced by the warm-up and subsequent optimization.

  5. Section 5 discusses EIIL and HRM, while §2 constructs a supplied family of margin-based groupings. What additional capability comes from retaining and weighting multiple candidates? A matched comparison with EIIL would help isolate the contribution of the grouping construction, particularly because Table 2 shows little benefit from learning its weights.

  6. Section 5 acknowledges that Augerino and OPTIMA jointly optimize augmentation distributions and model parameters. Could the authors explicitly compare their objectives and regularizers with Eq. (1), identifying the methodological distinction? A comparison using the same recoloring family would help establish what UAED adds beyond the fixed-coverage gains in Table 1.

  7. Section 3 identifies Proposition 1 as the classical Gibbs variational principle, while Corollary 2 uses an entropic-risk/KL-ball formulation. Could the authors clearly distinguish established identities from new results, particularly explaining what Propositions 2–3 contribute specifically to learning environment distributions?

1 comment
Area_Chair_i3A5OpenReview ↗

Reminder to engage in reviewer-author discussion

Hi gGkV,

As there are only two days left in the author-reviewer discussion period, I would like to remind you to kindly respond to the authors' rebuttal, posted over a week ago. After November 18, reviewer responses are not be allowed and you will no longer be able to ask any follow-up questions. If you have no follow-ups, a short and polite response acknowledging you have read and understood the rebuttal and have no further questions is sufficient.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.