acceptodds
Under review as a conference paper at ICLR 2027

Learning the Robustness Mechanism with Bilevel Optimization

Abstract

We propose a distributionally robust learning framework where parameters defining the robustness mechanism are learned from held-out data instead of extensively tuned. Using bilevel optimization with both upper and lower level minimax problems, we create two instances of our framework to tackle setups with and without group labels in the training set. Theoretically, we provide sample complexity analysis for our robustness mechanism learning paradigm, showing that it achieves generalization guarantees comparable to exhaustive grid search while being more computationally efficient. Empirically, we evaluate our framework under a challenging setup when both intra-group and inter-group test distribution shifts occur at the same time, thereby demonstrating the efficacy and scalability of our method.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Distributionally Robust Optimization, Bilevel Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_drqROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses the challenge of spurious shortcuts, i.e., non-causal features (e.g., background context) that lead models to make incorrect predictions when the shortcut is absent. In this context, the authors propose a novel reformulation for training a robust model to mitigate this issue. Specifically, while standard robustness methods often rely on a min-max formulation, the authors propose shifting this to a bilevel optimization problem, where both the outer and inner objectives are formulated as min-max problems. This nested structure forces the model to ignore spurious cues by optimizing for robustness across different settings. The authors validate their methodology on binary classification datasets (i.e., CMINST, CelebA, and CivilComments) where the spurious signal is known in advance and amplified. For example, in CMNIST, the color of the digit corresponds to the short signals that the authors want the model to prevent learning.

Critical strengths

The paper identifies a well-defined and highly relevant problem in ML, i.e., the tendency of models to rely on spurious shortcuts rather than causal features. The challenge is relevant as shortcuts can influence multiple trustworthy principles (e.g., integrity and robustness).

The presentation of related work provide a clear landscape of how the proposed method sits within the existing literature on robustness and optimization.

Beyond its complexity, the transition to a bilevel optimization framework is a mathematically ambitious attempt to tackle a difficult problem.

Critical weaknesses

### Complex conceptual interpretability and higher computational complexity

While reading the paper, I had the feeling that the transition from a standard min-max formulation to a nested bilevel min-max formulation is mathematically not obvious, sometimes complex in a few passages (*), and not intuitive. Yet the presentation and storytelling in the paper lack a clear intuitive or heuristic justification for why this specific structure is “natural” for solving spurious correlation problems. The formulation feels somewhat arbitrary. Furthermore, bilevel formulations are known to be computationally complex and are typically approximated as min-max processes, which means exactly the opposite of what the authors are doing in this paper. Therefore, I believe the paper is missing a clear explanation in this direction and justification for the bilevel formulation, and a more intuitive justification to support the readers.

Lastly, in doing so, the proposed bilevel formulation significantly increases the computational burden compared to existing state-of-the-art (SOTA) methods. As evidenced by the results in the Appendix (Table 7), the method can be up to 16x slower than HDRO in certain scenarios. In the context of modern deep learning, where training efficiency is a primary concern, a 16-fold increase in training time is a massive barrier to adoption.

(*) Passage complexity can possibly be a limitation on my side. Therefore, I’ll be happy to know the opinion of the other reviewers in this regard. Right now, this is not an ingredient that justifies my score.

Lack of self-consistency

The main paper (limited to 8 pages) is not self-consistent, meaning that without the Appendix, it is almost impossible to replicate and understand the experimental analysis and outcomes proposed in the main paper. To be specific, we can take the example of the experimental setup, where models, hyperparameters, and (importantly) evaluation metrics used in the main paper are reported only in the Appendix. The primary basis for comparison (the distinction between Average and Worst-case accuracy) is relegated entirely to the Appendix despite being the central pillar of the discussion and results. A reader should not be required to parse the Appendix, as also written in the ICLR guidelines, but without that the main paper hardly survives alone. The Appendix is not used as a section where to report deeper analysis or proofs, but as a continuation of the main.

Limited empirical scope and margin of improvement

T The experiments are confined to binary classification tasks using low-dimensional data, which does not sufficiently demonstrate how the method scales to complex, high-dimensional datasets (e.g., ImageNet or large-scale NLP tasks). I will revise my score if the authors will take the chance to include additional models and large scale datasets.

Furthermore, the performance gains appear marginal compared to the SOTA. Looking at Table 1, the proposed method is outperformed by both GroupDRO and HDRO in terms of Average accuracy in two out of three tested datasets (notably by up to 10% in the Civil Comments dataset). While the proposed method shows promise in improving Worst-case accuracy, the trade-off is a significant degradation in average performance and a massive increase in compute. In my opinion, the results do not demonstrate remarkable improvements over the SOTA, despite the higher computational costs. There is for sure an improvement over the worst-case accuracy, but is it something that cannot be addressed with a different hyperparameter setup from other methods? The authors should clarify whether these results are truly an improvement or simply a shift in the error distribution, and whether the baselines could achieve similar results through more intensive hyperparameter tuning.

(Minor) ambiguity in AI usage statement

The authors state that “AI tools have been utilized to clean and reformat dataset”. Given that the datasets used are standard benchmarks, it is unclear whether this refers to automated scripts, LLM-based cleaning, or standard preprocessing pipelines. To what clean and reform is referring to in these standard datasets?

Questions for authors

Can the authors provide a more intuitive view that justify the bilevel formulation? Would it scale to more complex and higher dimensional tasks?

Is the binary setting a clear limitation of the approach?

The methodology presents for sure an improvement over the worst-case accuracy, but is it something that cannot be addressed with a different hyperparameter setup from other methods?

In the AI usage statement, to what clean and reform is referring to in these datasets?

2 comments

Dear Reviewer drqR,

We will group your concerns among weaknesses and questions into the following to address them collectively. Please let us know if we did not understand or address your concern properly.

Intuition behind bilevel formulation

The bilevel formulation may seem contrived if the same task can be solved with a single-level variant. However, we want to emphasize that the penalty form of our bilevel is solved as a minimax problem in practice as shown in Eq (14). Our motivation is that recently developed DRO methods all introduce some form of robustness mechanisms, and as shown in examples like HDRO and PG-DRO, those newly introduced robustness parameters may be sensitive and tuning of them needs to be exhaustive. This is where our bilevel framework that can be used to automatically tune those parameters kicks in.

A hopefully intuitive roadmap. In Section 2.1, we show the formula of GDRO, which is a minimax optimization problem. In Section 2.2, we show that the canonical GDRO can be reformulated as a bilevel optimization problem where the upper level minimax problem over validation set is nonconvex-concave and lower level minimax problem over training set is convex-concave. In Section 2.3 and 2.4, we show the generalizability of this bilevel framework under two settings. Furthermore, Algorithm described in Section 2.5 can be used to solve these problems. Lastly, we provided generalization guarantee of our bilevel methods in Section 2.6, conjecturing that they will perform better as they avoid discretization error which is inevitable in grid search.

Our comment on conceptual interpretability and self-consistency. When content is abundant, it is natural to relegate less important experimental details (e.g., hyperparameters) to the appendix. Previously, due to page limit, we omitted problem setup details that are common to baselines/related works. Now we have added detailed explanations of metrics, possible extension to trade off the worst group and average performance, and more ablations in the main text.

Computational complexity

We fully agree with this concern. Previous runtime analysis included evaluation time and conveyed mixed messages, so we modified and updated Section 4.4 and Appendix E.3 to compare the validation accuracy given the same iteration budget and wall-clock per iteration, which shows that our methods takes more time per iteration (due to the upper level validation loss) but yield consistently higher quality over baseline as seen from the curve. Furthermore, we want to emphasize that while grid search is faster per step, it takes an exponential amount of time as grid sizes increase to achieve similar results.

Average performance degradation

We want to mention that our upper level objective is flexible enough to incorporate average accuracy to mitigate the trade-off, should one be more interested in overall accuracy, and we leave it as future work. Since the current upper level goal is worst-group accuracy, the trade-offs naturally manifest in our results. We additionally compared budget-controlled validation curves between Bi-HDRO and Bi-GDRO in Appendix E.4 (Figure 5 and 6). It shows that while the test results between our methods and baseline are very close in Table 1, the worst-group validation results of our methods are better. The average validation results are similar as shown in Figure 6.

Extension to multiclass classification and complex tasks

We respectfully disagree with your comment that our experiments are limited to “low-dimensional data” unless you meant target dimensions: our experiments are not based on the likes of LIBSVM, Gaussian samples, or tabular attributes of CelebA. CelebA contains raw images, and CivilComments contain natural language sentences. Both are high dimensional and respectively need ResNet and DistilBERT for representation learning. CMNIST, CelebA, and CivilComments are all well established benchmarks used in many previous DRO works.

However, we agree with your concern that our tasks are all binary classifications. We already derived an upper bound for the inner perturbation maximization problem in Appendix C.2, and we have yet to put it to use. To address this concern, we created an image dataset with inter-group shift based on ImageNet-9 [1] by pairing extracted foreground (which contains the object/label information) and background (with foreground cropped and filled with tiling). We could not perform intra-group shift since it does not have attribute annotations similar to CelebA. This new dataset contains 9 classes and 1 attribute (whether background donor class matches foreground donor class or not), naturally creating 18 groups. Moreover, we intentionally create samples with mismatching backgrounds so that they only constitute 5% of the entire training set, but 50% in the validation/test set. We report the setup and results in Appendix E.6. Results show that Bi-HDRO is competitive to robust single-level methods. This is expected since latent perturbation can really make a larger difference only when intra-group shifts are present, as reported in [2].

Alternative hyperparameter setup for baselines

Grid searching over ϵ\epsilon yields flat optimal validation curves within the optimal range, so it is not a matter of “tuning intensity”. While we have tuned all baseline methods to the best of our ability, we want to make our point more convincing, so we created a new single-level nonconvex-concave minimax baseline for Bi-HDRO with three hyperparameters to be tuned using grid search. This is essentially Bi-HDRO’s lower level with the nonconvex backbone θ\theta added to the minimization block. Without the upper level tuning problem, this baseline does not adaptively learn the parameters. This yields 75 hyperparameter combinations for CMNIST and CelebA and 45 combinations for CivilComments, and it still performs worse compared to our bilevel methods. See Appendix D.3.2 for implementation details and Table 1 for added results.

AI statement

It’s for creating and verifying intra-group shifts actually occurred. We clarified it along with new ImageNet-9 experiments.

References

[1] Xiao, Kai, et al. "Noise or signal: The role of image backgrounds in object recognition." ICLR. 2021.

[2] Jo, Sung Ho, Seonghwi Kim, and Minwoo Chae. "Mitigating spurious correlation via distributionally robust learning with hierarchical ambiguity sets." ICLR. 2026.

Reviewer_r5o3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This submission introduces a bilevel optimization framework for distributionally robust optimization (DRO) in which the parameters governing the robustness mechanism are learned from held-out data rather than set through extensive manual tuning. Two instantiations are presented. Bi-HDRO learns perturbation radius to address group distribution shift, while Bi-PG-DRO jointly learns probabilistic group assignments and robustness parameters for the setting where group labels are unavailable at training time.

In both instantiations, the upper level optimizes worst-group validation performance and the lower level solves a robust training objective. The authors adopt an existing first-order algorithm for bilevel minimax problems and derive a generalization analysis for continuous hyperparameter tuning. Experiments on shifted versions of various benchmarks show gains in worst-group accuracy over several existing DRO baselines.

Critical strengths
  1. The paper targets a real practical limitation of existing DRO methods: their robustness-related hyperparameters typically require extensive manual tuning. The bilevel framework offers a principled alternative by optimizing these parameters directly against worst-group validation performance.

  2. The formulations are well developed, and the paper is generally easy to follow. The authors clearly explain how bilevel optimization can be applied to existing DRO frameworks to learn robustness-related parameters. In addition, the generalization analysis for continuous hyperparameter tuning provides a useful theoretical perspective on the proposed framework.

Critical weaknesses
  1. Although the framework is well formulated, its core methodological contribution appears to lie in combining established bilevel optimization techniques with existing DRO formulations. Bi-HDRO extends HDRO by making the perturbation radii learnable, and Bi-PG-DRO embeds probabilistic group assignment and robustness parameter optimization into a bilevel framework. The optimization procedure also relies largely on an existing bilevel minimax algorithm. Closely related components have also been examined in prior work. For example, WFRM[1] studies worst-case perturbations in feature space and their regularization effect. While this method targets different settings, they suggest that several building blocks of the proposed framework have already been studied independently. I recognize that adapting these ideas to Group DRO raises additional technical challenges, especially in keeping the lower-level optimization tractable. Even so, I find it difficult to identify a sufficiently distinct methodological contribution beyond the combination and adaptation of existing techniques. A clearer account of the additional insight offered by the framework would strengthen the paper.

  2. The proposed methods improve worst-group accuracy over several conventional DRO baselines, but the margins over stronger ablations are considerably smaller. In Table 1, Bi-HDRO with learnable group-specific perturbation radii achieves worst-group accuracies of 74.2%, 82.8%, and 57.4%, compared with 73.7%, 82.1%, and 57.9% for grid search over η\eta and ϵ\epsilon. More importantly, on CivilComments, Bi-GDRO with ϵ=0\epsilon = 0 attains higher worst-group and average accuracy than Bi-HDRO with learnable group-specific radii. This raises the question of what the learned perturbation mechanism contributes beyond the bilevel formulation itself. Furthermore, on CivilComments the improvement over the original HDRO baseline comes with a substantial drop in average accuracy. Such a trade-off is expected to some degree when optimizing for the worst group, but the paper does not adequately characterize whether the proposed method achieves a more favorable worst-group–average accuracy trade-off than simpler alternatives. Table 2 raises a similar concern. Bi-PG-DRO substantially outperforms the original PG-DRO baseline, yet its additional gain over the fixed-η\eta variant with KL regularization is modest, particularly on CMNIST and CivilComments. Finally, the computational advantage is not consistently demonstrated. The single-run runtimes in Table 7 show considerably higher cost for Bi-HDRO on some datasets. Avoiding grid search may yield an overall computational saving, but a comparison under a matched compute budget would be needed to establish this convincingly. Overall, the results suggest that the bilevel framework can be effective, but the additional benefit of its learnable robustness mechanisms has not been sufficiently isolated.

[1] Lei, Jingshi, et al. "Worst-case Feature Risk Minimization for Data-Efficient Learning." Transactions on Machine Learning Research (2023).

Questions for authors

Q1. Methodological novelty and relationship to prior work

Could the authors clarify the main methodological contribution beyond applying an existing bilevel minimax algorithm to established DRO objectives? In particular, how does Bi-HDRO relate to prior work on worst-case feature-space optimization, such as WFRM (Lei et al., 2023)? I understand that they address different problems, but a discussion of their conceptual and technical differences would help clarify the novelty of the proposed approach.

Q2. Contribution of learnable robustness parameters and accuracy trade-offs

The results in Tables 1 and 2 suggest that much of the improvement over the original baselines may already be obtained by simpler bilevel variants. Could the authors provide further evidence that learning the perturbation radii or robustness parameters yields a consistent advantage over carefully tuned alternatives? In particular, how do the authors explain the CivilComments results, where Bi-GDRO without feature perturbation slightly outperforms Bi-HDRO with learnable group-specific radii in both worst-group and average accuracy? A comparison of worst-group versus average accuracy under matched compute budgets would help clarify the practical benefit of the proposed method.

Q3. Theoretical guarantees and smoothness assumptions

How do the theoretical guarantees carry over to the practical setting, which involves nonconvex deep encoders and approximate lower-level solutions? In addition, Remark F.10 describes the robust logistic loss involving ϵ∥W∥2\epsilon\|W\|_2 as globally smooth and gives a gradient expression containing W/∥W∥2W/\|W\|_2. Since this expression is undefined at W=0W = 0, could the authors clarify how this point is handled and whether the associated Lipschitz assumptions remain valid?

2 comments

Dear Reviewer r5o3,

Please see our rebuttal below, and let us know if you have any further questions!

W1 and Q1. Novelty and relationship to prior work.

We thank the reviewer for mentioning this relevant work, and have incorporated it into our related works section.

WFRM minimizes a linear combination of clean and perturbed loss, without considering the worst group. It uses one shared perturbation radius across and approximates the inner maximization problem (perturbation) with an upper bound in practice (Proposition 1).

HDRO minimizes worst group adversarial loss, accounting for both inter- and intra-group shifts. It sets each group’s perturbation radius with a heuristic ϵg=ϵ/ng\epsilon_g=\epsilon/\sqrt{n_g} and approximates inner maximization with one projected gradient ascent step.

Bi-HDRO also accounts for both inter- and intra-group shifts and can learn the group-specific radii by optimizing worst-group validation risk in the upper level. Furthermore, as shown in Appendix C, for a linear classifier, it has the exact worst-case perturbation expression for binary cross entropy and hinge losses, along with tractable upper bounds for multiclass cross entropy and hinge losses.

Our work provides a unified framework for tackling the joint optimization of model weights and hyper-parameters efficiently, based on a recently proposed bilevel minimax solver, and validates the framework across intra-group/inter-group shifts with strong empirical results.

W2 and Q2. Empirical results.

Computational advantage. Previous runtime analysis included evaluation time and conveyed mixed messages, so we modified and updated Section 4.4 and Appendix E.3 to compare the validation accuracy given the same iteration budget and wall-clock per iteration, which shows that our methods take more time per iteration (due to the upper level validation loss) but yield consistently higher quality over baseline as seen from the curve. Furthermore, we want to emphasize that while grid search is faster per step, it takes an exponential amount of time as grid sizes increase to achieve similar results.

Contribution of each bilevel component and Worst-group/average accuracy trade-off. We agree with your concern, so we performed additional experiments comparing budget-controlled validation curves between Bi-HDRO and Bi-GDRO in Appendix E.4 (Figure 5(c) for worst-group CivilComments). It shows that while the worst-group test results between our methods and baseline are very close in Table 1, the worst-group validation results of our method are better. The average validation results are similar as shown in Figure 6.

Lastly, we want to mention that our upper level objective is flexible enough to incorporate average accuracy to mitigate the trade-off, should one be more interested in overall accuracy instead. We leave it as future work. Since the current upper level goal is worst-group accuracy, the trade-offs naturally manifest in our results.

Q3. Theoretical guarantees.

We appreciate your attentiveness in parsing our work. We have updated Remark F.10 to also cover the case where W=0W=0 and ∣W∣|W| is not differentiable. We can use subgradient in the analysis and the result still holds. In practice, the optimization of perturbed loss is done by introducing the additional parameter uu in Eq (18), whose objective is jointly smooth in (w,u)(w,u) with a closed-form projection operator.

Thank you for the revision. The WFRM discussion, the MLL baseline and the fix to Remark F.10 address parts of my review, but my main concerns remain.

(1) The novelty concern is not resolved. The closed-form perturbation for a linear binary classifier is a known result, group-specific radii come from HDRO, and the bilevel minimax formulation follows Shen et al. (2026). I still do not see a methodological contribution beyond combining these.

(2) The contribution of the learned radii is still not isolated. Appendix E.4 compares validation curves, which the upper level optimizes directly. On test data, Bi-HDRO is within one standard deviation of grid search over η\eta and ϵ\epsilon, and the new MLL baseline matches it on CelebA and exceeds it in worst-group accuracy on CivilComments (at lower average accuracy). Much of the gain over HDRO seems to come from the modified lower level. Reporting the learned ϵg\epsilon_g values for Bi-HDRO would also help.

(3) Section 4.4 matches optimizer steps, but a Bi-HDRO step costs 2 to 3 times an HDRO step (Table 7), and the solver hyperparameters differ across the new experiments (E.5.1, E.6.2).

(4) With k=2+dθk = 2 + d_\theta (Corollaries F.7 and F.12), the bound is vacuous for the encoders used, and the implementation does not solve the lower level exactly (one SAPD step per outer iteration).

I will keep my current score.

Reviewer_vKimOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides a bi-level framework to better tune hyperparameters common in GDRO algorithms, and in doing so shows significant performance improvements. The method is developed through regularization and approximations of the GDRO objective (and extension of GDRO), until reaching hyperparameters problems that can be readily solved using existing bi-level frameworks. They further provide theory that this approach achieves better generalization than grid search. Empirical results show it improves past methods when shifts occur within the groups (for GDRO), and when group attributes do not exist for all the data and hence must be learned.

Critical strengths
  1. The empirical results are strong, and controls for the role of their approach to hyperparameter tuning.
  2. Separating and approximating the hyperparameter tuning for GDRO and its extensions to a form amenable for the existing bi-level framework is novel (to the best of my knowledge).
Critical weaknesses
  1. While the empirical results focus on intra group shifts or probabilistic group membership, in the case of intra group shifts recent results have shown GDRO reduces to a convex problem followed by ERM when formulated in function space [1] (i.e., if the goal is to learn the best model): in particular it is solved by finding a particular mixture (a convex problem) and then training a model on it (ERM). Given much of the cited methods are about making the non-convex minimax gradient optimization stable which is no longer necessary (given the aforementioned result), the relevancy of the proposed method for that setting is unclear. I suggest the authors to consider rephrasing parts of the paper to focus primarily on the inter and probabilistic group shifts where (to the best of my knowledge) broad reductions to ERM do not exist. This also fits the empirical focus.

  2. Some empirical questions are left unanswered with the current writing. For example, how does the compute compare between the methods? “Efficient” baselines, e.g., [1] or even uniform training on all groups [2] are not discussed at all right now.

[1] Thudi, Anvith, and Chris Maddison. "Mixmax: Distributional robustness in function space via optimal data mixtures." International Conference on Learning Representations. 2025.

[2] Idrissi, Badr Youbi, et al. "Simple data balancing achieves competitive worst-group-accuracy." Conference on Causal Learning and Reasoning. PMLR, 2022.

Questions for authors

My main concern is largely around presentation and the past literature (see weaknesses). I believe these are easily fixed with writing and at most a few more experiments, and am willing to raise my score. Below are some specific questions to answer.

  1. My understanding is the method from [1] could be applied to the inter-group shift problems discussed making it a convex problem followed by ERM (avoiding the common stability issues of non-convex minimax). Could the authors elaborate on some limitations of [1] or how the method could be readily applied ontop of it, e.g., I understand [1] doesn’t deal with within group shifts or having to learn the groups.

  2. Similarly, for the experiments, could the authors elaborate on the computational cost of the methods? In particular, could the authors contrast this with uniform training on the groups and [1] (with possibly extended group set to include some perturbed groups for intra-group shifts). If possible, a few experiments also comparing performance would help understand how much more compute helps.

Minor clarifications:

  1. In equation 3 I believe there is a typo as the top line say min_{W} but the bottom line says W \in \arg\min. I presume the W in the top line is not meant to be there, but maybe I missed something? Similarly, for equation 9 and 11.
2 comments

Dear Reviewer vKim,

Please see our rebuttal below, and let us know if you have any further questions!

W1 and Q1. Conceptual Comparison to MixMax.

We thank the reviewer for pointing out the relevance of MixMax to our work. If we understand the MixMax paper correctly, it reformulates standard min-max GDRO as a max-min problem. In practice, it performs ERM on each group and then mirror ascent on mixture weights, before performing ERM again on the final model with the optimized weights. There are critical differences between MaxMix and our bilevel methods.

  • (1) Our framework can adaptively learn the robustness parameters, e.g., uniform weight regularization η\eta, representation perturbation ϵ\epsilon, and group membership predictor ϕ\phi based on validation performance.
  • (2) Our lower level formulation for training is convex-concave, since we only train on the convex last-layer classifier WW of the neural network (concave in qq due to quadratic regularization). We chose to separate out WW since it has been shown that fine-tuning the last layer classifier is sufficient for model robustness (Page 3, footnote 2). The nonconvex backbone θ\theta (not including linear WW) is tuned by the validation set in the upper level. Hence, our method is not solving the same nonconvex minimization problem over (W,θ)(W,\theta) which GDRO or MixMax tried to solve (Eq (1)).
  • (3) MaxMix is a min-max-min problem that requires three stages with two potentially nonconvex losses (ERM per group and ERM overall). Furthermore, the maximization part (the second stage of this method) requires kernel density estimation, which seems impractical for large scale dataset and can only be used as 1/N to cancel out p(x)p(x) in complex data. To transform it into a bilevel optimization problem, one could, for example, put the convex weight learning loss in the lower level, and the overall ERM loss components in the upper level, and warm up the lower level solution with ERM per group. This deviates from our minimax framing, but many bilevel algorithms have been developed to solve these kinds of problems when lower level is convex and upper level can be potentially nonconvex.

W2 and Q2. Empirical Comparison to MixMax.

Please check our updated Appendix E.5 for compute and performance comparison. We attempted to match tabular CelebA experiments in the MixMax paper. We additionally added MixMax under our Table 1 CelebA setup. Since performing KDE in high dimensional is impractical, we used constant 1/N for second stage convex training.

W2 and Q2. Computational Cost.

Bi-HDRO and its baselines under the available group information setting already use group-balanced sampling (i.e., equal number of samples from all groups in a batch). Bi-PG-DRO and its baselines use uniform sampling since group membership ground truth is not available. We added ERM training baseline (i.e., uniform sampling and no learnable weights) in Table 1.

Furthermore, we modified and updated Section 4.4 and Appendix E.3 to compare the validation accuracy given the same iteration budget and wall-clock per iteration, which shows that our methods take more time per iteration (due to the upper level validation loss) but yield consistently higher quality over baseline as seen from the curve. Furthermore, we want to emphasize that while grid search is faster per step, it takes an exponentially increasing amount of time as grid sizes increase to achieve similar results.

Q3. Typo.

WW is a decision variable of the whole optimization problem, so we have to write it in the upper level. For the lower level problem, W∈arg⁡min⁡W\in \arg\min specifies the constraint; this is the standard notation in bilevel problems.

Reviewer_vKimOpenReview ↗

Response to Rebuttal

I thank the authors for the rebuttal; it indeed answered my main questions/concerns. In particular, I had missed that only the last layer was being tuned, and so the formulation was convex in model parameters. I also appreciate the added computational complexity analysis.

As such I am raising my score to a 4.

Comments on the paper

3 comments

Dear Reviewers, we want to kindly follow up on our rebuttal as the discussion period is approaching its deadline today AoE. We would be grateful if you could give feedback on our work and let us know whether they address your concerns, or if you need further clarification. Thank you all for your efforts!

Dear Authors,

Thank you sincerely for your thorough and prompt rebuttal. I can see the effort you have put into addressing each of my concerns, and I want to acknowledge that the updated computational analysis is an appreciated contribution. After carefully reading your response, however, I still have some core concerns. I would like to explain some of them transparently, in the hope that this feedback will be useful for a future, stronger submission.

On the bilevel formulation. I still find myself uncertain about the core contribution of the bilevel abstraction. The authors motivate the transition from a standard min-max formulation to a bilevel optimization framework as a principled way to handle spurious correlations. However, as acknowledged in the rebuttal itself, the bilevel problem is ultimately solved via a penalty-based min-max reformulation (Eq. 14). At this point, if the method is solved as a min-max in practice, what is the concrete added value of the bilevel abstraction? The rebuttal argues that it enables automatic tuning of robustness hyperparameters. Considering the last one as a valid point, but this benefit feels incremental rather than foundational, and it is not clearly demonstrated to justify the additional complexity.

On computational cost. A 16x overhead per step remains a significant concern in practice, and the argument that grid search scales exponentially does not fully compensate for this, as many practitioners adopt efficient random or Bayesian search strategies.

I feel the paper would benefit from further major refinements.

Dear Reviewer drqR,

Please know that we truly appreciate your continued engagement in the rebuttal process. We will address the final comments below and wish you all the best.

Our bilevel formulation.

  1. Eq (14) is a numerical procedure for solving the proposed bilevel learning problem. It does not remove the bilevel nature of the optimization model or dependencies between two learning levels.
  2. Existing Group DRO maximizes over all group mixtures, allowing the adversary to concentrate entirely on the group with the largest empirical training loss. This worst-case protection is valuable, but empirical group losses may be especially noisy on small groups, so the resulting mixture may not yield the best worst-group performance on unseen data. Our bilevel framework explicitly separates robust training from learning how robustness should be imposed. The lower level trains the classifier under a fixed robustness mechanism; the upper level evaluates the resulting classifier on held-out data and updates that mechanism accordingly.

Computational Cost.

  1. As mentioned in our rebuttal and the revised submission, we provided a fair comparison of per-step training costs across methods in updated Appendix E.3 Table 7, correcting the earlier version with 16x overhead which inadvertently included evaluation time (which varied due different evaluation frequencies for different methods). Excluding CMNIST –– where the small input resolution and model size amplify fixed overhead –– our method is at most 5x slower per step, which better aligns with the expected computational cost of our minimax solver. Furthermore, our method exhibits higher quality worst-group validation accuracy controlling optimizer iterations (Figure 1 and 2); this helps offset the higher per-step cost.
  2. Bayesian optimization does not alleviate the prohibitive cost of training multiple models for multiple rounds. Alongside Bayesian optimization, bilevel optimization is a well-established paradigm for hyperparameter tuning. For example,
  • Pedregosa. Hyperparameter optimization with approximate gradient. ICML, 2016
  • Franceschi et al. Bilevel programming for hyperparameter optimization and meta-learning. ICML, 2018.
  • Shaban et al. Truncated back-propagation for bilevel optimization. AISTATS, 2019.

Empirically, our approach achieves better or at least competitive performance compared to exhaustive manual tuning while avoiding multi-round retraining. This is exemplified in our added MLL baseline in updated Table 1, where we used as many as 72 hyperparameter combinations for tuning. We will further clarify this motivation and include these references in the final manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.