- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper addresses the challenge of spurious shortcuts, i.e., non-causal features (e.g., background context) that lead models to make incorrect predictions when the shortcut is absent. In this context, the authors propose a novel reformulation for training a robust model to mitigate this issue. Specifically, while standard robustness methods often rely on a min-max formulation, the authors propose shifting this to a bilevel optimization problem, where both the outer and inner objectives are formulated as min-max problems. This nested structure forces the model to ignore spurious cues by optimizing for robustness across different settings. The authors validate their methodology on binary classification datasets (i.e., CMINST, CelebA, and CivilComments) where the spurious signal is known in advance and amplified. For example, in CMNIST, the color of the digit corresponds to the short signals that the authors want the model to prevent learning.
Critical strengths
The paper identifies a well-defined and highly relevant problem in ML, i.e., the tendency of models to rely on spurious shortcuts rather than causal features. The challenge is relevant as shortcuts can influence multiple trustworthy principles (e.g., integrity and robustness).
The presentation of related work provide a clear landscape of how the proposed method sits within the existing literature on robustness and optimization.
Beyond its complexity, the transition to a bilevel optimization framework is a mathematically ambitious attempt to tackle a difficult problem.
Critical weaknesses
### Complex conceptual interpretability and higher computational complexity
While reading the paper, I had the feeling that the transition from a standard min-max formulation to a nested bilevel min-max formulation is mathematically not obvious, sometimes complex in a few passages (*), and not intuitive. Yet the presentation and storytelling in the paper lack a clear intuitive or heuristic justification for why this specific structure is “natural” for solving spurious correlation problems. The formulation feels somewhat arbitrary. Furthermore, bilevel formulations are known to be computationally complex and are typically approximated as min-max processes, which means exactly the opposite of what the authors are doing in this paper. Therefore, I believe the paper is missing a clear explanation in this direction and justification for the bilevel formulation, and a more intuitive justification to support the readers.
Lastly, in doing so, the proposed bilevel formulation significantly increases the computational burden compared to existing state-of-the-art (SOTA) methods. As evidenced by the results in the Appendix (Table 7), the method can be up to 16x slower than HDRO in certain scenarios. In the context of modern deep learning, where training efficiency is a primary concern, a 16-fold increase in training time is a massive barrier to adoption.
(*) Passage complexity can possibly be a limitation on my side. Therefore, I’ll be happy to know the opinion of the other reviewers in this regard. Right now, this is not an ingredient that justifies my score.
Lack of self-consistency
The main paper (limited to 8 pages) is not self-consistent, meaning that without the Appendix, it is almost impossible to replicate and understand the experimental analysis and outcomes proposed in the main paper. To be specific, we can take the example of the experimental setup, where models, hyperparameters, and (importantly) evaluation metrics used in the main paper are reported only in the Appendix. The primary basis for comparison (the distinction between Average and Worst-case accuracy) is relegated entirely to the Appendix despite being the central pillar of the discussion and results. A reader should not be required to parse the Appendix, as also written in the ICLR guidelines, but without that the main paper hardly survives alone. The Appendix is not used as a section where to report deeper analysis or proofs, but as a continuation of the main.
Limited empirical scope and margin of improvement
T The experiments are confined to binary classification tasks using low-dimensional data, which does not sufficiently demonstrate how the method scales to complex, high-dimensional datasets (e.g., ImageNet or large-scale NLP tasks). I will revise my score if the authors will take the chance to include additional models and large scale datasets.
Furthermore, the performance gains appear marginal compared to the SOTA. Looking at Table 1, the proposed method is outperformed by both GroupDRO and HDRO in terms of Average accuracy in two out of three tested datasets (notably by up to 10% in the Civil Comments dataset). While the proposed method shows promise in improving Worst-case accuracy, the trade-off is a significant degradation in average performance and a massive increase in compute. In my opinion, the results do not demonstrate remarkable improvements over the SOTA, despite the higher computational costs. There is for sure an improvement over the worst-case accuracy, but is it something that cannot be addressed with a different hyperparameter setup from other methods? The authors should clarify whether these results are truly an improvement or simply a shift in the error distribution, and whether the baselines could achieve similar results through more intensive hyperparameter tuning.
(Minor) ambiguity in AI usage statement
The authors state that “AI tools have been utilized to clean and reformat dataset”. Given that the datasets used are standard benchmarks, it is unclear whether this refers to automated scripts, LLM-based cleaning, or standard preprocessing pipelines. To what clean and reform is referring to in these standard datasets?
Questions for authors
Can the authors provide a more intuitive view that justify the bilevel formulation? Would it scale to more complex and higher dimensional tasks?
Is the binary setting a clear limitation of the approach?
The methodology presents for sure an improvement over the worst-case accuracy, but is it something that cannot be addressed with a different hyperparameter setup from other methods?
In the AI usage statement, to what clean and reform is referring to in these datasets?
2 comments
2/2
Extension to multiclass classification and complex tasks
We respectfully disagree with your comment that our experiments are limited to “low-dimensional data” unless you meant target dimensions: our experiments are not based on the likes of LIBSVM, Gaussian samples, or tabular attributes of CelebA. CelebA contains raw images, and CivilComments contain natural language sentences. Both are high dimensional and respectively need ResNet and DistilBERT for representation learning. CMNIST, CelebA, and CivilComments are all well established benchmarks used in many previous DRO works.
However, we agree with your concern that our tasks are all binary classifications. We already derived an upper bound for the inner perturbation maximization problem in Appendix C.2, and we have yet to put it to use. To address this concern, we created an image dataset with inter-group shift based on ImageNet-9 [1] by pairing extracted foreground (which contains the object/label information) and background (with foreground cropped and filled with tiling). We could not perform intra-group shift since it does not have attribute annotations similar to CelebA. This new dataset contains 9 classes and 1 attribute (whether background donor class matches foreground donor class or not), naturally creating 18 groups. Moreover, we intentionally create samples with mismatching backgrounds so that they only constitute 5% of the entire training set, but 50% in the validation/test set. We report the setup and results in Appendix E.6. Results show that Bi-HDRO is competitive to robust single-level methods. This is expected since latent perturbation can really make a larger difference only when intra-group shifts are present, as reported in [2].
Alternative hyperparameter setup for baselines
Grid searching over yields flat optimal validation curves within the optimal range, so it is not a matter of “tuning intensity”. While we have tuned all baseline methods to the best of our ability, we want to make our point more convincing, so we created a new single-level nonconvex-concave minimax baseline for Bi-HDRO with three hyperparameters to be tuned using grid search. This is essentially Bi-HDRO’s lower level with the nonconvex backbone added to the minimization block. Without the upper level tuning problem, this baseline does not adaptively learn the parameters. This yields 75 hyperparameter combinations for CMNIST and CelebA and 45 combinations for CivilComments, and it still performs worse compared to our bilevel methods. See Appendix D.3.2 for implementation details and Table 1 for added results.
AI statement
It’s for creating and verifying intra-group shifts actually occurred. We clarified it along with new ImageNet-9 experiments.
References
[1] Xiao, Kai, et al. "Noise or signal: The role of image backgrounds in object recognition." ICLR. 2021.
[2] Jo, Sung Ho, Seonghwi Kim, and Minwoo Chae. "Mitigating spurious correlation via distributionally robust learning with hierarchical ambiguity sets." ICLR. 2026.
1/2
Dear Reviewer drqR,
We will group your concerns among weaknesses and questions into the following to address them collectively. Please let us know if we did not understand or address your concern properly.
Intuition behind bilevel formulation
The bilevel formulation may seem contrived if the same task can be solved with a single-level variant. However, we want to emphasize that the penalty form of our bilevel is solved as a minimax problem in practice as shown in Eq (14). Our motivation is that recently developed DRO methods all introduce some form of robustness mechanisms, and as shown in examples like HDRO and PG-DRO, those newly introduced robustness parameters may be sensitive and tuning of them needs to be exhaustive. This is where our bilevel framework that can be used to automatically tune those parameters kicks in.
A hopefully intuitive roadmap. In Section 2.1, we show the formula of GDRO, which is a minimax optimization problem. In Section 2.2, we show that the canonical GDRO can be reformulated as a bilevel optimization problem where the upper level minimax problem over validation set is nonconvex-concave and lower level minimax problem over training set is convex-concave. In Section 2.3 and 2.4, we show the generalizability of this bilevel framework under two settings. Furthermore, Algorithm described in Section 2.5 can be used to solve these problems. Lastly, we provided generalization guarantee of our bilevel methods in Section 2.6, conjecturing that they will perform better as they avoid discretization error which is inevitable in grid search.
Our comment on conceptual interpretability and self-consistency. When content is abundant, it is natural to relegate less important experimental details (e.g., hyperparameters) to the appendix. Previously, due to page limit, we omitted problem setup details that are common to baselines/related works. Now we have added detailed explanations of metrics, possible extension to trade off the worst group and average performance, and more ablations in the main text.
Computational complexity
We fully agree with this concern. Previous runtime analysis included evaluation time and conveyed mixed messages, so we modified and updated Section 4.4 and Appendix E.3 to compare the validation accuracy given the same iteration budget and wall-clock per iteration, which shows that our methods takes more time per iteration (due to the upper level validation loss) but yield consistently higher quality over baseline as seen from the curve. Furthermore, we want to emphasize that while grid search is faster per step, it takes an exponential amount of time as grid sizes increase to achieve similar results.
Average performance degradation
We want to mention that our upper level objective is flexible enough to incorporate average accuracy to mitigate the trade-off, should one be more interested in overall accuracy, and we leave it as future work. Since the current upper level goal is worst-group accuracy, the trade-offs naturally manifest in our results. We additionally compared budget-controlled validation curves between Bi-HDRO and Bi-GDRO in Appendix E.4 (Figure 5 and 6). It shows that while the test results between our methods and baseline are very close in Table 1, the worst-group validation results of our methods are better. The average validation results are similar as shown in Figure 6.