- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes -Bayes, a Bayesian method for improving robustness under subpopulation shift without requiring group annotations. The method learns sample-specific likelihood weights and connects Bayesian updating with DRO and CVaR, providing theoretical worst-group guarantees. The authors combine the method with IVON and evaluate it on synthetic data, Colored MNIST, Waterbirds, and CelebA. The results show improved worst-group accuracy compared to ordinary IVON and several other baselines.
Critical strengths
I think the paper has an interesting idea, especially how it connects Bayesian inference with distributionally robust optimization. One of the main strengths is that the proposed method does not require group annotations during training which makes it useful in real applications where group information is not always available. Also, the theoretical part is good, and the connection between -Bayes, CVaR, and DRO is explained clearly. The experiments also show promising results, especially in terms of worst-group accuracy on the synthetic dataset, Waterbirds, and CelebA. Another positive point is that the paper is well organized, and the mathematical formulation is supported by an actual implementation, althouugh I think the presentation could be a bit improved. Regarding the derivation, I did not find any issues and looks correct to me but I believe a bit more time would better for checking the proofs.
Critical weaknesses
I have some concerns about the experimental part and the novelty of the method. First, since the proposed formulation is equivalent to a KL-regularized CVaR objective, I think the authors should explain more clearly what is actually new compared to existing CVaR and DRO approaches. Also, some of the experimental comparisons may not be completely fair because the IVON and -Bayes configurations use different learning rates and early-stopping settings, at least for cMNIST. This makes it difficult to know whether the improvement comes from the proposed weighting mechanism or from the different hyperparameters. Another concern is that the theoretical guarantees require some assumptions, such as the value of being smaller than the proportion of the smallest group, but in practice the method assumes that group information is not available. I also noticed some differences between the experimental settings described in the appendix and the provided code, which may make reproducing the results more difficult.
Questions for authors
I have a few questions for the authors.
-
What is the main novelty of -Bayes compared to directly using CVaR-DRO with a Bayesian posterior?
-
Can the authors provide an experiment where IVON and -Bayes use exactly the same hyperparameters, so we can better understand the effect of the adaptive weights?
-
How can be selected when the group proportions are unknown?
-
Since the method gives higher weights to high-loss samples, what happens when these samples are noisy or incorrectly labeled? Could this make the model focus more on outliers instead of minority groups?
-
Another question is about the CelebA results, where the worst-group accuracy improves but the average accuracy decreases. Can the authors explain this trade-off and show how it changes with different values of ?
-
Can the authors clarify the differences between the experimental configurations in the paper and the released code and provide the exact settings needed to reproduce the reported results?
Overall, I think the paper has a promising idea and a good theoretical motivation. However, I believe the experimental comparisons need to be more controlled, and the authors should clarify the novelty and practical limitations of the method. Addressing these points would make the contribution stronger and the results more convincing. Note: I am familiar with the literature in this field but I do not have a deep knowledge in it so I might have some misunderstanding of the paper. And, mostly, my concerns are related to the experimental part and novelty.
Dear Reviewer u4sr,
The authors have responded to your review. They addressed the relation to Mhammedi et al. (2020) and added ablations comparing the posterior and point-estimate versions trained with the same optimizer, which is your Q2. They also corrected the Waterbirds results after finding an implementation error.
Author–reviewer discussion closes on 18 Nov. Could you please read their response and see if they have addressed your concerns (and to what extent)? Please also add your own human-written assessment to the AI-use documentation field, as requested by PC.
Thank you,
AC