acceptodds
Under review as a conference paper at ICLR 2027

-Bayes: Robust Belief Updating Under Subpopulation Shift

Abstract

Subpopulation shift changes the prevalence of latent groups between training and test distributions. Without group annotations or an explicit model of this shift, ordinary Bayesian updating treats the training mixture as representative, potentially favoring patterns supported by overrepresented groups. We introduce -Bayes, a group-annotation-free Bayesian method that learns the influence of each observation on the posterior, which in practice upweights informative minority examples while downweighting overrepresented ones. Starting from the principle of minimum discrimination information, we represent each observation through a soft constraint on its posterior expected loss. The associated Lagrange multipliers become learned likelihood powers, with ordinary Bayes recovered under a uniform allocation. Rather than specifying a separate tolerance for every observation, -Bayes learns a shared loss threshold, while controls the degree to which the likelihood powers may concentrate. We show that the resulting dual is an empirical distributionally robust optimization (DRO) problem over an order- R\'enyi ambiguity set. Consequently, without using group annotations during optimization, its robust objective controls the empirical risk of every latent group containing at least a fraction of the training sample and provides population worst-group guarantees under subpopulation mixture shift. We combine primal–dual optimization with IVON to make the posterior update practical for deep neural networks. Experiments on controlled and benchmark shifts demonstrate improved worst-group performance over ordinary IVON and show how the learned likelihood powers redistribute influence across observations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
bayesian deep learning, distribution shift, robust machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LoB5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ν\nu-Bayes, a Bayesian method for improving robustness under subpopulation shift without requiring group annotations. The method learns sample-specific likelihood weights and connects Bayesian updating with DRO and CVaR, providing theoretical worst-group guarantees. The authors combine the method with IVON and evaluate it on synthetic data, Colored MNIST, Waterbirds, and CelebA. The results show improved worst-group accuracy compared to ordinary IVON and several other baselines.

Critical strengths

I think the paper has an interesting idea, especially how it connects Bayesian inference with distributionally robust optimization. One of the main strengths is that the proposed method does not require group annotations during training which makes it useful in real applications where group information is not always available. Also, the theoretical part is good, and the connection between ν\nu-Bayes, CVaR, and DRO is explained clearly. The experiments also show promising results, especially in terms of worst-group accuracy on the synthetic dataset, Waterbirds, and CelebA. Another positive point is that the paper is well organized, and the mathematical formulation is supported by an actual implementation, althouugh I think the presentation could be a bit improved. Regarding the derivation, I did not find any issues and looks correct to me but I believe a bit more time would better for checking the proofs.

Critical weaknesses

I have some concerns about the experimental part and the novelty of the method. First, since the proposed formulation is equivalent to a KL-regularized CVaR objective, I think the authors should explain more clearly what is actually new compared to existing CVaR and DRO approaches. Also, some of the experimental comparisons may not be completely fair because the IVON and ν\nu-Bayes configurations use different learning rates and early-stopping settings, at least for cMNIST. This makes it difficult to know whether the improvement comes from the proposed weighting mechanism or from the different hyperparameters. Another concern is that the theoretical guarantees require some assumptions, such as the value of ν\nu being smaller than the proportion of the smallest group, but in practice the method assumes that group information is not available. I also noticed some differences between the experimental settings described in the appendix and the provided code, which may make reproducing the results more difficult.

Questions for authors

I have a few questions for the authors.

  • What is the main novelty of ν\nu-Bayes compared to directly using CVaR-DRO with a Bayesian posterior?

  • Can the authors provide an experiment where IVON and ν\nu-Bayes use exactly the same hyperparameters, so we can better understand the effect of the adaptive weights?

  • How can ν\nu be selected when the group proportions are unknown?

  • Since the method gives higher weights to high-loss samples, what happens when these samples are noisy or incorrectly labeled? Could this make the model focus more on outliers instead of minority groups?

  • Another question is about the CelebA results, where the worst-group accuracy improves but the average accuracy decreases. Can the authors explain this trade-off and show how it changes with different values of ν\nu?

  • Can the authors clarify the differences between the experimental configurations in the paper and the released code and provide the exact settings needed to reproduce the reported results?

Overall, I think the paper has a promising idea and a good theoretical motivation. However, I believe the experimental comparisons need to be more controlled, and the authors should clarify the novelty and practical limitations of the method. Addressing these points would make the contribution stronger and the results more convincing. Note: I am familiar with the literature in this field but I do not have a deep knowledge in it so I might have some misunderstanding of the paper. And, mostly, my concerns are related to the experimental part and novelty.

Reviewer_GzvSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies subpopulation mixture shift, where the frequencies of subgroups shift between training and test. The paper proposes ν\nu-Bayes, a constrained objective that generalizes Bayesian inference to learn a shared loss threshold that controls the influence of each training data point on the posterior. ν\nu-Bayes does not need group annotations, and the paper gives population worst-group guarantees under mixture shift. The paper validates ν\nu-Bayes against baselines on a synthetic mixture example and an image classification task. Results show favorable worst-group accuracy for ν\nu-Bayes against annotation-free baselines like ERM, JTT and IVON across the three datasets (cMNIST, Waterbirds and CelebA), and close to annotation-based methods like GroupDRO. When combined with deep feature reweighting (DFR), which uses group labels, ν\nu-Bayes is on par with annotation-based methods like GroupDRO and ERM-DFR.

Critical strengths

The paper establishes an annotation-free variational method for robustness to subpopulation mixture shift that scales (works with IVON). To the best of my knowledge, the closest Bayesian approach requires annotations to specify a prior [1]; ν\nu-Bayes therefore contributes an annotation-free alternative that gives an approximate posterior distribution rather than a MAP estimate. This is combined with better worst-group accuracy than annotation-free baselines and performance close to the annotation-based method GroupDRO when evaluated empirically.

Reference

Rudner, Tim GJ, et al. “Mind the gap: Improving robustness to subpopulation shifts with group-aware priors.” International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

Critical weaknesses

Without the group labels, we cannot know if ν≤nmin/N\nu \leq n_{\mathrm{min}}/N, so we cannot know if the guarantees that Theorem 2 provides actually apply to the models we are training. This extends to evaluating the risk bound given in Theorem 2, eq. 11: the denominator uses nminn_\mathrm{min}, which requires the group labels to know the group sizes. This considerably weakens the usefulness of the guarantee, and therefore the contribution, because the theorem’s applicability is unverifiable in the annotation-free setting.

Questions for authors
  • How do ν\nu-Bayes and ν\nu-Bayes-DFR compare against baselines on average accuracy?
  • ν\nu is an input to Algorithm 1, and ν\nu must be chosen such that ν≤nmin/N\nu \leq n_{\mathrm{min}} / N for the guarantee to hold. So how do we set ν\nu without group labels available? How was ν\nu chosen in the experiments?
  • How do you choose the prior pp for the KL term?
Reviewer_u4srOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes v-Bayes, which learns how much each training example contributes to a Bayesian posterior to improve robustness under subpopulation shift. The authors connect the method to distributionally robust optimization and CVaR, derive worst-group guarantees under stated assumptions, and implement it using IVON. Experiments on synthetic data, Colored MNIST, Waterbirds, and CelebA show improved worst-group accuracy over ordinary IVON and CVaR-Adam.

Critical strengths
  1. The formulation gives a clear interpretation of the learned weights as likelihood powers. The connection to robust optimization explains how these weights can protect underrepresented groups without group labels during fitting.

  2. The empirical results are promising. v-Bayes achieves higher mean worst-group accuracy than IVON and CVaR-Adam on all three image benchmarks. The weight analysis also shows that minority groups receive greater influence during training.

Critical weaknesses
  1. The paper needs a clearer comparison with Mhammedi et al. (2020) [1], who already study CVaR of posterior-expected losses with PAC-Bayesian guarantees. Without this comparison, it is difficult to assess what the proposed objective and theoretical results add.

  2. The comparison with CVaR-Adam changes both the optimizer and the use of posterior inference. Since v-Bayes uses IVON, some of the gains could come from the optimizer. More detail on hyperparameter tuning and checkpoint selection is also needed to assess whether the comparisons are fair.

[1] Mhammedi, Z., Guedj, B. and Williamson, R.C., 2020. Pac-bayesian bound for the conditional value at risk. Advances in Neural Information Processing Systems, 33, pp.17919-17930.

Questions for authors
  1. How do Equation (13) and Theorem 2 differ from the results of [1]? What does the proposed formulation add?

  2. Could you compare the posterior and point-estimate versions using the same optimizer and comparable regularization? This would help establish whether posterior inference explains the gains.

  3. How were hyperparameters and checkpoints selected for each method? Were group labels used, and did the methods receive comparable tuning budgets?

[1] Mhammedi, Z., Guedj, B. and Williamson, R.C., 2020. Pac-bayesian bound for the conditional value at risk. Advances in Neural Information Processing Systems, 33, pp.17919-17930.

1 comment

Dear Reviewer u4sr,

The authors have responded to your review. They addressed the relation to Mhammedi et al. (2020) and added ablations comparing the posterior and point-estimate versions trained with the same optimizer, which is your Q2. They also corrected the Waterbirds results after finding an implementation error.

Author–reviewer discussion closes on 18 Nov. Could you please read their response and see if they have addressed your concerns (and to what extent)? Please also add your own human-written assessment to the AI-use documentation field, as requested by PC.

Thank you,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.