acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Mixture Variational Inference for Spike-and-Slab Regression

Abstract

Correlated predictors can support competing sparse explanations with similar predictions, making joint uncertainty about variable inclusion difficult to capture with mean-field approximations. We develop an adaptive fitting procedure for mixtures of product distributions in Gaussian regression with a point-mass spike-and-slab prior. It minimizes reverse Kullback-Leibler divergence directly on inclusion indicators and active coefficients, jointly refining component parameters and weights as the mixture grows. This avoids an additional divergence penalty on unused latent coefficients under independent augmentation. Our analysis relates approximation accuracy to mixture size, support coverage and dependence within supports, and establishes contraction, selection consistency and a Bernstein-von Mises approximation under explicit conditions on the prior, posterior concentration and variational error. On all 250 simulated datasets with exact posterior references, mixtures reduce errors in inclusion probabilities, grouped support probabilities and coefficient covariance relative to multistart mean field. Comparisons at fixed mixture size and common initialization favor direct joint refinement over augmented or restricted refinement in posterior divergence. Complete stagewise fitting can nevertheless be more accurate near collinearity. The results support direct joint refinement for posterior approximation while showing that local gains do not ensure superiority of the full adaptive search.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Variational inference, Spike-and-slab priors, Bayesian variable selection, Mixture approximations, High-dimensional regression, Bernstein-von Mises theorem

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3CuPOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

/

Critical strengths

/

Critical weaknesses

/

Questions for authors

/

1 comment

Thank you for reading the paper and for your assessment. The revision clarifies the algorithm and its relation to prior work, focuses the theory on refinement and splitting, and adds LSVI, SuSiE, a larger formal simulation study and a real-data application. We have also made the scope of the theoretical and empirical claims more explicit.

Reviewer_PqahOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper propose a variational inference method to approximate the spike-and-slab posterior distributions. Intuitively, they try to approximate the posterior with mixtures of Gaussian-Bernoulli components and design a way to adaptively adjust the component number and weight.

On the theory side, the prove certain aympototic guarantee (contraction, selection consistency) under several extra assumptions.

On the experiment side, they conduct a small-scale simulation to compare their method with the vanilla MFVI.

Critical strengths

N/A

Critical weaknesses

In general, the paper does not have an independent literature review section and thus lack of comparisons with many important works on this line of research. [1-5]

Theoretical Arguments: Overall I do not see any technical advantage of the analysis. Even after taking one step back, these theorems also fail to provide proper insight or serve as complementary of their story. The evidence of my statement is as follows:

(a) the requirement of the sparse prior in Assumption 1 is very restricted and lack of applicability: The require wn=p−aw_n = p^{-a} for a>11a > 11. However, under this prior, the probability of empty set tends to one, which implies significantly strong preference for null model and is rarely used in this field. Thus, the authors should justify the motivation for using this assumption.

(b) The authors also state several important extra assumptions in the statement of Theorem 1-3. All of them are given without proper discussion and justification. e.g. the spectral lower bound for sparse submatrices in Thm 1; the support mismatch decay in Thm 2. Although there is no direct prior work can recover the author's results, but there are several works established rigorous non-asymptotic guarantees with similar settings, such as [1-3]. In particular, the hardness of sampling from this posterior, but thm 2 just assume the support mismatch rarely happens. Also in thm1 and 2, they directly assume an optimization error; in thm 3 they even assume both the approximation and optimization error is small, which intuitively is assuming their proposed family can well approximate the posterior and their algorithm can approximately find the optimum. With this type assumption, any result is not surprising to me. I doubt the applicability of those conclusions without further justification.

If the authors' intention is to show technical novelty, the author should properly justify most of these assumptions mentioned and compare with other related works; if the the authors' intention is to show some statistical insight, they should articulate those assumptions in the beginning and explain the scope and limitations behind them, e.g. which of them are standard assumptions, which of them are established by prior works, which of them are new and how they affect the argument.

Experiments: The experiment reesults is insufficient to support the empirical advantage of their method:

  1. They only run experiment on synthetic environment. Given the weakness on the theory side, lacking real-world dataset report is unacceptable. The scale of the synthetic environment is also restricted (s0=5,n=80,p≤100s_0 = 5, n = 80, p \le 100, signal is designed.)
  2. Spike-and-slab regression is an important task in statistics field and many methods have been proposed to conduct variable selection. The authors should also incorporate several recent algorithms as other baseline to demonstrate the empirical advantage of their methods, e.g. [4,5] may be applicable.
  3. Under the case when ρ=0.9,0.99\rho = 0.9, 0.99, the simple heuristic is comparabale (ρ=0.9\rho = 0.9) or much better than (ρ=0.99\rho = 0.99) their method as shown Table 7. Given the fact that the motivation of this work is to address issues under these cases, the author should also explain why their method is worse than stagewise.

[1] Yang Y, Wainwright M J, Jordan M I. On the computational complexity of high-dimensional Bayesian variable selection[J]. 2016.

[2]Montanari A, Wu Y. Provably efficient posterior sampling for sparse linear regression via measure decomposition[J]. Journal of the American Statistical Association, 2026, 121(553): 636-654.

[3]Kumar, S., Sarkar, P., Tian, K. & Zhu, Y.. (2025). Spike-and-Slab Posterior Sampling in High Dimensions. <i>Proceedings of Thirty Eighth Conference on Learning Theory</i>, in <i>Proceedings of Machine Learning Research</i> 291:3407-3462 Available from https://proceedings.mlr.press/v291/kumar25a.html.

[4] Fay Y L, Chopin N, Barthelmé S. Least squares variational inference[J]. arXiv preprint arXiv:2502.18475, 2025.

[5] Wang G, Sarkar A, Carbonetto P, et al. A simple new approach to variable selection in regression, with application to genetic fine mapping[J]. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2020, 82(5): 1273-1300.

Questions for authors

See weakness.

1 comment

1. Related work and contribution

Section 2 now discusses all five papers you suggested. Yang et al., Montanari and Wu, and Kumar et al. study statistical or sampling guarantees. Le Fay et al. and Wang et al. provide two of our baselines. AMVI builds on sequential boosting and joint optimization. Its specific construction gives an added component and a distribution-preserving Bernoulli split separate opportunities for refinement before choosing between them (Sections 2 and 4).

2. Sparse prior

We removed the original asymptotic theorems on contraction and selection consistency, along with the sparse-prior scaling they required. The revised results study refinement for the observed data and allow any fixed inclusion probability 0<ω<10 < \omega < 1 (Section 5 and Appendix A.3).

3. Assumptions and scope

You are right that assuming small optimization error does not establish that the fitted algorithm achieves it. We withdrew the original asymptotic claims. The local analysis requires neither a sparse eigenvalue bound nor vanishing support mismatch. Proposition 2 gives a sufficient condition for one clipped update to improve the objective. A global error rate and selection consistency for the returned fit remain open (Section 5.3 and Appendix A.3).

4. What the theory explains

At an interior stationary fit, splitting one Bernoulli component preserves the distribution and the ELBO. Theorem 1 shows that the ELBO derivative at the start of exact refinement satisfies dduELBO⁡(q(u))∣u=0>0\left.\frac{d}{du}\operatorname{ELBO}(q(u))\right|_{u=0} > 0 if the split coordinate has a nonzero residual interaction with another indicator, averaged under that component. AMVI chooses where to split using component weight and inclusion uncertainty. It can therefore choose a coordinate whose averaged interactions are all zero, in which case the initial ELBO derivative is zero. Corollary 1 shows that dependence can change to first order even when all marginal inclusion probabilities have zero initial derivative. For the first addition, Proposition 3 also shows why revisiting a stationary old component helps: moving any differing inclusion probability slightly farther from the new component’s probability increases the objective (Section 5.2 and Appendix A.2).

5. Study size and real data

Section 6 now includes 100 independent datasets in each of 18 settings. We enumerate the posterior at p=10,12,14p=10,12,14 and compare evidence lower bounds (ELBOs) at p=20,40,80p=20,40,80. The superconductivity application has 21,197 observations and 81 predictors. On one composition-grouped train/test split, AMVI had higher ELBO than sequential VBoost in all 20 paired fits across two priors and ten optimizer seeds. Its mean prediction error was slightly higher than VBoost’s and LSVI’s (Table 2).

6. External baselines

We added both suggested methods using the authors’ LSVI optimizer and the original susieR package. MFVI, LSVI and our Bernoulli adaptation of sequential VBoost target the same spike-and-slab posterior as AMVI. SuSiE uses a different prior, so we compare its predictions. Table 4 and Appendix B.3 document the implementations and the changes needed to apply VBoost to binary indicators.

7. When stagewise fitting wins

In the original comparison at correlation ρ=0.99\rho=0.99, stagewise fitting achieved lower mean KL than joint refinement. The two procedures followed different proposal paths, so that comparison could not isolate the effect of joint refinement. In the revised study, sequential VBoost had lower KL on 175 of the 900 exact-reference datasets and AMVI had lower KL on 725. The median paired difference favored AMVI in all nine settings (Section 6.1 and Appendix C.2).

8. Variable selection

At predictor correlation ρ=0.99\rho=0.99, AMVI’s true positive rate and empirical false discovery rate were 83.7% and 21.4%, compared with 83.7% and 21.1% for the exact posterior at the same inclusion threshold (Table 9). In a post hoc check on those 300 datasets, AMVI’s mean absolute error for the probability of selecting exactly one predictor from a correlated group was 1.08 percentage points. Imposing independence with the same marginals raised it to 19.44 points (Section 6.1). This measures the contribution of fitted dependence to that selection probability.

Reviewer_jXTqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies variational inference for spike-and-slab regression using mixtures of product spike-and-slab distributions to capture joint uncertainty among correlated predictors.

The paper proposes two elements in the variational framework: (1) direct joint refinement, which optimizes all parameters by directly minimizing reverse KL divergence between the variational posterior and the true posterior ("direct" means there is no use of auxiliary variables introduced in prior work). (2) an adaptive fitting procedure, which sequentially increases the number of mixture components and applies joint refinement (1).

The proposed method is validated on synthetic datasets. While the proposed direct joint refinement (1) was generally effective, the effectiveness of the adaptive fitting procedure (2) was less pronounced.

Critical strengths

1. Improved posterior approximation via direct joint refinement

The proposed direct joint refinement achieves lower reverse KL divergence on synthetic data.

Critical weaknesses

1. Poor presentation and readability

Throughout the paper, the presentation lacks clarity, making the technical content difficult to follow. The logical connections between sentences and paragraphs are often unclear, particularly in the Abstract and Introduction. Even when individual sentences seem understandable, the intended meaning of certain expressions and the reason for their placement in the argument are not apparent. As a result, considerable effort is required to follow the reasoning, and the meaning of earlier passages often becomes clear only after reading later sections. Further, several factors of this confusion for this confusion are summarized below.

a. Insufficient explanation of methodological concepts

  • Line 18: It is hard to identify what "augmentation" means in this context.
  • Line 53: The distinction between the existing objective and the proposed direct objective is not clearly explained in the introduction. It is still understandable that they are different. But the comprehension for how different is hindered by obscure expressions comprising the sentences.

b. Unclear significance of theoretical contributions

The theoretical results are mostly enumerated without explaining their main implications in Abstract, Section 1 and Section 5. Besides it is unclear how they contribute to answer the central research question.

c. Ambiguous or insufficiently explained terminologies

Many expressions are used without clearly identifying the mathematical objects or computational procedures they refer to. The expressions below are examples:

  • Line 144: discarded-variable conditional law
  • Line 153: conditional cost
  • Line 202: reference point
  • Line 279: exact references

2. Insufficient evidence for the effectiveness of adaptive fitting

While the proposed direct joint refinement seem to be effective, the other proposal, the benefits of the proposed adaptive fitting procedure are less clear. Especially, there are some situations where stagewise fitting outperforms the adaptive approach (under strong collinearity between predictors). A deeper investigation into the situations where the proposed adaptive procedure is preferable would be necessary to establish its practical value.

Questions for authors

Useful implications for the future practitioners:

Stagewise fitting and joint refinement seem to perform better at different levels of predictor correlation. Can the authors provide examples of typical real-world datasets for which each method would be preferable?

1 comment

1a. Clarifying the objective

We removed the discussion of augmentation. Section 3.3 now states directly that AMVI maximizes the evidence lower bound.

1b. Why the theory matters

Theorem 1 explains how a split can be useful even though it initially leaves the fit unchanged. At an interior stationary fit, a nonzero residual interaction involving the split coordinate gives refinement an improving direction. Proposition 2 states when a finite numerical update increases the ELBO (Section 5).

1c. Terminology and algorithm

Section 4 and Algorithm 1 now trace one round from a single incumbent. AMVI constructs an addition and a split, refines each separately, and uses fresh integration samples to check the ELBO gains before accepting at most one. The exact reference in Section 6.1 is the posterior obtained by enumerating all variable subsets after fitting.

2. Evidence for adaptive fitting

Across all nine settings with an exact posterior, AMVI had lower mean KL divergence, support total variation and coefficient covariance error than the other methods targeting that posterior. Sequential VBoost still won on KL for 175 of 900 individual datasets. These results assess the whole procedure. See Figure 2 and Section 6.1.

Question for authors. When is each fitting strategy preferable?

We added a superconductivity application in which 81 material descriptors are used to predict critical temperature. On the same train/test split, AMVI achieved a higher ELBO than sequential VBoost in all 20 paired fits (ten seeds under each of two priors). VBoost had slightly lower mean test MSE under both priors, with differences below 0.06%. Thus AMVI gave the closer posterior approximation on this dataset, with little difference in prediction. One dataset cannot establish when predictor correlation favors either strategy (Section 6.2, Table 2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.