acceptodds
Withdrawn from ICLR 2027

DSS-LVR Flow: A Unified Method for Structural Sparsity in Bayesian Neural Networks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bayesian neural networks (BNNs) provide uncertainty-aware modeling of complex nonlinear relationships in scientific applications. Their flexibility, however, comes with rapidly expanding and redundant parameterizations, making it important to identify which predictors, hidden units, and larger computational structures are actually supported by the data. Discrete spike-and-slab (DSS) priors provide a natural Bayesian mechanism for this purpose: the point-mass spike enables exact exclusion, while the continuous slab retains posterior uncertainty for active parameters. We introduce DSS-LVR Flow, a flow-based method built on a DSS latent-variable representation (DSS-LVR), in which a group-specific activation latent variable and a shared continuous threshold jointly determine each inclusion decision. By changing which parameters share an inclusion state, the same construction supports predictor- and hidden-unit-level selection, while combinations of these primitive states induce edge and path activity without additional structural latent variables. DSS-LVR Flow uses a role-aware inverse autoregressive flow (IAF) to approximate the joint posterior. It also introduces a smooth, bounded, exact-zero structural activation function with a controllable transition region for gradient-based optimization. Experiments show that DSS-LVR Flow efficiently approximates parameter posteriors with accuracy close to that of long-run MCMC. In multilayer perceptrons, it also supports multiple structural sparsity targets, producing highly sparse structures while preserving accurate nonlinear function recovery.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Bayesian neural networks, spike-and-slab priors, structural sparsity, normalizing flows, variable selection, posterior uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_kaZmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a spike-and-slab prior for BNNs hat can switch off whole groups of weights. Each group is turned on or off by comparing a latent variable to a shared threshold. Experiments use synthetic regression data, plus a comparison to MCMC on a small one-hidden-layer network.

The paper is in its current form clearly in an unfinished stage.

Critical strengths

--

Critical weaknesses

The first critical weakness of this paper is that it is incomplete, e.g., the main text points to appendices which are not yet written. I see this alone as a reason for a clear reject and kindly ask to next time please submit a finished paper.

Also, I would highly recommend the authors to a) do a thorough experimental study of the proposed method including on real-world data and also report metrics such as NLL and b) before submission double check all the claims and ground them in theoreticla or empirical results.

Lastly, the current submission only has 7 references. Of course numbers are not everything here but it still shows that a thorough review of related literature on BNNs and relevant subvariants has not taken place.

Questions for authors

I have no further clarifying questions.

Reviewer_HSKnOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a DSS-LVR Flow for structural sparsity in Bayesian neural networks. Network weights are represented as continuous slab values multiplied by group-specific activation gates, with shared thresholds controlling sparsity. Group definitions enable predictor and hidden-unit selection, while edge and path activity follow from these decisions. A role-aware inverse autoregressive flow approximates the joint posterior over weights, activation, and threshold variables. Smooth exact-zero gates facilitate gradient-based optimisation, while hard threshold events determine posterior inclusion probabilities. Synthetic experiments evaluate predictor selection, nonlinear function recovery, structural sparsity, and posterior agreement with MCMC in a shallow network.

Critical strengths

The framework provides a joint construction for combining predictor-level and hidden-unit sparsity while retaining posterior uncertainty. Joint flow-based inference can represent dependencies omitted by mean-field approximations, and the exact-zero gates offer an appealing alternative to purely continuous shrinkage. The experiments use repeated simulations and assess both selection and function recovery. The paper also explicitly reports failure regimes, helping distinguish successful compression from excessive shrinkage that destroys predictive information.

Critical weaknesses

The evidence is insufficient to establish the framework’s reliability and distinct contributions. Evaluation is limited to synthetic settings, and performance collapses under lower signal-to-noise ratios and higher dimensionality, undermining its high-dimensional motivation. Missing ablations leave the benefits of role-aware ordering, flow flexibility, and the proposed gate unresolved. Several appendix sections contain headings without substantive details. The relationship between the smoothed optimisation target and the original hard-gated posterior is insufficiently justified; hard-event summaries do not establish posterior equivalence. MCMC validation covers only a shallow setting, and the runtime comparison lacks controlled, clearly documented hardware equivalence.

Questions for authors
  1. Does optimisation with the smooth gate target the original hard-gated posterior or a modified posterior? Can you quantify sensitivity to the transition width?

  2. How much does role-aware ordering improve over ordinary coordinate permutations and a capacity-matched flow?

  3. Can you isolate the contributions of the grouped prior, posterior dependence, and gate design through ablations?

  4. Why do several baselines either retain nearly every predictor or collapse entirely? What tuning and convergence checks were performed?

  5. Can you provide real-data results, complete implementation details, and runtime comparisons under matched hardware conditions?

Reviewer_gMWYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DSS-LVR Flow, a framework for structural sparsity in BNNs. Network parameters are constructed from continuous latent variables through group-specific activation maps, allowing input variables and hidden-units selection within a shared formulation. A role-aware AR flow approximates the joint posterior over slab variables, activation variables, and thresholds. The paper also introduces smooth activation maps that retain an exact-zero region to facilitate gradient-based optimization. Synthetic experiments assess variable selection, function recovery, and network compression, while a shallow-network experiment compares posterior summaries with an MCMC reference.

Critical strengths
  • The paper addresses a meaningful problem - identifying relevant variables while controlling the internal complexity of BNN. The overlapping-group construction provides a coherent way to combine predictor(variable)- and hidden-unit-level selection.
  • Approximating the joint distribution of the parameters (slab, activation and thresholds) is a reasonable approach to represent posterior dependencies.
  • The experimental evaluation considers several complementary objectives, including support recovery, function estimation, and structural compression.
Critical weaknesses
  1. Substantial parts of the appendix are missing, which prevents verification of central methodological and experimental claims. Appendices A.1-A.6, B.1, B.2, C.1-C.6, and D.1 contain headings without content. These sections are supposed to provide the construction details, baseline specifications, ablations, etc.
  2. The baseline comparison does not sufficiently establish superiority in predictor selection. The existing methods target different forms of sparsity: LBBNN methods primarily use weight-level inclusion indicators, wheras SS-GL and SS-GHS primarily target hidden-node sparsity. Their relevance to network compression does not automatically establish a comparable predictor-selection procedure. ISLaB-FLOW is more directly relevant to covariate selection, so its inclusion is appropriate. Additional comparisons should include Bayesian methods that explicitly address nonlinear predictor or component selection. Relevant examples include DART, sparse SBART, and Bayeisan-TPNN and could be evaluated using a clearly specified predictor-level summary. These methods would provide additional reference points for function estimation.
  3. The synthetic evaluation is too limited to support the scope of the claims, and Table 2 exposes substantial failures in the intended setting. Although four settings are reported, the low-SNR and high-dimensional settings reuse the baseline regression function; only two distinct regression functions are considered. I would like to see comparisons across at least three qualitatively different data-generating functions, including additive effects, interactions without strong marginal effects, and more heterogeneous nonlinear effects. The existing sensitivity results also require greater emphasis. In the low-SNR and high-dimensional settings, R2R^2 fails (values are almost zero) and TPR decreases to 0.190 and 0.300, respectively. These failures are particularly relevant because high-dimensional sparse predictor settings are a central motivation of the paper. Baseline comparisons and sensitivity analyses across these regimes are needed to determine when the proposed method is useful and whether its collapse can be addressed through principled prior or optimization choices.
  4. The reported selection metrics and prediction summaries do not fully substantiate the interpretation of successful sparse recovery. With 10 active predictors among 200 candidates, excluding every predictor already yields an accuracy of 0.95, close to the reported baseline accuracy of 0.956. Moreover, FPR (=0.043) corresponds to approximately (190\times0.043=8.17) false positives per replicate on average, alongside approximately (10\times0.935=9.35) true positives. Thus, the relatively low FPR does not by itself establish that the selected set contains few irrelevant variables. Precision, false discovery proportion, F1, selected-set size, and exact-support recovery would provide a more informative assessment.
Questions for authors

Please see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.