acceptodds
Under review as a conference paper at ICLR 2027

SVI-DAG: A Structured Variational Inference Approach to Bayesian Causal Discovery

Abstract

Bayesian causal discovery seeks to determine the posterior distribution of causal theories, which are interpreted as directed acyclic graphs (DAGs) that explain the observed data. The resulting posterior allows systematic reasoning regarding epistemic uncertainty within these theories. Nonetheless, finding such graphs is difficult due to identifiability problems and limited observational data. Furthermore, precisely approximating posterior over graphs is challenging given vast range of potential DAGs. Recent Bayesian approaches have addressed some of these challenges, yet they remain limited as they fail to encode dependencies between edges, and lack principled ways to incorporate domain knowledge as inductive biases during the search process. To overcome these limitations, we propose SVI-DAG, a structured variational inference approach to Bayesian causal discovery using observational data and prior beliefs that uses normalizing flows to model dependencies between edges, supporting expressive and multimodal posterior learning over DAGs. To mitigate mode seeking behaviour in evidence lower bound optimization and promote mode coverage, we use stein variational gradient descent to update the node potentials using a kernel in acyclicity space. We evaluate SVI-DAG against 5 state-of-the-art Bayesian DAG learning methods and demonstrate competitive performance in terms of both accuracy and uncertainty quantification.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Variational Inference, Normalizing Flows, Stein Variational Gradient Descent, Causal Discovery

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EwdMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SVI-DAG aims to estimate not just one causal graph, but a range of plausible graphs and their probabilities. It uses a flexible model to represent connections that tend to occur together, a way to include prior knowledge about likely connections, and a particle-based method to explore different node orderings. The authors test it on simulated data and the Sachs biological dataset.

Critical strengths
  1. Estimating several plausible graphs is worthwhile when the data do not determine one clear graph. The paper gives a coherent method and tests what happens when major components are removed.

  2. The evaluation looks at both graph recovery and the quality of the reported probabilities. On Sachs, SVI-DAG has the best expected F1 score when comparing fully directed graphs (Table 2), meaning it achieves the best balance of recovered and spurious connections under that measure.

Critical weaknesses
  1. The extra complexity has no clear overall payoff on the real-data benchmark. On Sachs, SVI-DAG has the worst Brier score among six methods for fully directed graphs; this score measures how well its predicted edge probabilities match the reference graph. Its expected structural Hamming distance, the number of edge changes needed to match that graph, ranks 4th of six for both directed and equivalence-class comparisons. It is also the third slowest method (Table 1). Its F1 result is a real strength, but the paper needs to explain when that gain matters more than the weaker probability estimates, larger graph error, and run time.

  2. Removing a proposed component sometimes helps. In Table 4, removing the particle-based SVGD step improves structural Hamming distance from 69.3 to 62.0 and F1 from 26.9% to 34.8%, although other measures become worse. Removing the prior knowledge causes a much larger decline than removing the other components. The authors should explain these trade-offs and compare methods using the same prior information before attributing the gains to the new graph model.

Questions for authors
  1. For the intended use, which matters most: recovering graph connections, ranking likely connections, or assigning accurate probabilities? How should readers weigh the Sachs results across these measures?
  2. Can the authors compare methods given the same prior information and explain why removing SVGD improves graph distance and F1 in Table 4?
Reviewer_x1rkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides a structured variational inference approach to Bayesian Causal Discovery. In early papers for that topic edges were considered to be independent which is a simplification but allowed tractable calculations. More recently normalizising flows have been popular for causality due to their appealing properties and this papere here combines these approaches for Bayesian causal discovery.

Critical strengths

Technical the approach combines a couple of interesting details and seems to be correctly dervied. The hard part is however not deriving a new causal bayesian structure discovery algorithm but really showing that it is useful and does better. That is unfortunately lacking.

Critical weaknesses

The evaluation of Bayesian Causality Approaches are difficult see [1]

[1] Mamaghan, A. M. K., Tigas, P., Johansson, K. H., Gal, Y., Annadani, Y., & Bauer, S. (2024). Challenges and considerations in the evaluation of bayesian causal discovery. ICML

Many existing metrics fail to be suitable for evaluation. This is a core issue for the propose approach which claims better uncertainty quantification but lacks the robust and nuanced evaluation to really make that claim. While it all reads nice and technically sound it is not clear at all if the approach actually provides any benefit. Why would e.g. better scores of SVI Dag lead to better posteriors and better downstream decisisons? What about the identifiability problems in causality? Sachs has inherent problems as a dataset for causality usage and is just not a good dataset for it [2]. There are more papers which really show sachs was used as a standard but is simply not good for it, But in partcular the empirical claims are too strong and reliant on individual metrics which are just not good indicators. We know that the reported metrics are flawed and yet they get continuously reported for individual papers which then do not hold up in practical use cases.

[2]Mooij, Joris M., Sara Magliacane, and Tom Claassen. "Joint causal inference from multiple contexts." Journal of machine learning research 21.99 (2020): 1-108.

Questions for authors

Please fix the evaluations with respect to the true posterior

DIBS is just a copy of [3] Ke, Nan Rosemary, et al. "Learning to induce causal structure." arXiv preprint arXiv:2204.04875 (2022). They even write that in the paper as concurrent work.

The approach is for observational data but the reason why many works focus on active learning is that this makes the evaluation more robust. I.e. is it more efficient within an active learning loop. The same goes for any other downstream task like experimental design or treatment effect estimation or missing data.

Can you show a clear scalability plot of the approach.

Reviewer_a5YYOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SVI-DAG, a structured varitional inference framework for bayesian causal discovery. The basic idea is to combine (a) conditional normalizing flow to model dependencies among edge logits; (b) a hierarchical beta-bernouilli prior to incorporate domain knwoledge and (c) Stein variational gradient descent to encourage coverage of DAG posterior modes.

The proposed idea is interesting, and the experiments show competitive performance as compared to other baselines in the literature.

Critical strengths
  • The paper address an important challenge in bayesian causal discovery: representing uncertainity over structures rather than producing a single estimated DAG. The conditional dependence between node potentials and edge logits is a sensible modelling choice.

  • The seperation between hard DAG sampling and soft backward relaxations is also clearly motivated.

  • The experimental setup includes both linear and nonlinear synthetic settings, larger graphs, a real-world benchmark, and good set of ablations. The results support the conclusion that SVI-DAG is competitive, although they do not demonstrate better results as compared to baselines across all structural metrics (which is ok).

Critical weaknesses
  • Improved multimodal posterior inference: The main motivation for SVI-DAG is to capture dependencies between edges and improve coverage of multiple DAG posterior modes. Currently, the experimental evaluation primarily relies on marginal edge-level and structural accuracy metrics, which do not directly establish the quality of the joint posterior over DAGs. Currently, Table 4 introduce MEC coverage, the full method achieves only (0.06 \pm 0.02), compared with (0.05 \pm 0.04) without SVGD, providing limited evidence for mode coverage. Also, removing SVGD improves expected SHD and expected F1. A comparison against an exact posterior on small graphs would more directly establish whether the proposed method accurately captures multiple posterior modes and their probabilities.

  • Robustness of the hierarchical prior: Beta–Bernoulli prior is presented as a way to incorporate uncertain domain knowledge and improving robustness to misspecified prior beliefs. Since each latent edge probability governs a single Bernoulli variable, marginalising it yields an ordinary Bernoulli prior with an adjusted edge probability, ((\nu_{ij}p_{ij}+1)/(\nu_{ij}+2)). Thus, at the level of the induced structural prior, the hierarchy is equivalent to parameterised Bernoulli model. While the continuous latent representation is useful for the proposed variational formulation, it is unclear whether the hierarchy provides additional robustness beyond this reparameterisation. It seems to me that Figure 4 demonstrates substantial sensitivity to incorrect prior beliefs. The absence of a comparison against a Bernoulli prior with matched marginal probabilities makes it difficult to establish the claimed robustness contribution.

  • Ablations: The ablation results indicate that the prior (conditioned on domain information) improves performance than either the normalizing flow or SVGD. However, the construction of the edge-specific nominal prior probabilities in the main benchmarks is not sufficiently specified, making it difficult to assess how much of the reported improvement comes from prior information rather than the inference method. Additionally, the SVGD ablation replaces the particle-based approximation with a Gaussian variational and hence it changes the inference mechanism and the representation of the posterior over node potentials.

Questions for authors

See Critical Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.