CauRisk: When to Trust Causal Discovery Foundation Models under Distribution Shift

Causal discovery foundation models can predict causal graphs for new datasets, but their confidence may not reflect graph accuracy when the underlying causal mechanisms change. We introduce CauRisk, a post-hoc framework that estimates graph-level prediction error under shifts in structural causal models (SCMs), without access to target ground-truth graphs at test time. Trained on synthetic source environments with known graphs, the risk estimator combines predictive uncertainty, stability across resampled datasets, and compatibility between the predicted graph and statistical patterns in the observed data. These diagnostics provide evidence about structural error without assuming that observational data can resolve non-identifiable causal directions. The discovery model remains frozen. CauRisk uses the estimated risk to decide which graph predictions to retain and which to defer. We specify an evaluation protocol that holds out entire causal mechanism and graph families, reserving target graphs exclusively for evaluation. The protocol compares CauRisk with uncertainty-only and stability-only estimators on failure detection, risk calibration, and risk–coverage trade-offs. It tests whether combining these diagnostics helps predict graph error beyond the source environments and supports decisions about when to rely on a predicted causal graph.