- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
In this paper, the authors introduced a post-hoc framework, namely CauRisk, that estimates graph-level prediction errors for causal discovery foundation models (Arrow, AVICI). Combining four diagnostics, uncertainty, stability, compatibility, and metadata, through mechanism-wise (worst-group) aggregation into a single risk score, CauRisk estimator calibrates and thresholds the result to decide which graph predictions to retain or defer. Three theoretical propositions are introduced to bound the results: selective ambiguity floor, a finite-grid validation bound, and a shift amplification after deferral. The framework is trained and evaluated on synthetic SCMs under three evaluation conditions: same-family, mechanism-shifts, and topology-shifts. The paper reports that across all three test conditions, CauRisk’s failure-detection ranking ability transfers well, unlike its numerical calibration and fixed acceptance threshold, especially under topology shifts, where the retained graphs have higher average error than the declared risk budget.
Critical strengths
1- The paper addresses an important problem: when to trust causal discovery foundation models’ predictions when deployed and evaluated across new datasets. 2- The authors supported their framework with theoretical analysis rather than relying solely on empirical results, and the framing effectively supported the motivation. 3- The experimental setup is designed to hold out mechanisms and graph-topology families from fitting and calibration, rather than reusing the same training setup, and uses these held-out families to assess transfer under distribution shift. 4- The paper’s main claim, that failure-detection ranking ability transfers under shifts while the numerical calibration and the fixed-threshold acceptance do not, is supported by the results and findings reported in the paper.
Critical weaknesses
1- The paper uses only synthetic data, and the mechanism and topology shifts evaluated are limited. All tasks use the same fixed parameters [d=10, n=500] and the same noise generation process, noise family, and scale for both training (discovery rows) and evaluation (audit rows), with only changes in the functional form or the edge-selection form. With this much similarity in data generation, the reported performance degradation is likely a lower bound on what a real deployment shift in datasets would look like. Also, generating data without latent confounding for a specific data type (continuous) is not always the case in real datasets, where data types can differ, and there will (in most cases) be latent confounding.
2- The paper includes several evaluation tables in the appendix and uses them in the analysis (Section 6: Results) without properly referencing them, such as in Q3 and Q4, leaving the reader to infer where each reported figure comes from. Also, it does not clarify how these tables connect to Table 1 in the main text. Each table presents a different evaluation process, such as ranking quality using AURC (Table 1), numerical calibration using MAE (Table 4), and the actual retain and defer decisions (Tables 8 and 13). It would help if the paper more clearly identified and distinguished these evaluation processes before presenting the numbers. They are conceptually distinct but presented consecutively, with no explicit distinction separating them. This made it harder for me, as a reader less familiar with this field, to follow what each table’s results were measuring. Additionally, the main algorithm of this method, which could provide a clearer description of the process, is placed in the last appendix of the paper (Appendix F) rather than in the main body.
3- As per my understanding, the compatibility scope is limited: it only tests conditional independencies implied by the predicted graph’s local Markov property, which does not reflect full faithfulness. Because of this, it cannot test whether a graph’s existing edge between adjacent variables should exist. It is excluded from the tested independence queries by the Markov property itself. So an extra added edge in this case cannot be directly identified by this diagnostic. I would appreciate further clarification from the authors on this point.
4- The relationship between the paper’s proposition theory and the deployed decision rule remains unclear to me. Propositions 1 and 2 seem to support earlier steps in the pipeline, while the final defer/retain decision is chosen by an empirical method that the paper itself explains is different from risk-control methods (Section 3.3). It is not clear to me from the text whether the propositions provide only motivation or an actual guarantee for the deployed decision-rule mechanism. I would appreciate further clarification on this point.
5- Minor Comments: Section 6: Results reports results for Q1- Q4 with minimal explanation/analysis, and the results are spread out across 14 tables that are not connected or properly summarised. Figure 2 in the paper has a minor misplacement of the y-axis title.
Questions for authors
1- Could the authors validate their CauRisk framework against a real-world dataset, even on a small scale, to check whether the learned diagnostic can track graph errors on a real dataset, given that all current evaluation is on synthetic SCMs?
2- Since only a small set of variables is used across all evaluations (d=10), could the authors extend their analysis to a larger scale to check whether their diagnostic patterns still hold? Or is the choice of (d=10) a backbone constraint rather than an evaluation choice?
CauRisk Discussion Summary
Dear Area Chair and Reviewers,
Thank you for your reviews and constructive feedback. CauRisk addresses a practical question: after a discovery model returns a graph, how can we assess its structural error when the true graph is unknown? We learn how observable diagnostics relate to graph error on source tasks with known graphs, then use those diagnostics to score predictions from frozen discovery models on new datasets.
A risk score can help us decide which graphs to inspect first, even when a threshold chosen on source data misses its risk budget under shift. CauRisk achieves lower AURC than permutation-based graph falsification in all six Arrow and AVICI settings. The paired 95% bootstrap intervals favor CauRisk in each setting, including after averaging over exact-score ties (Table 4 and Appendix B.9).
In our offline evaluation, flagging about one in five predictions by CauRisk score identifies 23.5%–35.2% of high-error graphs, compared with 20.0%–20.4% under random review (Appendix I). The all-feature tree baseline finds the same share of high-error graphs and achieves lower AURC under Arrow topology shift (Table 1). The review experiment thus gives a concrete use for both learned scores.
The distinction matters for acceptance decisions. Under topology shift, the source rule set at a budget of 0.5 retains graphs with mean losses of 0.6111 for Arrow and 0.5500 for AVICI (Table 2). Appendix B.8 compares four source-only rules, and Proposition 3 shows why target coverage matters: the same amount of joint distribution shift contributes a larger term to the retained-risk bound when fewer target predictions are retained.
The extended tests cover graphs with up to 50 variables, different sample sizes and noise conditions, and latent-root perturbations. The causalAssembly experiments use industrial station graphs with generated observations (Appendix H.2). The Sachs case uses real cell measurements from one biological system and compares the predicted graphs with a literature reference (Appendix H.3).
Together, these results demonstrate the value of learning graph-level structural-error scores while distinguishing useful error ranking from the stronger requirements of reliable numerical acceptance under distribution shift.
Best regards, The Authors