- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CLAIM-FUSE for causal discovery from observational data and uncertain external causal knowledge. Directed-edge, adjacency, ancestry, and ordering claims are represented as binary predicates on candidate graphs. Answers and their confidence scores are used to update graph weights. Sequential Monte Carlo and graph edits explore additional candidates, while expected information gain determines which variables and claim type to query next. The paper establishes conditions under which finitely many claim updates preserve consistency of the exact target distribution, and shows that expanding the query pool can increase the maximum one-step expected information gain. Experiments cover synthetic data, a neuropathic-pain subnetwork, and ALARM, evaluating structural recovery, claim-type selection, token usage, and runtime. Improvements are clearer on the pain subnetwork than on ALARM, and combining claim types does not always outperform using a single family.
Critical strengths
The method chooses not only which variables to ask about, but also whether to request a direct or more global relation. This expands the information available under a limited elicitation budget. Updating the same graph distribution with these different answers gives the framework a coherent structure.
Critical weaknesses
The main distinguishing feature of this paper is active selection across claim types, but the baseline comparisons do not clearly separate its contribution from those of particle inference and graph editing. The claim-family ablation provides some evidence, although the union is not consistently best on ALARM. Moreover, in Table 2, the stated-confidence variant improves ED over GES only from 22.65 to 22.63 and from 11.85 to 11.55. These differences are small relative to the reported standard errors; paired differences and their uncertainty are needed to assess whether the gains are consistent. The results therefore support effectiveness in some settings, but provide more limited evidence for a general advantage and for the specific contribution of claim-type selection.
Questions for authors
-
Ordering is presented as the fourth claim type, but the query-pool counts in Appendices E.2 and F.2 include only directed-edge, adjacency, and ancestry queries. Were ordering claims queried in the experiments? If so, how were they counted, and how is their binary truth value defined for a DAG that admits multiple topological orders?
-
Could you report per-seed paired differences, with uncertainty estimates, between the all-claim pool and the local and ancestral pools? Is the gain from the central design consistent, particularly on ALARM? If random-query results using the same inference pipeline are available, please also provide them to help distinguish the effect of active selection from that of inference itself.
3 comments
"Q2. Could you report per-seed paired differences, with uncertainty estimates, between the all-claim pool and the local and ancestral pools? Is the gain from the central design consistent, particularly on ALARM? If random-query results using the same inference pipeline are available, please also provide them […]"
We reran the local and ancestral pools, and a random-query arm (uniform selection over the same four-type pool), with everything else identical to Tables 1–2. With type-specific reliability, the all-claim pool beats both single pools and random selection in all 12 settings (table). The ALARM reversal you noted appears only with a single global reliability, because the LLM is less accurate on ancestral and ordering questions; reliability-aware selection removes it.
| Type-specific selection, final budget | All − local | All − ancestral | EIG − random (same pool) | |
|---|---|---|---|---|
| Neuropathic, Qwen, stated | 200 | −0.90 [−1.54, −0.28] | −3.54 [−4.39, −2.61] | −6.43 [−7.20, −5.59] |
| 1000 | −1.11 [−2.00, −0.37] | −3.24 [−4.48, −2.05] | −6.70 [−7.42, −5.90] | |
| Neuropathic, Qwen, consistency | 200 | −1.14 [−1.85, −0.44] | −1.94 [−2.96, −0.83] | −5.58 [−6.56, −4.49] |
| 1000 | −1.66 [−2.96, −0.61] | −1.63 [−3.01, −0.43] | −6.21 [−7.38, −5.15] | |
| Neuropathic, Terra, stated | 200 | −1.14 [−1.97, −0.38] | −0.45 [−1.66, 0.70] | −4.86 [−6.05, −3.64] |
| 1000 | −1.68 [−2.14, −1.24] | −0.45 [−1.71, 0.75] | −5.30 [−6.28, −4.33] | |
| Neuropathic, Terra, consistency | 200 | −0.37 [−1.46, 0.77] | −0.62 [−1.78, 0.34] | −4.63 [−6.03, −3.29] |
| 1000 | −0.83 [−1.44, −0.23] | −0.59 [−2.01, 0.92] | −5.27 [−6.45, −4.07] | |
| ALARM, stated | 200 | −0.79 [−1.62, −0.01] | −2.15 [−3.07, −1.15] | −3.42 [−4.58, −2.29] |
| 1000 | −0.34 [−0.68, 0.03] | −1.17 [−1.78, −0.57] | −2.00 [−2.74, −1.34] | |
| ALARM, consistency | 200 | −0.87 [−1.49, −0.32] | −1.65 [−2.77, −0.62] | −2.51 [−3.95, −1.18] |
| 1000 | −0.48 [−0.87, −0.08] | −1.01 [−1.54, −0.49] | −1.94 [−2.77, −1.13] |
Mean ED difference [95% paired bootstrap interval]; bold: interval excludes zero.
We hope these paired analyses address your concerns and would be grateful if you would consider raising your score.
Dear Reviewer uS1d,
Thank you again for your thoughtful review. We would like to briefly follow up on our responses and revised manuscript.
To further isolate the contribution of heterogeneous querying, we added Table 3 and Appendix A.5, directly comparing the four-type policy against eight restricted claim pools under matched experimental conditions. The four-type policy is never significantly worse than a restricted pool and achieves significant improvements in 28 ED and 20 orientation F1 comparisons.
These results complement the paired local-only, ancestral-only, and random-query comparisons provided in our earlier response. It directly addresses your concern about separating the benefits of active query selection from those of graph inference.
We would appreciate hearing whether these additional results resolve your remaining concerns. Thank you for considering our revisions.
Dear Reviewer uS1d,
Thank you for recognizing that choosing among local and global relation expands the information available under a limited budget, and that a single graph distribution gives the framework a unified structure. We respond to each concern below. Line numbers refer to the revised PDF.
All CLAIM-FUSE rows of the revised Tables 1–2 share the same starting particles, SMC inference, Metropolis–Hastings edits, cached answers and budget, and differ only in the elicitation design (lines 1287–1293). This lets us isolate each component. Replacing the single global reliability (hetero.-G) with type-specific reliabilities (hetero.-T) changes only how queries are selected, and lowers ED in 11 of 12 settings. CaPE starts from the same particles with the same number of queries but its own likelihood and selection, and is worse in all six settings below, with every interval excluding zero. The GES numbers you quote are from the original submission. In the revised Table 2, CLAIM-FUSE reaches 18.11 and 9.33, by using the newly added type-specific reliability model. Relative to each seed's own starting particles, the 50 LLM queries lower ED by −4.48 [−6.45, −2.78] at n=200 and −2.14 [−2.93, −1.44] at n=1000.
Mean ED difference [95% paired bootstrap interval]; bold: interval excludes zero.
Yes. The heterogeneous rows of Tables 1–2 draw from four-type pools: 1,330 questions on neuropathic pain and 4,662 on ALARM, each including 380 or 1,332 ordering questions. We apologize for the confusion: the counts in Appendices E.2/F.2 describe the three-family pools of Figure 2, and the revised appendices now also state the four-type pools. Ordering is permitted precedence: Xi≺Xj holds in G iff Xi precedes Xj in at least one valid topological order. By Lemma 1 (Appendix B.1), this is equivalent to Xj⇝Xi, so the predicate is well defined on every DAG however many orders it admits. Each ordering question is one attempted query; abstentions count toward the budget, and asking an ordering question removes its complementary reverse-ancestry question from the pool.