acceptodds
Under review as a conference paper at ICLR 2027

CLAIM-FUSE: Causal Discovery with Heterogeneous and Uncertain Knowledge

Abstract

Causal discovery aims to recover cause-and-effect relationships from observational data, and large language models (LLMs) can help generate textual causal statements when data alone is not sufficient to identify the correct causal relationship. However, existing LLM-assisted methods either incur high computational costs by querying the causal relationship for every variable pair, or ask repeated questions despite some question types being more informative and more reliably answered than others. We therefore propose CLAIM-FUSE, which chooses which variables and which type of causal claim to query (directed edge, adjacency, ancestry, or order) at each step, minimizing the uncertainty within a fixed budget. Our empirical results show that CLAIM-FUSE achieves the lowest structural error among LLM-assisted methods, up to 19% below the strongest baseline, using orders of magnitude fewer tokens than exhaustive methods. Choosing among all claim types also improves recovery over querying only local or only ancestral claims, though it can degrade when the LLM is less reliable on the chosen type. Our work lays the foundation for knowledge elicitation that adapts not only the content, but also the type of the question.

Primary area:
causal reasoning
Keywords:
Causal discovery, heterogeneous knowledge integration, knowledge elicitation, large language models (LLMs)

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uS1dOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CLAIM-FUSE for causal discovery from observational data and uncertain external causal knowledge. Directed-edge, adjacency, ancestry, and ordering claims are represented as binary predicates on candidate graphs. Answers and their confidence scores are used to update graph weights. Sequential Monte Carlo and graph edits explore additional candidates, while expected information gain determines which variables and claim type to query next. The paper establishes conditions under which finitely many claim updates preserve consistency of the exact target distribution, and shows that expanding the query pool can increase the maximum one-step expected information gain. Experiments cover synthetic data, a neuropathic-pain subnetwork, and ALARM, evaluating structural recovery, claim-type selection, token usage, and runtime. Improvements are clearer on the pain subnetwork than on ALARM, and combining claim types does not always outperform using a single family.

Critical strengths

The method chooses not only which variables to ask about, but also whether to request a direct or more global relation. This expands the information available under a limited elicitation budget. Updating the same graph distribution with these different answers gives the framework a coherent structure.

Critical weaknesses

The main distinguishing feature of this paper is active selection across claim types, but the baseline comparisons do not clearly separate its contribution from those of particle inference and graph editing. The claim-family ablation provides some evidence, although the union is not consistently best on ALARM. Moreover, in Table 2, the stated-confidence variant improves ED over GES only from 22.65 to 22.63 and from 11.85 to 11.55. These differences are small relative to the reported standard errors; paired differences and their uncertainty are needed to assess whether the gains are consistent. The results therefore support effectiveness in some settings, but provide more limited evidence for a general advantage and for the specific contribution of claim-type selection.

Questions for authors
  1. Ordering is presented as the fourth claim type, but the query-pool counts in Appendices E.2 and F.2 include only directed-edge, adjacency, and ancestry queries. Were ordering claims queried in the experiments? If so, how were they counted, and how is their binary truth value defined for a DAG that admits multiple topological orders?

  2. Could you report per-seed paired differences, with uncertainty estimates, between the all-claim pool and the local and ancestral pools? Is the gain from the central design consistent, particularly on ALARM? If random-query results using the same inference pipeline are available, please also provide them to help distinguish the effect of active selection from that of inference itself.

3 comments

Dear Reviewer uS1d,

Thank you for recognizing that choosing among local and global relation expands the information available under a limited budget, and that a single graph distribution gives the framework a unified structure. We respond to each concern below. Line numbers refer to the revised PDF.


"[…] the baseline comparisons do not clearly separate its contribution from those of particle inference and graph editing. […] in Table 2, the stated-confidence variant improves ED over GES only from 22.65 to 22.63 and from 11.85 to 11.55. […] paired differences and their uncertainty are needed to assess whether the gains are consistent."

All CLAIM-FUSE rows of the revised Tables 1–2 share the same starting particles, SMC inference, Metropolis–Hastings edits, cached answers and budget, and differ only in the elicitation design (lines 1287–1293). This lets us isolate each component. Replacing the single global reliability (hetero.-G) with type-specific reliabilities (hetero.-T) changes only how queries are selected, and lowers ED in 11 of 12 settings. CaPE starts from the same particles with the same number of queries but its own likelihood and selection, and is worse in all six settings below, with every interval excluding zero. The GES numbers you quote are from the original submission. In the revised Table 2, CLAIM-FUSE reaches 18.11 and 9.33, by using the newly added type-specific reliability model. Relative to each seed's own starting particles, the 50 LLM queries lower ED by −4.48 [−6.45, −2.78] at n=200n=200 and −2.14 [−2.93, −1.44] at n=1000n=1000.

ED difference (stated, final budget)Selection rule: hetero.-T − hetero.-GSame start, other method: hetero.-T − CaPE
Neuropathic, Terra, n=200n=200+0.12 [−1.31, 1.43]−3.16 [−4.80, −1.66]
Neuropathic, Terra, n=1000n=1000−1.01 [−2.91, 0.67]−4.77 [−6.17, −3.50]
Neuropathic, Qwen, n=200n=200−1.15 [−2.13, 0.02]−9.55 [−12.66, −6.69]
Neuropathic, Qwen, n=1000n=1000−2.09 [−3.74, −0.63]−7.05 [−9.09, −5.20]
ALARM, n=200n=200−1.72 [−2.36, −1.05]−6.42 [−8.68, −4.41]
ALARM, n=1000n=1000−1.43 [−2.66, −0.42]−5.91 [−7.57, −4.39]

Mean ED difference [95% paired bootstrap interval]; bold: interval excludes zero.


"Q1. […] Were ordering claims queried in the experiments? If so, how were they counted, and how is their binary truth value defined for a DAG that admits multiple topological orders?"

Yes. The heterogeneous rows of Tables 1–2 draw from four-type pools: 1,330 questions on neuropathic pain and 4,662 on ALARM, each including 380 or 1,332 ordering questions. We apologize for the confusion: the counts in Appendices E.2/F.2 describe the three-family pools of Figure 2, and the revised appendices now also state the four-type pools. Ordering is permitted precedence: Xi≺XjX_i \prec X_j holds in GG iff XiX_i precedes XjX_j in at least one valid topological order. By Lemma 1 (Appendix B.1), this is equivalent to Xj⇝̸XiX_j \not\rightsquigarrow X_i, so the predicate is well defined on every DAG however many orders it admits. Each ordering question is one attempted query; abstentions count toward the budget, and asking an ordering question removes its complementary reverse-ancestry question from the pool.


"Q2. Could you report per-seed paired differences, with uncertainty estimates, between the all-claim pool and the local and ancestral pools? Is the gain from the central design consistent, particularly on ALARM? If random-query results using the same inference pipeline are available, please also provide them […]"

We reran the local and ancestral pools, and a random-query arm (uniform selection over the same four-type pool), with everything else identical to Tables 1–2. With type-specific reliability, the all-claim pool beats both single pools and random selection in all 12 settings (table). The ALARM reversal you noted appears only with a single global reliability, because the LLM is less accurate on ancestral and ordering questions; reliability-aware selection removes it.

Type-specific selection, final budgetnnAll − localAll − ancestralEIG − random (same pool)
Neuropathic, Qwen, stated200−0.90 [−1.54, −0.28]−3.54 [−4.39, −2.61]−6.43 [−7.20, −5.59]
1000−1.11 [−2.00, −0.37]−3.24 [−4.48, −2.05]−6.70 [−7.42, −5.90]
Neuropathic, Qwen, consistency200−1.14 [−1.85, −0.44]−1.94 [−2.96, −0.83]−5.58 [−6.56, −4.49]
1000−1.66 [−2.96, −0.61]−1.63 [−3.01, −0.43]−6.21 [−7.38, −5.15]
Neuropathic, Terra, stated200−1.14 [−1.97, −0.38]−0.45 [−1.66, 0.70]−4.86 [−6.05, −3.64]
1000−1.68 [−2.14, −1.24]−0.45 [−1.71, 0.75]−5.30 [−6.28, −4.33]
Neuropathic, Terra, consistency200−0.37 [−1.46, 0.77]−0.62 [−1.78, 0.34]−4.63 [−6.03, −3.29]
1000−0.83 [−1.44, −0.23]−0.59 [−2.01, 0.92]−5.27 [−6.45, −4.07]
ALARM, stated200−0.79 [−1.62, −0.01]−2.15 [−3.07, −1.15]−3.42 [−4.58, −2.29]
1000−0.34 [−0.68, 0.03]−1.17 [−1.78, −0.57]−2.00 [−2.74, −1.34]
ALARM, consistency200−0.87 [−1.49, −0.32]−1.65 [−2.77, −0.62]−2.51 [−3.95, −1.18]
1000−0.48 [−0.87, −0.08]−1.01 [−1.54, −0.49]−1.94 [−2.77, −1.13]

Mean ED difference [95% paired bootstrap interval]; bold: interval excludes zero.

We hope these paired analyses address your concerns and would be grateful if you would consider raising your score.

Dear Reviewer uS1d,

Thank you again for your thoughtful review. We would like to briefly follow up on our responses and revised manuscript.

To further isolate the contribution of heterogeneous querying, we added Table 3 and Appendix A.5, directly comparing the four-type policy against eight restricted claim pools under matched experimental conditions. The four-type policy is never significantly worse than a restricted pool and achieves significant improvements in 28 ED and 20 orientation F1 comparisons.

These results complement the paired local-only, ancestral-only, and random-query comparisons provided in our earlier response. It directly addresses your concern about separating the benefits of active query selection from those of graph inference.

We would appreciate hearing whether these additional results resolve your remaining concerns. Thank you for considering our revisions.

Reviewer_rTYEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLAIM-FUSE represents four types of causal claims as binary predicates over candidate directed acyclic graphs (DAGs). It maintains a weighted particle approximation using sequential Monte Carlo (SMC) and selects variable pairs and claim types based on expected information gain (EIG). The method is evaluated on a neuropathic pain subnetwork, the ALARM network, and synthetic structural causal models (SCMs) generated by GPT-5.6 Terra and Qwen3.5-9B.

Critical strengths
  1. Since all claim types update a shared particle distribution, their EIG values can be directly compared within a common framework.

  2. The provided prompt and response logs are useful for reproducibility, and the ablation studies, including the analysis of miscalibration, are comprehensive.

  3. The proposed algorithm appears relatively straightforward to apply.

  4. The research motivation is clearly presented, and the figures are generally well organized and easy to interpret.

Critical weaknesses

1. Insufficient baseline comparisons: Although the paper emphasizes its distinction from existing active-elicitation methods [1–4] that select variable pairs for querying, none of these methods are included as baselines in Tables 1 and 2.

Furthermore, comparisons with data-driven causal discovery methods are limited to PC, GES, and NOTEARS, with NOTEARS evaluated only on ALARM and the synthetic datasets. Several relevant approaches are missing, including ordering-based methods such as BOSS and GRaSP, continuous-optimization methods such as DAGMA, and methods that produce distributions over causal graphs. The latter would be particularly relevant given the paper's use of distributional evaluation metrics such as ED.

A broader comparison would help establish whether the proposed framework offers advantages over existing causal discovery and active-elicitation approaches.

2. Absence of an LLM-only baseline: The experiments compare data-only methods with combinations of data-driven methods and LLMs, but do not include an LLM-only setting.

For example, an exhaustive pairwise querying baseline, such as the approach in [5], would provide a useful comparison. This would help assess whether the observed improvements arise from the proposed SMC/EIG mechanism or primarily from causal knowledge already available to the LLM.

Such a comparison could also help investigate potential benchmark memorization, although it would not by itself rule out data contamination.

3. Limited performance improvements: On the ALARM dataset, 100 LLM queries yield only marginal improvements in edit distance (ED). At (n=200), ED decreases from 22.65 for GES to 22.63 for CLAIM-FUSE, while at (n=1000), it decreases from 11.85 to 11.55.

Moreover, the variant incorporating consistency constraints performs worse than GES at (n=200), with ED values of 24.65 and 22.65, respectively.

More substantial improvements are reported on the neuropathic pain dataset, where ED decreases from 20.05 to 15.23 and from 18.55 to 12.59. However, these results are obtained on a single 20-node subnetwork.

The current evidence therefore provides limited support for consistent improvements across different graph structures and scales.

4. Insufficient validation of the central contribution: selecting among four claim types The central contribution of CLAIM-FUSE is its ability to select among four types of causal claims. However, Table 3 reports only two categories, Local and Ancestral, while Figure 2 compares only Local, Ancestral, and All.

The paper does not provide a detailed breakdown of how frequently each of the four claim types is selected or how accurately the LLM answers each type.

This omission makes it difficult to determine whether EIG effectively distinguishes among different claim types, particularly ordering and ancestral relationships, when selecting queries.

Such an analysis is especially important for understanding the performance reversal observed on ALARM in RQ2, where differences in claim-type accuracy appear relevant to the relative performance of the query-selection strategies.

5. Limited control for benchmark memorization and reliance on simulated experts: ALARM and the neuropathic pain simulator are long-established public benchmarks. Even when variable descriptions are masked, their semantic content and hierarchical relationships may still provide clues that allow an LLM to recognize familiar causal structures.

In contrast, the synthetic experiments in Appendix D use a simulated expert rather than an actual LLM. Consequently, these experiments do not establish how CLAIM-FUSE performs with real LLM responses when variables lack meaningful descriptions, as may occur in anonymized datasets.

The authors should clarify the extent to which the method depends on informative variable descriptions and evaluate its robustness under more restrictive information settings.

6. Unvalidated confidence estimates and limitations of the EIG formulation: The confidence parameter (c_k) does not necessarily represent the true probability that an LLM response is correct. Instead, it is estimated from the LLM's self-reported confidence or response consistency across eight samples. Nevertheless, Equation 3.2 appears to treat this estimate as the probability of a correct response.

Although Brier score and expected calibration error (ECE) are defined for the LLM experiments, the corresponding results do not appear to be reported.

A related concern is the information used by the EIG formulation in Proposition 2. The calculation does not have access to the true graph or the actual correctness of future responses. Moreover, it appears to assume a common confidence parameter across claim types rather than accounting for their potentially different accuracies.

Consequently, higher modeled information gain does not necessarily imply that the selected query will receive a more accurate response or lead to better reconstruction of the true causal graph. A query may be highly informative under the assumed response model while providing little useful information when the LLM is systematically incorrect or miscalibrated.

This distinction may help explain why the ancestral strategy underperforms the local strategy on ALARM despite its reported 78% selection rate. However, additional analysis is needed to establish whether differences in response accuracy are responsible for this result.

7. Symmetric error assumption for Yes/No responses: The assumption that the LLM answers correctly with probability (c_k) for either outcome appears to impose symmetric error rates for Yes and No responses.

This assumption deserves further examination. Previous studies [6–8] have reported that LLMs may exhibit a bias toward affirmative responses in causal reasoning tasks. If such a bias is present, the reliability of Yes and No responses may differ substantially.

A more flexible response model could account for this asymmetry by estimating separate error probabilities for the two outcomes, rather than relying on a single confidence parameter.

Without such an analysis, it remains unclear whether the proposed particle updates appropriately reflect the reliability of the LLM's judgments.

Questions for authors

1. Claim-type selection and accuracy: Could the authors confirm whether the query pool includes both ancestral and ordering relationships? Please provide a breakdown of the four claim types, including their proportions in the candidate pool, selection frequencies, and response accuracies.

2. Experimental settings and consistency: Could the authors clarify what Causal Order+PC refers to in Table 1? The random seeds and query budgets also appear to differ between Tables 1 and 2. Could these settings be standardized, with baseline execution times re-evaluated under comparable conditions?

3. Evaluation on larger graphs: If the 20-node neuropathic pain subnetwork was selected because of baseline scalability limitations, could CLAIM-FUSE and the applicable data-only methods also be evaluated on the full 222-node network? Additional experiments on larger bnlearn networks, such as Hepar2, Win95pts, and Andes, would help assess scalability and generalization.

4. Particle count and computational efficiency: The number of particles (J) used in the main experiments does not appear to be specified. Could the authors report (J), the wall-clock runtime excluding LLM latency, memory usage, and scalability results for graphs larger than 37 nodes?

5. Confidence calibration: Could the authors compare stated confidence and consistency-based confidence using coverage, precision, Brier score, and ECE? Please also report how many valid responses were obtained from the 100 query attempts.

6. Intermediate model sizes: The LLMs used in the experiments differ substantially in parameter count. Could the authors include additional models of intermediate sizes to better characterize how performance changes with model scale?

7.Extension to time-series data: How could the proposed framework be extended to time-series causal discovery, where temporal ordering, lagged dependencies, and potentially time-varying causal structures must be considered?

8 comments

Thank you for suggesting additional baselines and related studies. We noticed that references [1–8] are cited in the review, but their bibliographic details do not appear to be included. Could you kindly clarify which papers these refer to, particularly the active-elicitation baselines [1–4]? This would help us ensure that we address your comparisons accurately.

For clarity, references [1]–[4] refer to Bonilla et al. (2026), da Silva et al. (2026), Verma et al. (2026), and Tench and Demeester (2026), respectively, as discussed in the paper’s Introduction. Reference [5] refers to Kıcıman et al.’s “Causal Reasoning and Large Language Models: Opening a New Frontier for Causality.”

[1] Edwin V. Bonilla, He Zhao, and Daniel M. Steinberg. Causal preference elicitation. CoRR, abs/2602.01483, 2026. doi: 10.48550/ARXIV.2602.01483. URL https://doi.org/10.48550/arXiv.2602.01483.

[2] Tiago da Silva, Bruna Bazaluk, Eliezer de Souza da Silva, Ant´ onio G´ ois, Salem Lahlou, Dominik Heider, Samuel Kaski, Diego Mesquita, and Ad` ele Helena Ribeiro. Expert-aided causal discovery of ancestral graphs. Inf. Sci., 756:123816, 2026. doi: 10.1016/J.INS.2026.123816. URL https://doi.org/10.1016/j.ins.2026.123816.

[3] Prakhar Verma, David Arbour, Sunav Choudhary, Harshita Chopra, Arno Solin, and Atanu R. Sinha. Sequential causal discovery with noisy language model priors. Trans. Mach. Learn. Res., 2026, URL https://openreview.net/forum?id=wFs71JzEO7.

[4] Adrick Tench and Thomas Demeester. Dynamic expert-guided model averaging for causal discovery. CoRR, abs/2601.16715, 2026. doi: 10.48550/ARXIV.2601.16715. URL https://doi.org/10.48550/arXiv.2601.16715.

[5] Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality, 2024. URL https://arxiv.org/abs/2305.00050.

[6] Obozov, M., Salygin, E., Losev, P., Alekseev, A., Bushkov, N., & Moiseev, S. The Yes-Bias in LLM Reasoning. In ICLR 2026 Workshop on Logical Reasoning of Large Language Models.

[7] Verma, N., & Landrum, A. R. (2026, April). All Accept, No Reject: Evaluating LLMs as “Peer” Reviewers. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (pp. 1-14).

[8] Huang, H. (2026). The yes-no bias of large language models reflects answer order and wording, not shifts in moral judgment. arXiv preprint arXiv:2607.05552.

I highlighted [1]–[4] because the manuscript explicitly discusses how CLAIM-FUSE differs from these approaches, making them particularly relevant comparisons. Including these methods, or explaining the practical or methodological reasons for their exclusion, would help readers better assess the proposed contribution. Additional relevant baselines are also available. Fair comparisons with both closely related LLM-guided approaches and a broader range of recent data-driven causal discovery methods, under comparable experimental conditions, would provide a clearer picture of the method’s strengths and limitations. I therefore consider broader baseline coverage an important opportunity to strengthen the experimental evaluation.

Dear Reviewer rTYE,

Thank you for your detailed and constructive review, and for finding the shared particle representation, the reproducibility material, the ablations, and the presentation valuable. We respond to each concern below; line numbers refer to the revised PDF.


"1. Insufficient baseline comparisons: […] none of these methods [1–4] are included as baselines in Tables 1 and 2. […] Several relevant approaches are missing, including ordering-based methods such as BOSS and GRaSP, continuous-optimization methods such as DAGMA, and methods that produce distributions over causal graphs."

We agree a broad comparison is important and have substantially extended Tables 1–2 (lines 351–407). We added the closest active-elicitation method, CaPE [1], which also selects variable pairs by EIG over a particle posterior. Since its release has no LLM interface, we added a minimal adapter for its three-way local judgments while keeping its likelihood, selection and inference (lines 1280–1286). We evaluate CaPE both from its released initialization and from CLAIM-FUSE's starting population, which removes initialization as a confounder. We also added BOSS and GRaSP on both datasets, and DAGMA on neuropathic pain (lines 1260–1266); DAGMA has no multinomial loss for categorical ALARM data. At the same number of queries, CLAIM-FUSE has lower mean ED than every added method in every setting, for example:

Neuropathic pain, Qwen3.5-9B, n=1000n=1000ED ↓
BOSS / GRaSP / DAGMA15.80 / 16.55 / 81.70
CaPE–LLM (released / shared init.)50.27 / 19.30
CLAIM-FUSE (all rows)12.25–14.33

"2. Absence of an LLM-only baseline: […] an exhaustive pairwise querying baseline, such as the approach in [5], would provide a useful comparison. This would help assess whether the observed improvements arise from the proposed SMC/EIG mechanism or primarily from causal knowledge already available to the LLM."

We added an exhaustive pairwise LLM-only baseline in the spirit of Kıcıman et al. [5] (lines 1268–1272). It queries every ordered pair (380 on neuropathic pain, 1,332 on ALARM) and converts each answer and its capped confidence into a log-odds edge score. Exact acyclic-subgraph optimization then returns the highest-scoring DAG, with no observations or initialization, so one graph per answer bank applies to both sample sizes. Even when every pair is asked, LLM knowledge alone is far from the true graph. The improvements therefore come from combining LLM answers with data through CLAIM-FUSE's likelihood and active selection, not from the LLM already knowing the graph. This also argues against memorization: on ALARM, the LLM-only graph makes 143–210 structural errors on a 46-edge network.

ED ↓CLAIM-FUSE (range over rows and nn)Exhaustive pairwise (stated / consistency)
Neuropathic, GPT-5.6 Terra13.50–16.5624.00 / 24.00
Neuropathic, Qwen3.5-9B12.25–16.6040.00 / 32.00
ALARM, Qwen3.5-9B9.33–20.85210.00 / 143.00

"3. Limited performance improvements: On the ALARM dataset, 100 LLM queries yield only marginal improvements in edit distance (ED). […] these results are obtained on a single 20-node subnetwork."

The revised experiments use type-specific reliabilities (Model B, Eq. 3.9), a matched 50-query budget on ALARM, and a common start for every ALARM CLAIM-FUSE row, a mixture of GES settings (lines 1299–1301). To isolate the gain from the LLM queries, the table compares each run's ED before and after querying from the same start. On ALARM, CLAIM-FUSE now has the lowest mean ED and highest orientation F1 in every column of Table 2. That includes all data-only methods (GES 22.65 / 11.85, BOSS 22.80 / 13.95, GRaSP 25.00 / 13.25) and the strongest LLM-assisted baselines. Beyond the 20-node clinical network, the evaluation covers the 37-node categorical ALARM network, 50 random 20-node linear-Gaussian SCMs (Appendix D), and a latent-confounding setting (Appendix G).

CLAIM-FUSE (hetero.-T, stated)Start (zero-query ED)After querying
ALARM, n=200n=200 (50 queries)22.59 (1.56)18.11 (1.64) (−20%)
ALARM, n=1000n=1000 (50 queries)11.48 (1.29)9.33 (1.21) (−19%)
Neuropathic, Qwen, n=200n=200 (100 queries)23.67 (0.34)15.45 (0.26) (−35%)
Neuropathic, Qwen, n=1000n=1000 (100 queries)20.45 (0.23)12.25 (0.41) (−40%)

Reviewer_5JnDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a hybrid causal discovery method, called CLAIM-FUSE, based on both numerical data and LLMs’ knowledge. Compared to existing methods, CLAIM-FUSE selects the optimal LLM queries based on entropy reduction contribution iteratively, with an initial causal graph distribution determined by the numerical data. The work evaluates CLAIM-FUSE against both synthetic and empirical benchmarks, and it compares CLAIM-FUSE with several baseline methods, to show the efficacy of CLAIM-FUSE.

Critical strengths
  1. The motivation is reasonable: it might increase the efficiency of LLM-assisted causal discovery methods by treating heterogeneous queries with different priority, though it might need further discussion whether the graph entropy strategy is good or not.
  2. The experimental results demonstrate the efficiency of CLAIM-FUSE.
Critical weaknesses
  1. The contribution remains incremental. The method has potential to improve the efficiency of existing hybrid causal discovery methods, however, the benefit might be limited. The technical contributions, especially Propositions 2 & 3, are limited.
  2. I am not fully convinced by the LLM query selection strategy. CLAIM-FUSE iteratively selects the optimal causal claims that maximally reduce the graph uncertainty. However, this may correspond to unreliable claim selection. For instance, claims related to long causal paths may have lower reliability than the ones of short paths, since the relationship might become weaker as causal path length increases. However, reliability is not considered in determining claim selection.
  3. Some parts of the methodology need further explanation. For instance, how does CLAIM-FUSE initialise the graph distribution from the numerical data? In a causal graph, a causal order may correspond to multiple graphical structures. How does the method handle it?
Questions for authors

See the weakness part.

3 comments

Dear Reviewer 5JnD,

Thank you for recognizing that heterogeneous causal claims can make LLM-assisted causal discovery more efficient. Below we quote each concern and respond to it. All line numbers refer to the revised PDF. Changes made in response to your review are shown in pink. Changes that also address other reviewers are shown in purple.


"1. The contribution remains incremental. […] Propositions 2 & 3, are limited."

LLM-assisted causal discovery faces three challenges that neither single-type nor exhaustive methods address together, our work addresses these challenges:

  1. Knowledge comes in different types. Existing active methods fix the question type (lines 133–141). CLAIM-FUSE represents directed-edge, adjacency, ancestry and ordering claims as binary predicates fk(G)f_k(G) on the same graphs (lines 168–175), under one likelihood with answer-specific confidence (Eq. 3.2–3.3).
  2. Which variable pair and what causal relation is most informative. CLAIM-FUSE ranks all unasked questions of all four types by EIG (Eq. 3.4–3.6).
  3. Reliability differs across types. See Model B under point 2.

Benefits.

  • Accuracy (Tables 1–2). CLAIM-FUSE has the lowest ED and highest orientation F1 in every comparison. On neuropathic pain its ED is 11.7–22.0% below the strongest LLM-assisted baseline, e.g. 12.25 vs. 15.70 (SCP) and 15.80 (BOSS) with Qwen at n=1000n=1000.
  • Cost (Figure 3). It uses 53–1,700× fewer tokens and 20–155× less runtime than exhaustive methods (lines 491–493). Theorem 6 in our revised paper show adding ancestral claims can greatly reduce the number of queries needed to concentrate the posterior on the true causal graph (lines 319-323, Appendix B.4).
  • Heterogeneity (Figure 2). Querying all types gives the lowest ED in most matched conditions.

"2. […] claims related to long causal paths may have lower reliability […] reliability is not considered in determining claim selection."

Thank you. Varying reliability is a central challenge for LLM elicitation.

Experiments.

  1. Table 3. With uniform reliability, Qwen on ALARM sends 74–79% of queries to ancestral claims, though its ancestral accuracy (64–71%) is below its local accuracy (70–81%), consistent with your intuition.
  2. Type-specific reliability (hetero.-T vs. hetero.-G). Reliabilities are estimated per LLM and confidence on four other labelled graphs, never the evaluation graphs (lines 1287–1293). They lower ED in 11 of 12 settings, e.g. ALARM n=1000n=1000 10.76 → 9.33 and neuropathic Qwen n=1000n=1000 14.33 → 12.25.
  3. Robustness (Appendix D.3). Calibrated sources improve on data-only PC+GES; overconfident ones hurt, motivating bounded confidence (Assumption 1).

Revision (Section 3.3). We define question-specific reliability rk∗=P(Yk=fk(G∗)∣qk,K)r_k^*=P(Y_k=f_k(G^*) \mid q_k,\mathcal K) (Eq. 3.7), which may depend on path length but needs the true graph. We use two approximations:

  • Model A (global, Eq. 3.8): EIG picks the claim with posterior probability closest to 1/2 (Appendix A.4).
  • Model B (type-specific, Eq. 3.9): less reliable types are selected less often.

Each answer also reweights graphs by its own confidence (Eq. 3.2–3.3), so unreliable answers move the distribution less. Path-length models plug directly into Eq. 3.7 and 3.9.


"3. How does CLAIM-FUSE initialise the graph distribution […] a causal order may correspond to multiple graphical structures?"

Initialization (lines 263–268; Appendix C.3). We fit PC (six significance levels) and GES (3 × 4 BDeu settings), complete outputs to consistent DAGs, and allocate 384 particles over an equal-mass mixture, each refined by 192 score-only MH moves. Guided MH edits keep other graphs reachable (Eq. 3.10–3.11).

Orders. An ordering claim is never mapped to one structure. Permitted precedence Xi≺XjX_i \prec X_j holds if XiX_i precedes XjX_j in some topological order of GG; by Lemma 1 this equals Xj⇝̸XiX_j \not\rightsquigarrow X_i. One answer reweights all candidates at once, so a single answer can rule out many graphs (lines 73–75).

We hope this addresses your concerns and would be grateful if you would consider raising your score.

Dear Reviewer 5JnD,

Thank you again for your review. We would like to briefly follow up on our revised manuscript.

To address your concerns about the theoretical contribution and query reliability, we added

  1. Theorem 6 (Appendix B.4), establishing a logarithmic-versus-linear query-complexity for certain sparse causal graph families, showing benefit in efficiency of heterogeneous claims;
  2. type-specific reliability-aware EIG (Section 3.3);
  3. the new Figure 2 that demonstrates how answer reliability affects query selection, while Table 3 and Appendix A.5 provide matched comparisons with eight restricted claim pools. Altogether, they show CLAIM-FUSE achieves the best performance and introducing heterogeneous claims can greatly improve the efficiency and accuracy.

We would greatly appreciate your thoughts on whether these address your main concerns. Thank you for your time and consideration.

I appreciate the authors’ answers.

(1) For the third weakness point, I checked the details based on the answer, and it solved my questions.

(2) The second concern still remains unresolved. I understand that the work considers reliability in the pipeline, however, I would clarify that the main objective of this work is different. In essence, the main objective of causal discovery is to accurately recover the causal graph, preferably with better efficiency. The query selection process in this work is mainly driven by graph noise reduction. Though Model B aims to enhance reliability, it selects the queries that the LLMs are more likely to provide a reliable answer, instead of that the query is more important for an accurate graph recovery. The reported results show that the strategy can improve graph recovery, but they do not establish how it compares with a strategy that directly targets structural error under the same query budget.

(3) For the technical contribution, based on my own experience, I would rate it as incremental, though I see value in combining different claim types within a single active querying framework, and in the reported savings over exhaustive querying.

For the above reasons, I would like to maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Summary

We sincerely thank all reviewers for their constructive feedback. We are encouraged that the reviewers recognize the value of introducing heterogeneous causal queries and reliability-aware active querying on causal knowledge. We would like to highlight three contributions of CLAIM-FUSE.

1. Unified representation of heterogeneous causal knowledge. Causal knowledge takes different forms, including directed edges, adjacency, ancestry, and topological ordering. Natural-language descriptions can be ambiguous about these distinctions, particularly between direct and indirect causation. CLAIM-FUSE provides causal relation specific natural-language queries and a unified likelihood for integrating their answers into the same graph distribution. As Reviewer rTYE notes, "Since all claim types update a shared particle distribution, their EIG values can be directly compared within a common framework."

2. Reliability-aware heterogeneous querying. Although causal relations are logically connected (Lemma 3 in Appendix B.1), LLMs may answer questions about them with different levels of accuracy (Figure 2 in subsection RQ2). CLAIM-FUSE accounts for answer reliability during query selection and answer-specific confidence during graph updates. Our new Figure 2 shows substantial variation in answer accuracy across causal relation types, models, and datasets, and demonstrates how type-specific reliability changes query allocation. We appreciate that Reviewer rTYE finds "the evidence for reliability-aware query selection persuasive."

3. Theoretical and empirical improvements in causal discovery. Our new Theorem 6 (Appendix B.4) shows that, on certain sparse causal graph families, heterogeneous queries can concentrate the exact graph posterior with a number of queries growing logarithmically in the number of variables, whereas local-only queries require at least linearly many. Tables 1–2 establish the best mean graph-recovery performance comprard to evaluated baselines. Table 3 and Appendix A.5 further show that reliability-aware querying over all four types is never significantly worse than eight restricted query pools, while significantly improving ED and orientation F1 most of the time.

In response to the reviewers, we substantially strengthened the paper:

  • C1: Added CaPE, BOSS, GRaSP, DAGMA, and an exhaustive LLM-only baseline (Tables 1–2).
  • C2: Introduced type-specific reliability-aware EIG and analyzed per-type answer accuracy and query share during active querying (Section 3.3, Figure 2).
  • C3: Added matched comparisons against eight restricted claim pools to show the advantage of introducing heterogeneous claims (Table 3, Appendix A.5).
  • C4: Established a logarithmic-versus-linear query-complexity (Theorem 6, Appendix B.4), showing the improvement in efficiency by using heterogeneous claims.
  • C5: Added confidence-calibration and robustness analyses, including Brier score, ECE, and miscalibration experiments.

Together, these contributions establish CLAIM-FUSE as a framework for actively acquiring heterogeneous causal knowledge while accounting for the uncertainty and reliability of its answers, with substantial theoretical and empirical evidence of its benefits.

We thank the reviewers again for helping us strengthen the work and hope the revised manuscript addresses their concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.