acceptodds
Under review as a conference paper at ICLR 2027

Causal Referee: Hierarchical Causal Discovery with LLM-guided Expert Reweighting

Abstract

Causal discovery aims to uncover causal structures from observational data, which is crucial for real-world decision-making. However, different causal discovery algorithms can produce divergent results that conflict with each other, complicating the identification of accurate causal graphs. Traditional approaches rely on numerical values and statistical assumptions, often ignoring rich domain-specific information, such as feature descriptions, which could also help structure learning. While recent works explore using Large Language Models (LLMs) to infer causal relations via direct queries, such methods can be unreliable due to a lack of alignment with the actual data. To address these limitations, we propose **Ca**usal **Re**feree (**CaRe**), a novel framework that aggregates structural insights from statistical discovery experts across different graph levels via linear opinion pooling, and uses an LLM as a meta-referee to dynamically reweight experts on disputed relations, thereby composing an improved and more complete causal graph. Extensive experiments on both synthetic and real-world datasets demonstrate that CaRe achieves the best average performance across a wide range of causal discovery methods, highlighting the effectiveness of using LLMs for meta-analysis in causal discovery.

Primary area:
causal reasoning
Keywords:
Causal Discovery, Large Language Models, Ensemble Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iBkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Causal Referee (CaRe), a framework that uses an LLM to coordinate multiple statistical causal discovery algorithms. Repeated resampling estimates each expert’s support for candidate relations. These supports are aggregated hierarchically at the skeleton, v-structure, and edge-orientation levels. For relations near the decision boundary, the LLM assigns relation-specific expert weights based on variable semantics, dataset characteristics, and the experts’ assumptions and opinions. Direction propagation, conflict checks, and acyclicity constraints produce the final DAG.

The evaluation covers five datasets sampled from benchmark Bayesian networks and three real-world or domain-derived datasets. The paper reports structural error, precision-recall AUC, and LLM query counts, with overall improvements over the evaluated individual algorithms and uniform aggregation. Additional experiments examine benchmark-identity masking, a newly generated graph, expert removal, the hierarchical design, query selection, and final DAG completion.

Critical strengths

The method provides a clear and implementable way to combine LLMs with statistical causal discovery. Statistical methods retain their own judgments, while the LLM changes their influence on selected disputed relations. Separating skeletons, v-structures, and orientations provides a common interface for heterogeneous graph outputs. Compared with asking an LLM to produce an entire graph, this design makes the sources of individual decisions easier to inspect.

Critical weaknesses

The evidence does not sufficiently isolate expert-reliability assessment as the source of improvement. The LLM sees each expert’s stance, and the prompts explicitly ask it to assess the causal plausibility of the candidate relation. It could therefore first decide whether the relation is plausible from semantics and then assign higher weights to experts that support that decision. Although expressed through weights, this need not differ from direct causal judgment constrained by the experts’ candidate relations. The direct causal expert in Table 2 predicts an entire graph, which differs from CaRe’s task of resolving individual disputed relations. Without a direct-relation baseline using the same disputed cases and context, and comparisons with global or layer-specific expert weights, the central interpretation of the LLM’s role remains insufficiently supported.

The evaluation does not fully characterize uncertainty in the complete pipeline, and graph conversion is insufficiently specified. Appendix D.3 fixes the data, expert outputs, and resampling support scores, then repeats only the LLM stage. This measures LLM variability but does not establish whether improvements persist across data realizations and expert estimation errors. Four sample sizes are not a substitute for independent repetitions. Moreover, DAGs, CPDAGs, and PAGs are converted into directed adjacency matrices without sufficiently explaining how undirected edges, circle endpoints, and bidirected edges are handled. These choices affect orientation errors and AUC, and therefore the interpretation of comparisons with statistical baselines.

Questions for authors
  1. Could the LLM directly select the relation state for the same disputed cases, using the same variable information and dataset context, followed by the same graph-construction and validity checks? How does this compare with CaRe? This would help establish whether reweighting is more reliable than direct relation judgment under matched conditions.
  2. Were multiple independent data seeds used at each sample size in the synthetic experiments? Could you report paired results and uncertainty after regenerating the data and rerunning the experts and resampling procedure, rather than repeating only the LLM stage with fixed expert outputs?
  3. How are DAGs, CPDAGs, and PAGs converted into directed adjacency matrices for evaluation? Is AUC computed from binary final graphs or continuous support scores? For Arctic-Sea, how is the mismatch between the cyclic/bidirectional reference graph and the DAG output space handled?
Reviewer_UmGzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that statistical causal discovery (SCD) methods suffer from inductive biases and scenario-dependent performance, while LLM-based causal discovery methods face high computational costs and reliability issues. It therefore proposes a framework that combines the two approaches by using an LLM as a causal referee rather than allowing it to directly infer causal relationships from input data. The LLM evaluates conflicting causal relations in the graphs produced by multiple SCD methods and leverages its prior knowledge to reconcile disagreements and construct a consensus causal graph. The proposed method is evaluated against 6 SCD algorithms, 2 ensemble-based methods, and 2 LLM-based methods.

Critical strengths
  1. Combining SCD algorithms with LLMs is a promising direction for causal discovery.
  2. The experiments demonstrate the benefit of hierarchical aggregation and LLM-guided aggregation over SCD baselines.
  3. Reducing the number of LLM queries by applying LLMs only on borderline edges is well motivated.
Critical weaknesses
  1. The main concern is that the proposed method constructs a causal graph by aggregating graphs produced by different SCD algorithms under different assumptions. The edges and orientations in these graphs can carry different semantics because of the assumptions and representations used to derive them, e.g., DAGs, CPDAGs, and PAGs represent different kinds of causal information and uncertainty. The paper does not justify how these heterogeneous causal representations can be mapped into a common relation space and meaningfully aggregated. This raises concerns about how valid the constructed graph is in a causality sense.

  2. Some benchmarks like Asia and Sachs are widely available in literature and have been seen by LLMs in training. These LLMs can correctly recover the causal graphs, however, CaRe misses the edge between "Asia" and "Tub" in Fig. 11 because none of the SCDs proposes this edge. This indicates a fundamental limitation: CaRe assumes that the causal relationships proposed by SCD algorithms define the admissible hypothesis space, while the LLM cannot independently propose missing relationships. Although this restriction reduces the risk of introducing unsupported LLM predictions, it also prevents the framework from exploiting potentially valid causal knowledge when all SCD algorithms make the same omission.

  3. Although the paper compares with 2 LLM-based causal discovery methods, the two baselines selected are based on direct LLM querying the causal relations of variable pairs. Recent work grounding LLM outputs on numerical data [1] and literature [2] are not included as baselines. Note that [2] is discussed in the paper but not included in the experiments. These methods are more suitable to be the baselines of the LLM-based methods in a sense that they provide alternatives to hypothesis aggregation.

References: [1] Huaming Du, Yujia Zheng, Baoyu Jing, Yu Zhao, Gang Kou, Guisong Liu, Tao Gu, Weimin Li, and Carl Yang. 2025. Causal Discovery through Synergizing Large Language Model and Data-Driven Reasoning. KDD 2025. [2] Yuzhe Zhang, Yipeng Zhang, Yidong Gan, Lina Yao, and Chen Wang. Causal graph discovery with retrieval-augmented generation based large language models, 2024. URL https://arxiv.org/abs/2402.15301.

Questions for authors

see weaknesses

Reviewer_6d9uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CaRe, a framework that aggregates the outputs of multiple structural causal discovery (SCD) experts by combining their bootstrap support scores through linear opinion pooling. For relationships whose aggregated support scores fall within a predefined margin around a decision threshold, an LLM (GPT-5.5) evaluates causal plausibility, expert reliability, and dataset compatibility to assign relationship-specific expert weights at the graph level.

Critical strengths

This study presents experiments across multiple datasets, supported by a range of ablation studies, providing a substantial body of empirical evidence.

The authors attempt to mitigate LLM hallucinations, which are a known limitation of LLM-based causal discovery compared with data-driven causal discovery.

The research motivation is compelling and is clearly presented in the introduction.

Critical weaknesses

1. Limited contribution from the data-driven ensemble components

Uniform-CaRe underperforms the best individual expert on the Asia and Sachs datasets, performs comparably on Alarm and Insurance, and outperforms the best expert only on Child and only in RNB. These results suggest that the improvements achieved by CaRe are primarily attributable to the LLM-assigned weights rather than to the data-driven aggregation itself. Moreover, the same comparison appears in Table 2, where the 'w/o LLM' column is identical to Uniform-CaRe. This raises questions about the contribution of the ensemble framework beyond the LLM-based weighting mechanism.

2. Benchmark familiarity and potential memorization

The five bnlearn networks are publicly available benchmarks that have been used for decades, raising concerns about possible memorization. In the direct-expert variant, the model achieves an RNB score of 0.000 on Asia, which the authors themselves interpret as a possible indication of information leakage.

The prompts in the main experiments include both dataset and expert names. Although these names are removed in the masked setting, the variable descriptions still retain their semantic meaning, potentially allowing the LLM to recognize familiar causal relationships. Furthermore, the evaluation relies on a single closed-source LLM, while the newly introduced graph based on CauseNet contains only 10 nodes, providing a relatively limited test of generalization to more complex and unfamiliar settings.

3. Insufficient validation of the core motivation

Figure 1 motivates the proposed framework by illustrating the failure of direct LLM queries when given misleading variable descriptions. However, the authors do not systematically evaluate how CaRe performs when variable descriptions are degraded, removed, or contaminated. Such experiments would help determine whether the proposed framework genuinely addresses the limitations identified in the motivation or remains sensitive to the same sources of misleading information.

4. Limited evaluation metrics and statistical reliability

The evaluation relies on AUC computed from a single binary adjacency matrix, while other standard graph-reconstruction metrics, such as SHD and F1, are not reported. This limits the ability to assess both adjacency recovery and edge orientation.

In addition, the tables appear to report results from a single run for each sample size, with variance estimates provided only for LLM resampling. Without repeated experiments across different data seeds, it is difficult to assess the stability and statistical reliability of the reported improvements.

5. Missing baselines and comparison targets

Several ensemble and LLM-hybrid methods, including [1–2], are discussed in the paper but are not included in the empirical comparisons, without a clear justification for their exclusion. The evaluation also lacks comparisons with non-LLM, data-driven weighting strategies and appropriate upper-bound references.

Furthermore, although established causal discovery methods such as NOTEARS, NTS-NOTEARS, PCMCI, DYNOTEARS, DAGMA, and FANTOM are available, the experiments consider only a limited subset of relevant approaches. A broader and better-justified baseline selection would help establish the effectiveness of the proposed framework.

6. Limited comparison with existing LLM-based causal discovery methods

The comparison with existing LLM-based causal discovery methods is insufficient to establish a clear advantage. For example, Kiciman et al. [3] report an F1 score of 0.65 on the Arctic sea ice dataset, which exceeds the performance of many causal discovery algorithms. However, the paper does not report sufficiently comparable metrics to assess this result directly. Moreover, the performance differences between CaRe and the evaluated LLM-based methods appear relatively small, and an LLM-based baseline performs better on one of the two reported metrics. Additional evaluations using common datasets and metrics, particularly F1, would provide a more convincing assessment of the proposed method's relative performance.

7. Limited methodological novelty and excessive reliance on the LLM

The central mechanism of CaRe relies on an LLM to evaluate causal relationships and determine how much weight to assign to individual experts. This raises concerns about the novelty of the proposed meta-referee framework and the extent to which its performance depends on the LLM.

According to Section 3.2, the LLM is asked to evaluate the causal validity of each contested relationship, together with dataset compatibility and expert reliability, and to derive expert weights from these assessments. Because contested relationships are precisely those on which the experts disagree, the convex-hull constraint may still permit a broad range of aggregated support scores. Consequently, the LLM's causal validity assessment can become the decisive factor in determining whether an edge is selected.

In this sense, CaRe may function more as an LLM-driven edge-selection method constrained by the candidates proposed by the experts than as a genuinely data-driven expert-reweighting framework. The claimed methodological contribution therefore appears somewhat overstated without further evidence separating the effects of LLM judgments from those of expert aggregation.

8. Insufficient analysis of expert selection and composition

The criteria for selecting the SCD experts are not clearly defined, and the paper provides little analysis of how the number or composition of experts affects performance. For example, it remains unclear how candidate-edge coverage changes as experts are added or removed, or whether certain combinations of experts are consistently more effective than others. Since the proposed method depends on the candidate relationships and support scores produced by these experts, its performance may be sensitive to their selection, potentially in a dataset-dependent manner. Additional experiments varying the number, type, and composition of experts would help establish the robustness and generalizability of the framework.

[1] Ahmed Abdulaal, adamos hadjivasiliou, Nina Montana-Brown, Tiantian He, Ayodeji Ijishakin, Ivana Drobnjak, Daniel C. Castro, and Daniel C. Alexander. Causal modelling agents: Causal graph discovery through synergising metadata- and data-driven reasoning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=pAoqRlTBtY.

[2] Vy Vo, Haoxuan Li, and Mingming Gong. Causal discovery in the wild: A voting-theoretic ensemble approach. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=WtbPaWO8lH.

[3] Emre Kıcıman, Robert Ness, Amit Sharma, and Chenhao Tan. Causal reasoning and large language models: Opening a new frontier for causality, 2024. URL https://arxiv.org/abs/2305.00050.

Questions for authors

1. Direct LLM edge selection: Could the authors include a baseline in which the LLM directly selects edges from the union of candidate relationships proposed by the experts, without assigning expert weights? The current results show only marginal improvements over some existing LLM-based methods. If direct edge selection achieves comparable performance, the methodological distinction between the proposed meta-referee framework and LLM-driven causal judgment would require further clarification.

2. Identical ablation results: In Table 3, the results for "w/o Opinions" are identical to those for "w/o LLM" in Table 2 across all eight datasets, down to the third decimal place. Could the authors explain this exact correspondence and clarify whether the two variants are functionally equivalent?

3. Additional graph-recovery metrics: Could the authors report SHD and F1 scores for both adjacency recovery and edge orientation, together with means and standard deviations across multiple data seeds for each sample size?

4. Comparison with prior work: Paper [4] reports an F1 score of 0.65 on the Arctic sea ice dataset. Could the authors report additional metrics and compare CaRe with the results in [4], preferably using the same datasets and evaluation protocols?

5. Matched LLM baseline: I recommend including an exhaustive pairwise LLM-based causal discovery baseline following the approach in [4], using the same GPT-5.5 model. This would help isolate the contribution of expert aggregation from the causal knowledge already available to the LLM.

6. Computational cost: Each query involves six methodological profiles and dataset statistics. How do the total wall-clock runtime, memory requirements, and LLM token costs compare with those of competing methods, including the computational cost of fitting and bootstrapping the individual experts?

7. Consistency between claims and results: The caption of Figure 1 states that the proposed method "consistently achieves the best results." However, this claim does not appear to align with the results on datasets such as Sangiovese, Alzheimer's, and Arctic sea ice. Could the authors verify this statement and revise it if necessary?

8. Definition of the alignment score: The alignment score reported in Figure 3 is not clearly defined. Could the authors provide its mathematical definition, explain how it is computed, and clarify how the reported values should be interpreted?

9. Extension to time-series data: How could the proposed framework be extended to time-series causal discovery, where temporal dependencies and lagged relationships must be considered?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.